diff options
author | Xin LI <delphij@FreeBSD.org> | 2023-01-01 04:23:22 +0000 |
---|---|---|
committer | Xin LI <delphij@FreeBSD.org> | 2023-01-01 04:23:22 +0000 |
commit | f6a891c2b422c077f3c8b764d0ebe4356f88fd23 (patch) | |
tree | 0ad7f804aae1b93eefbf55d70e33fa8a0d8ea366 | |
parent | 46780ea2dc97a43538e5dca6513baea560aeb839 (diff) | |
download | src-f6a891c2b422c077f3c8b764d0ebe4356f88fd23.tar.gz src-f6a891c2b422c077f3c8b764d0ebe4356f88fd23.zip |
Vendor import of xz 5.4.0 (trimmed)vendor/xz/5.4.0
86 files changed, 12753 insertions, 6177 deletions
@@ -19,6 +19,18 @@ Authors of XZ Utils Andrew Dudman helped adapting the scripts and their man pages for XZ Utils. + The initial version of the threaded .xz decompressor was written + by Sebastian Andrzej Siewior. + + The initial version of the .lz (lzip) decoder was written + by Michał Górny. + + CLMUL-accelerated CRC code was contributed by Ilya Kurdyukov. + + Other authors: + - Jonathan Nieder + - Joachim Henke + The GNU Autotools-based build system contains files from many authors, which I'm not trying to list here. diff --git a/ChangeLog b/ChangeLog index b71401c51521..f2c2bd3cad3f 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,23 +1,366 @@ -commit d8a898eb9974683bc725c49ec76722f9a8758f48 +commit b69da6d4bb6bb11fc0cf066920791990d2b22a06 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-11-30 18:33:05 +0200 +Date: 2022-12-13 20:37:17 +0200 - Bump version and soname for 5.2.9. + Bump version to 5.4.0 and soname to 5.4.0. - src/liblzma/Makefile.am | 2 +- - src/liblzma/api/lzma/version.h | 2 +- + src/liblzma/Makefile.am | 2 +- + src/liblzma/api/lzma/version.h | 6 +++--- + src/liblzma/liblzma_generic.map | 2 +- + src/liblzma/liblzma_linux.map | 2 +- + 4 files changed, 6 insertions(+), 6 deletions(-) + +commit 20869eb3fb280ff4f271ef527b12b6bf68b05e19 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-13 20:29:39 +0200 + + Update INSTALL: CMake on Windows isn't experimental anymore. + + Using CMake to build liblzma should work on a few other OSes + but building the command line tools is still subtly broken. + + It is known that shared library versioning may differ between + CMake and Libtool builds on some OSes, most notably Darwin. + + INSTALL | 10 ++++------ + 1 file changed, 4 insertions(+), 6 deletions(-) + +commit cbbd84451944e3e8c63acfaa3c923f6d8aff7852 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-13 19:47:53 +0200 + + Add NEWS for 5.4.0. + + NEWS | 202 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 202 insertions(+) + +commit c3e94d37e8d10a3e96019864b6f5d7b578db2c14 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-13 17:41:20 +0200 + + Fix a typo in NEWS. + + NEWS | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 0d2a2e0a545c3da2b3e9500f1e531eb903087245 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-13 17:41:03 +0200 + + Add NEWS for 5.2.10. + + NEWS | 12 ++++++++++++ + 1 file changed, 12 insertions(+) + +commit 177ece1c8eb007188fb1b04eff09ca2193fbdea6 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-13 12:30:45 +0200 + + Tests: Fix a typo in tests/files/README. + + tests/files/README | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 60f45bdbaa6b07558b3f4baac285739b0c6342f5 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-13 12:30:09 +0200 + + Tests: Add two ARM64 test files. + + tests/files/README | 7 +++++++ + tests/files/good-1-arm64-lzma2-1.xz | Bin 0 -> 512 bytes + tests/files/good-1-arm64-lzma2-2.xz | Bin 0 -> 488 bytes + tests/test_files.sh | 5 +++++ + 4 files changed, 12 insertions(+) + +commit f5e419550619c548c7c35d7e367cf00580a56521 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-12 22:44:21 +0200 + + Translations: Update the Catalan translation. + + po/ca.po | 657 +++++++++++++++++++++++++++++---------------------------------- + 1 file changed, 306 insertions(+), 351 deletions(-) + +commit 0fb9d355da3789b1757040af475b4e6bbc8b8af8 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-12 19:18:12 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit e5b6c161c61a37c54dcb76a99bbb83ac4abe02dc +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-12 19:07:58 +0200 + + Update AUTHORS. + + AUTHORS | 12 ++++++++++++ + 1 file changed, 12 insertions(+) + +commit f2d98e691099d82054d5f3071ef6f5e809932e44 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-12 15:31:14 +0200 + + Docs: Omit multi-threaded decompress from TODO. + + The TODO file outdated still. + + TODO | 2 -- + 1 file changed, 2 deletions(-) + +commit b42908c42a4cc091db45a7e5ba0e0ecceaa3f6da +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-11 21:16:09 +0200 + + Docs: Update xz-file-format.txt to 1.1.0 for ARM64 filter. + + doc/xz-file-format.txt | 29 ++++++++++++++++++++++------- + 1 file changed, 22 insertions(+), 7 deletions(-) + +commit 854f2f5946b353cb0963fd6dfd54d363adc89b9f +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-11 21:13:57 +0200 + + xz: Rename --experimental-arm64 to --arm64. + + src/xz/args.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 31dbd1e5fb65831915a7bbb531c3f19aea8d57a5 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-11 21:13:06 +0200 + + liblzma: Change LZMA_FILTER_ARM64 to the official Filter ID 0x0A. + + src/liblzma/api/lzma/bcj.h | 6 +----- + 1 file changed, 1 insertion(+), 5 deletions(-) + +commit 01b3549e523edac899ec4925b282ceddd20da116 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-08 19:24:22 +0200 + + xz: Make args_info.files_name a const pointer. + + src/xz/args.c | 2 +- + src/xz/args.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -commit efd4430e21f798e198a6ee8a368a79065139eb54 +commit bc665b84ea6bf7946394a08122177efe41b26a5f +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-08 19:18:16 +0200 + + xz: Don't modify argv[]. + + The code that parses --memlimit options and --block-list modified + the argv[] when parsing the option string from optarg. This was + visible in "ps auxf" and such and could be confusing. I didn't + understand it back in the day when I wrote that code. Now a copy + is allocated when modifiable strings are needed. + + src/xz/args.c | 23 +++++++++++++++++++---- + 1 file changed, 19 insertions(+), 4 deletions(-) + +commit a13064e1c290de7933db72b6dffbd65cfce59c9f Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-11-30 18:31:16 +0200 +Date: 2022-12-08 18:18:57 +0200 + + Translations: Update the German man page translations. + + po4a/de.po | 4570 ++++++++++++++++++------------------------------------------ + 1 file changed, 1374 insertions(+), 3196 deletions(-) + +commit 8bdbe42a8d0d75dff70206b923fc4bce5c69a40a +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-12-06 23:05:56 +0800 + + Translations: Update the German translation. + + po/de.po | 586 ++++++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 315 insertions(+), 271 deletions(-) + +commit 5c304b57c24ef40ff57f864301065f0244c05bde +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-12-06 23:04:25 +0800 + + Translations: Update the Turkish translation. + + po/tr.po | 221 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 108 insertions(+), 113 deletions(-) + +commit 6d86781fdb937970486500447ebb49b98244235b +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-12-06 23:02:11 +0800 + + Translations: Update the Croatian translation. + + po/hr.po | 228 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 113 insertions(+), 115 deletions(-) + +commit 7a5b4b8075eb36026b1796f04ffed5830c42396a +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-12-06 22:56:30 +0800 + + Translations: Add Romanian translation of man pages. + + Thanks to Remus-Gabriel Chelu. + + po4a/po4a.conf | 2 +- + po4a/ro.po | 3692 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 3693 insertions(+), 1 deletion(-) + +commit c6977e7400088177556e8771bcb839eb7d90caa3 +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-12-06 22:52:13 +0800 + + Translations: Update the Romanian translation. + + po/ro.po | 294 +++++++++++++++++++++++++++++++-------------------------------- + 1 file changed, 147 insertions(+), 147 deletions(-) + +commit ac2a747e939c2cbccff7a49c399769af5e02d2ab +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-08 17:30:09 +0200 + + liblzma: Check for unexpected NULL pointers in block_header_decode(). + + The API docs gave an impression that such checks are done + but they actually weren't done. In practice it made little + difference since the calling code has a bug if these are NULL. + + Thanks to Jia Tan for the original patch that checked for + block->filters == NULL. + + src/liblzma/common/block_header_decoder.c | 4 ++++ + 1 file changed, 4 insertions(+) + +commit 24790f49ae66938c1c7574315e1c0aba1ed5ed25 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-01 20:59:32 +0200 + + Bump version number for 5.3.5beta. + + This also sorts the symbol names alphabetically in liblzma_*.map. + + src/liblzma/api/lzma/version.h | 4 ++-- + src/liblzma/liblzma_generic.map | 10 +++++----- + src/liblzma/liblzma_linux.map | 10 +++++----- + 3 files changed, 12 insertions(+), 12 deletions(-) + +commit 7e53c5bcb3c2c17f47c096c06ff6b1481e6ecafa +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-01 20:57:26 +0200 + + Add NEWS for 5.3.5beta. + + NEWS | 43 +++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 43 insertions(+) + +commit 5865f2aaac326fcbd9f8a7d62defa230e4cb644e +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-01 20:57:09 +0200 + + Update THANKS. + + THANKS | 3 +++ + 1 file changed, 3 insertions(+) + +commit 62b270988ec67314d69976df484d2974c6eacfda +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-01 20:04:17 +0200 + + liblzma: Use __has_attribute(__symver__) to fix Clang detection. + + If someone sets up Clang to define __GNUC__ to 10 or greater + then symvers broke. __has_attribute is supported by such GCC + and Clang versions that don't support __symver__ so this should + be much better and simpler way to detect if __symver__ is + actually supported. + + Thanks to Tomasz Gajc for the bug report. + + src/liblzma/common/common.h | 15 ++++++++++++++- + 1 file changed, 14 insertions(+), 1 deletion(-) + +commit f9ca7d45162664ddd9fb70e19335c2426e5d75bb +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-01 18:51:52 +0200 + + liblzma: Omit zero-skipping from ARM64 filter. + + It has some complicated downsides and its usefulness is more limited + than I originally thought. So this change is bad for certain very + specific situations but a generic solution that works for other + filters (and is otherwise better too) is planned anyway. And this + way 7-Zip can use the same compatible filter for the .7z format. + + This is still marked as experimental with a new temporary Filter ID. + + src/liblzma/api/lzma/bcj.h | 2 +- + src/liblzma/simple/arm64.c | 81 +++++++++++++--------------------------------- + 2 files changed, 24 insertions(+), 59 deletions(-) + +commit 5baec3f0a9c85e6abf45c0f652f699b074129a8b +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-01 18:13:27 +0200 + + xz: Omit the special notes about ARM64 filter on the man page. + + src/xz/xz.1 | 5 ++--- + 1 file changed, 2 insertions(+), 3 deletions(-) + +commit 0c3627b51862eb0dcdd4fc283d046250571991c6 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-01 18:12:03 +0200 + + liblzma: Don't be over-specific in lzma_str_to_filters API doc. + + src/liblzma/api/lzma/filter.h | 3 +-- + 1 file changed, 1 insertion(+), 2 deletions(-) + +commit 94adf057f27b1970f493dc99cd166407d7255639 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-01 17:54:23 +0200 + + liblzma: Silence unused variable warning when BCJ filters are disabled. + + Thanks to Jia Tan for the original patch. + + src/liblzma/common/string_conversion.c | 15 +++++++++++++++ + 1 file changed, 15 insertions(+) + +commit c68af4441744e5ffc41a472e1be9c9d53a1d9780 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-12-01 17:38:03 +0200 + + Translations: Update the Chinese (simplified) translation. + + po/zh_CN.po | 608 ++++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 348 insertions(+), 260 deletions(-) + +commit 3be6942e5c27d29995d41da52fbe274e4ce4a537 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-30 18:55:03 +0200 Add NEWS for 5.2.9. NEWS | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) -commit 2dc1bc97a562ef4d4134c8df4d344d0f403428b8 +commit 7c16e312cb2f40b81154c0e5be13a3c6b8da485d +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-11-30 23:33:08 +0800 + + xz: Remove message_filters_to_str function prototype from message.h. + + This was forgotten from 7484744af6cbabe81e92af7d9e061dfd597fff7b. + + src/xz/message.h | 16 ---------------- + 1 file changed, 16 deletions(-) + +commit 764955e2d4f2a5e8d6d6fec63af694f799e050e7 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-30 18:08:34 +0200 @@ -35,7 +378,23 @@ Date: 2022-11-30 18:08:34 +0200 windows/README-Windows.txt | 2 +- 5 files changed, 9 insertions(+), 10 deletions(-) -commit fb13a234d9def06cbe2b8ed88ece7db0c6d5c39f +commit c21983c76031e01da01ad3c6cc716fe4b8a75070 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-30 17:50:17 +0200 + + Build: Add string_conversion.c to CMake, DOS, and VS files. + + CMakeLists.txt | 1 + + dos/Makefile | 1 + + windows/vs2013/liblzma.vcxproj | 1 + + windows/vs2013/liblzma_dll.vcxproj | 1 + + windows/vs2017/liblzma.vcxproj | 1 + + windows/vs2017/liblzma_dll.vcxproj | 1 + + windows/vs2019/liblzma.vcxproj | 1 + + windows/vs2019/liblzma_dll.vcxproj | 1 + + 8 files changed, 8 insertions(+) + +commit 30be0c35d24eb5175459d69dbf7d92e2b087ef82 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-30 17:38:32 +0200 @@ -44,7 +403,170 @@ Date: 2022-11-30 17:38:32 +0200 AUTHORS | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) -commit 841448e36d4455d39ff1e1115843360934e7a507 +commit 0a72b9ca2fe20082da9b7128fe0d908af947a851 +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-11-30 00:52:06 +0800 + + liblzma: Improve documentation for string to filter functions. + + src/liblzma/api/lzma/filter.h | 17 +++++++++-------- + 1 file changed, 9 insertions(+), 8 deletions(-) + +commit a6e21fcede3b196160a52dd294d965c508a4bb33 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-29 22:27:42 +0200 + + liblzma: Two fixes to lzma_str_list_filters() API docs. + + Thanks to Jia Tan. + + src/liblzma/api/lzma/filter.h | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit 7484744af6cbabe81e92af7d9e061dfd597fff7b +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-28 21:57:47 +0200 + + xz: Use lzma_str_from_filters(). + + Two uses: Displaying encoder filter chain when compressing with -vv, + and displaying the decoder filter chain in --list -vv. + + src/xz/list.c | 28 ++++++--- + src/xz/message.c | 175 +++---------------------------------------------------- + 2 files changed, 28 insertions(+), 175 deletions(-) + +commit cedeeca2ea6ada5b0411b2ae10d7a859e837f203 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-28 21:37:48 +0200 + + liblzma: Add lzma_str_to_filters, _from_filters, and _list_filters. + + lzma_str_to_filters() uses static error messages which makes + them not very precise. It tells the position in the string + where an error occurred though which helps quite a bit if + applications take advantage of it. Dynamic error messages can + be added later with a new flag if it seems important enough. + + src/liblzma/api/lzma/filter.h | 258 +++++++ + src/liblzma/common/Makefile.inc | 1 + + src/liblzma/common/string_conversion.c | 1302 ++++++++++++++++++++++++++++++++ + src/liblzma/liblzma_generic.map | 3 + + src/liblzma/liblzma_linux.map | 3 + + 5 files changed, 1567 insertions(+) + +commit 072ebf7b1335421193ffa9d4a70d5533786b8995 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-28 21:02:19 +0200 + + liblzma: Make lzma_validate_chain() available outside filter_common.c. + + src/liblzma/common/filter_common.c | 8 ++++---- + src/liblzma/common/filter_common.h | 3 +++ + 2 files changed, 7 insertions(+), 4 deletions(-) + +commit 5f22bd2d37e3bd01a5d701b51750eb51f09c11bf +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-28 10:51:03 +0200 + + liblzma: Remove lzma_lz_decoder_uncompressed() as it's now unused. + + src/liblzma/lz/lz_decoder.c | 14 -------------- + src/liblzma/lz/lz_decoder.h | 3 --- + 2 files changed, 17 deletions(-) + +commit cee83206465b95729ab649aa2f57fdbde8dcaf89 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-28 10:48:53 +0200 + + liblzma: Use LZMA1EXT feature in lzma_microlzma_decoder(). + + Here too this avoids the slightly ugly method to set + the uncompressed size. + + Also moved the setting of dict_size to the struct initializer. + + src/liblzma/common/microlzma_decoder.c | 15 ++++++++------- + 1 file changed, 8 insertions(+), 7 deletions(-) + +commit e310e8b6a490dfb468f4ed68feff246d776b323c +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-28 10:28:20 +0200 + + liblzma: Use LZMA1EXT feature in lzma_alone_decoder(). + + This avoids the need to use the slightly ugly method to + set the uncompressed size. + + src/liblzma/common/alone_decoder.c | 18 +++++++++++++----- + 1 file changed, 13 insertions(+), 5 deletions(-) + +commit 33b8a24b6646a9dbfd8358405aec466b13078559 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-27 23:16:21 +0200 + + liblzma: Add LZMA_FILTER_LZMA1EXT to support LZMA1 without end marker. + + Some file formats need support for LZMA1 streams that don't use + the end of payload marker (EOPM) alias end of stream (EOS) marker. + So far liblzma API has supported decompressing such streams via + lzma_alone_decoder() when .lzma header specifies a known + uncompressed size. Encoding support hasn't been available in the API. + + Instead of adding a new LZMA1-only API for this purpose, this commit + adds a new filter ID for use with raw encoder and decoder. The main + benefit of this approach is that then also filter chains are possible, + for example, if someone wants to implement support for .7z files that + use the x86 BCJ filter with LZMA1 (not BCJ2 as that isn't supported + in liblzma). + + src/liblzma/api/lzma/lzma12.h | 123 ++++++++++++++++++++++++++++++-- + src/liblzma/common/filter_common.c | 7 ++ + src/liblzma/common/filter_decoder.c | 6 ++ + src/liblzma/common/filter_encoder.c | 9 +++ + src/liblzma/lzma/lzma2_encoder.c | 2 +- + src/liblzma/lzma/lzma_decoder.c | 26 ++++++- + src/liblzma/lzma/lzma_encoder.c | 40 +++++++++-- + src/liblzma/lzma/lzma_encoder.h | 3 +- + src/liblzma/lzma/lzma_encoder_private.h | 3 + + 9 files changed, 204 insertions(+), 15 deletions(-) + +commit 9a304bf1e45b3ddf61aaeaa7c764915b34618ede +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-27 18:43:07 +0200 + + liblzma: Avoid unneeded use of void pointer in LZMA decoder. + + src/liblzma/lzma/lzma_decoder.c | 3 +-- + src/liblzma/lzma/lzma_decoder.h | 2 +- + 2 files changed, 2 insertions(+), 3 deletions(-) + +commit 218394958c7683f892275bb40eae880620feebcc +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-27 18:20:33 +0200 + + liblzma: Pass the Filter ID to LZ encoder and decoder. + + This allows using two Filter IDs with the same + initialization function and data structures. + + src/liblzma/common/alone_decoder.c | 1 + + src/liblzma/common/alone_encoder.c | 1 + + src/liblzma/common/common.h | 7 +++++-- + src/liblzma/common/lzip_decoder.c | 1 + + src/liblzma/common/microlzma_decoder.c | 1 + + src/liblzma/common/microlzma_encoder.c | 1 + + src/liblzma/lz/lz_decoder.c | 5 +++-- + src/liblzma/lz/lz_decoder.h | 3 ++- + src/liblzma/lz/lz_encoder.c | 5 +++-- + src/liblzma/lz/lz_encoder.h | 3 ++- + src/liblzma/lzma/lzma2_decoder.c | 3 ++- + src/liblzma/lzma/lzma2_encoder.c | 3 ++- + src/liblzma/lzma/lzma_decoder.c | 2 +- + src/liblzma/lzma/lzma_encoder.c | 2 +- + 14 files changed, 26 insertions(+), 12 deletions(-) + +commit 1663c7676b76f4c514031797f3db1896e8100f7f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-27 01:03:16 +0200 @@ -53,7 +575,92 @@ Date: 2022-11-27 01:03:16 +0200 src/liblzma/common/filter_encoder.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -commit b61da00c7f535bd48930a878efdf6843d804983e +commit 11fe708db783ac36ebeeb85da164e29e8c300910 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-26 22:25:30 +0200 + + xz: Use lzma_filters_free(). + + src/xz/list.c | 8 ++------ + 1 file changed, 2 insertions(+), 6 deletions(-) + +commit e782af9110d8499c7ac2929bc871540eefea5ea1 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-26 22:21:13 +0200 + + liblzma: Use lzma_filters_free() in more places. + + src/liblzma/common/block_header_decoder.c | 20 ++------------------ + src/liblzma/common/stream_decoder.c | 4 +--- + src/liblzma/common/stream_decoder_mt.c | 22 +++++----------------- + 3 files changed, 8 insertions(+), 38 deletions(-) + +commit 90caaded2dc6db1d6a55b01160d7e87f4a423628 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-25 18:04:37 +0200 + + liblzma: Omit simple coder init functions if they are disabled. + + src/liblzma/simple/arm.c | 4 ++++ + src/liblzma/simple/armthumb.c | 4 ++++ + src/liblzma/simple/ia64.c | 4 ++++ + src/liblzma/simple/powerpc.c | 4 ++++ + src/liblzma/simple/sparc.c | 4 ++++ + src/liblzma/simple/x86.c | 4 ++++ + 6 files changed, 24 insertions(+) + +commit 5cd9f0df78cc4f8a7807bf6104adea13034fbb45 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-24 23:24:59 +0200 + + xz: Allow nice_len 2 and 3 even if match finder requires 3 or 4. + + Now that liblzma accepts these, we avoid the extra check and + there's one message less for translators too. + + src/xz/options.c | 5 ----- + 1 file changed, 5 deletions(-) + +commit 3be88ae071371caa279b44e13f4836fb178fe4ae +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-24 23:23:55 +0200 + + liblzma: Allow nice_len 2 and 3 even if match finder requires 3 or 4. + + That is, if the specified nice_len is smaller than the minimum + of the match finder, silently use the match finder's minimum value + instead of reporting an error. The old behavior is annoying to users + and it complicates xz options handling too. + + src/liblzma/lz/lz_encoder.c | 14 +++++++++----- + src/liblzma/lz/lz_encoder.h | 9 +++++++++ + src/liblzma/lzma/lzma_encoder.c | 11 ++++++++--- + 3 files changed, 26 insertions(+), 8 deletions(-) + +commit 93439cfafe1768b3b18d67d2356ef7e7559bba59 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-24 16:25:10 +0200 + + liblzma: Add lzma_filters_update() support to the multi-threaded encoder. + + A tiny downside of this is that now a 1-4 tiny allocations are made + for every Block because each worker thread needs its own copy of + the filter chain. + + src/liblzma/api/lzma/filter.h | 36 +++++++------ + src/liblzma/common/stream_encoder_mt.c | 96 +++++++++++++++++++++++++++++++--- + 2 files changed, 109 insertions(+), 23 deletions(-) + +commit 17ac51e689794eb41cab3e80946fec689caea2d2 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-24 14:53:22 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 06824396b2b6c84f3a235cb7c19c2a9701167797 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-24 14:52:44 +0200 @@ -81,16 +688,25 @@ Date: 2022-11-24 14:52:44 +0200 src/liblzma/common/common.h | 12 ++++ 3 files changed, 111 insertions(+), 49 deletions(-) -commit 6c29793b3cac292b25801d011db6d8ccade50396 +commit e1acf7107291f8b3d6d609a7133331ff36d35d14 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-11-24 00:02:31 +0200 +Date: 2022-11-24 01:32:16 +0200 - CMake: Don't use symbol versioning with static library. + liblzma: Refactor to use lzma_filters_free(). + + lzma_filters_free() sets the options to NULL and ids to + LZMA_VLI_UNKNOWN so there is no need to do it by caller; + the filter arrays will always be left in a safe state. + + Also use memcpy() instead of a loop to copy a filter chain + when it is known to be safe to copy LZMA_FILTERS_MAX + 1 + (even if the elements past the terminator might be uninitialized). - CMakeLists.txt | 10 +++++++--- - 1 file changed, 7 insertions(+), 3 deletions(-) + src/liblzma/common/stream_encoder.c | 16 ++++------------ + src/liblzma/common/stream_encoder_mt.c | 11 ++--------- + 2 files changed, 6 insertions(+), 21 deletions(-) -commit 872623def5e3e5c7f9f14f366d908f3e012d1007 +commit cb05dbcf8b868441ec805016222f3fd77f1c5caa Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-24 01:26:37 +0200 @@ -109,7 +725,7 @@ Date: 2022-11-24 01:26:37 +0200 src/liblzma/common/stream_encoder_mt.c | 4 ++++ 1 file changed, 4 insertions(+) -commit b0f8d9293ca5a0a56964695a59a098c9d1e82b99 +commit 75f1a6c26df4ce329da0882786403e3ccf5cd898 Author: Jia Tan <jiat0218@gmail.com> Date: 2022-05-05 20:53:42 +0800 @@ -124,7 +740,31 @@ Date: 2022-05-05 20:53:42 +0800 src/liblzma/common/block_encoder.c | 1 + 1 file changed, 1 insertion(+) -commit 6997e0b5e2339025646cfaec13a3317fb340729b +commit d0901645170b638c517f5c50866b6ef48f491c65 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-24 01:02:50 +0200 + + liblzma: Add new API function lzma_filters_free(). + + This is small but convenient and should have been added + a long time ago. + + src/liblzma/api/lzma/filter.h | 21 +++++++++++++++++++++ + src/liblzma/common/filter_common.c | 26 ++++++++++++++++++++++++++ + src/liblzma/liblzma_generic.map | 1 + + src/liblzma/liblzma_linux.map | 1 + + 4 files changed, 49 insertions(+) + +commit ae1f8a723dcde2f2c5cf444bcbb5fc5026b3c3c5 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-24 00:02:31 +0200 + + CMake: Don't use symbol versioning with static library. + + CMakeLists.txt | 10 +++++++--- + 1 file changed, 7 insertions(+), 3 deletions(-) + +commit 48c1b99dc537a27e1ca929d8837e778e5ba32191 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-23 21:55:22 +0200 @@ -133,7 +773,7 @@ Date: 2022-11-23 21:55:22 +0200 src/liblzma/api/lzma/filter.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -commit f94a3e34603c56c55777056bb5412bfd0e948f0b +commit 10430fbf3820dafd4eafd38ec8be161a6978ed2b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-23 21:26:21 +0200 @@ -155,7 +795,7 @@ Date: 2022-11-23 21:26:21 +0200 src/liblzma/common/stream_encoder.c | 39 +++++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 8 deletions(-) -commit 8309385b444bce23e56256e21fb008a170434008 +commit cafd6dc397ca8b5b5f7775e8d6876b8fe70f8e70 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-22 16:37:15 +0200 @@ -164,7 +804,7 @@ Date: 2022-11-22 16:37:15 +0200 src/liblzma/common/stream_encoder.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 5fecba6022cbbeed8103b49d79b2fb36086be72d +commit c392bf8ccba857baaf50399c4b460119befacd54 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-22 11:20:17 +0200 @@ -181,7 +821,7 @@ Date: 2022-11-22 11:20:17 +0200 src/liblzma/lzma/lzma_encoder.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) -commit 1946b2b141d8b12c9325ba48d3a04ad8e8206750 +commit f50534c973a591ccf65485adfc827a8a7126ca6c Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-21 13:02:33 +0200 @@ -193,26 +833,300 @@ Date: 2022-11-21 13:02:33 +0200 src/liblzma/api/lzma/index_hash.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -commit 5476089d9c42b9b04e92b80e1800b384a98265cb +commit 649d4872ed2f55196114a061d45b416fc4353569 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-11-13 19:58:47 +0200 +Date: 2022-11-19 19:09:55 +0200 - Bump version and soname for 5.2.8. + xz: Refactor duplicate code from hardware_memlimit_mtenc_get(). - src/liblzma/Makefile.am | 2 +- - src/liblzma/api/lzma/version.h | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) + src/xz/hardware.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit d327743bb547a53364e5951a16e5f1663fe4b9ff +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-19 19:06:13 +0200 + + xz: Add support --threads=+N so that -T+1 gives threaded mode. + + src/xz/args.c | 18 +++++++++++++++--- + src/xz/hardware.c | 17 +++++++++++++++-- + src/xz/hardware.h | 1 + + src/xz/xz.1 | 21 ++++++++++++++++++++- + 4 files changed, 51 insertions(+), 6 deletions(-) + +commit a11a2b8b5e830ba682c1d81aaa7078842b296995 +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-11-19 23:18:04 +0800 + + CMake: Adds test_memlimit to CMake tests + + CMakeLists.txt | 1 + + 1 file changed, 1 insertion(+) + +commit 2af8d9e9b3f44f62d19e7c39297ec63af2e8c64f +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-15 19:10:21 +0200 + + Translations: Update the Korean translation. + + po/ko.po | 652 ++++++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 371 insertions(+), 281 deletions(-) -commit f9994f395d04d37b75e879f90397258b86b466cc +commit 16ac05677292f7e21a4feaddcfb2ab062ea5f385 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-11-13 19:57:26 +0200 +Date: 2022-11-15 19:09:28 +0200 + + Translations: Update the Turkish translation. + + po/tr.po | 568 ++++++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 310 insertions(+), 258 deletions(-) + +commit b9a67d9a5fa207062d4aa8a01639234609315d31 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-15 10:58:39 +0200 + + Bump version number for 5.3.4alpha. + + src/liblzma/api/lzma/version.h | 2 +- + src/liblzma/liblzma_generic.map | 2 +- + src/liblzma/liblzma_linux.map | 2 +- + 3 files changed, 3 insertions(+), 3 deletions(-) + +commit 5b999ba289b3280457b7386b9ac65dbbdf1575a5 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-15 10:54:40 +0200 + + Add NEWS for 5.3.4alpha. + + NEWS | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 96 insertions(+) + +commit ce8db9e37da4f6c87691c5066f51f91f2411c44a +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-15 10:54:08 +0200 Add NEWS for 5.2.8. NEWS | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) -commit cdf14b28993ef2f3168821372467e3aa7cf1e973 +commit b56bc8251d2736224af6bdaaae734ceb8926a879 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-14 23:19:57 +0200 + + Revert "liblzma: Simple/BCJ filters: Allow disabling generic BCJ options." + + This reverts commit 177bdc922cb17bd0fd831ab8139dfae912a5c2b8 + and also does equivalent change to arm64.c. + + Now that ARM64 filter will use lzma_options_bcj, this change + is not needed anymore. + + src/liblzma/simple/arm.c | 2 +- + src/liblzma/simple/arm64.c | 2 +- + src/liblzma/simple/armthumb.c | 2 +- + src/liblzma/simple/ia64.c | 2 +- + src/liblzma/simple/powerpc.c | 2 +- + src/liblzma/simple/simple_coder.c | 4 ++-- + src/liblzma/simple/simple_private.h | 2 +- + src/liblzma/simple/sparc.c | 2 +- + src/liblzma/simple/x86.c | 3 +-- + 9 files changed, 10 insertions(+), 11 deletions(-) + +commit 8370ec8edf9ddf8d1d9fef03d8d1027503ec4c35 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-14 23:14:41 +0200 + + Replace the experimental ARM64 filter with a new experimental version. + + This is incompatible with the previous version. + + This has space/tab fixes in filter_*.c and bcj.h too. + + src/liblzma/api/lzma/bcj.h | 41 +----- + src/liblzma/common/filter_common.c | 14 +- + src/liblzma/common/filter_decoder.c | 12 +- + src/liblzma/common/filter_encoder.c | 17 +-- + src/liblzma/simple/arm64.c | 283 ++++++++++++++---------------------- + src/liblzma/simple/simple_decoder.h | 4 - + src/liblzma/simple/simple_encoder.h | 2 - + src/xz/args.c | 2 +- + src/xz/message.c | 13 +- + src/xz/options.c | 39 ----- + src/xz/options.h | 7 - + 11 files changed, 147 insertions(+), 287 deletions(-) + +commit f644473a211394447824ea00518d0a214ff3f7f2 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-14 21:34:57 +0200 + + liblzma: Add fast CRC64 for 32/64-bit x86 using SSSE3 + SSE4.1 + CLMUL. + + It also works on E2K as it supports these intrinsics. + + On x86-64 runtime detection is used so the code keeps working on + older processors too. A CLMUL-only build can be done by using + -msse4.1 -mpclmul in CFLAGS and this will reduce the library + size since the generic implementation and its 8 KiB lookup table + will be omitted. + + On 32-bit x86 this isn't used by default for now because by default + on 32-bit x86 the separate assembly file crc64_x86.S is used. + If --disable-assembler is used then this new CLMUL code is used + the same way as on 64-bit x86. However, a CLMUL-only build + (-msse4.1 -mpclmul) won't omit the 8 KiB lookup table on + 32-bit x86 due to a currently-missing check for disabled + assembler usage. + + The configure.ac check should be such that the code won't be + built if something in the toolchain doesn't support it but + --disable-clmul-crc option can be used to unconditionally + disable this feature. + + CLMUL speeds up decompression of files that have compressed very + well (assuming CRC64 is used as a check type). It is know that + the CLMUL code is significantly slower than the generic code for + tiny inputs (especially 1-8 bytes but up to 16 bytes). If that + is a real-world problem then there is already a commented-out + variant that uses the generic version for small inputs. + + Thanks to Ilya Kurdyukov for the original patch which was + derived from a white paper from Intel [1] (published in 2009) + and public domain code from [2] (released in 2016). + + [1] https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf + [2] https://github.com/rawrunprotected/crc + + CMakeLists.txt | 26 ++- + INSTALL | 12 ++ + configure.ac | 59 +++++- + src/liblzma/check/crc64_fast.c | 449 +++++++++++++++++++++++++++++++++++++++- + src/liblzma/check/crc64_table.c | 21 +- + 5 files changed, 554 insertions(+), 13 deletions(-) + +commit 3b466bc79672bb2b06d1245a500588e6026e0ba0 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-14 20:14:34 +0200 + + Translations: Update the Swedish translation one more time. + + po/sv.po | 6 +++--- + 1 file changed, 3 insertions(+), 3 deletions(-) + +commit e963379a8622ebdff6ce78e76b803bcd1e1d16d6 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-14 19:34:15 +0200 + + Translations: Update the Swedish translation again. + + po/sv.po | 16 ++++++++-------- + 1 file changed, 8 insertions(+), 8 deletions(-) + +commit a4bc689a823a2254f29ac9d233170add5121b307 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-14 19:07:45 +0200 + + Translations: Update the Swedish translation. + + po/sv.po | 671 ++++++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 382 insertions(+), 289 deletions(-) + +commit bbf2073d824ab4ba33bed4b77f467435abd333a5 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-14 18:58:09 +0200 + + Translations: Update the Ukrainian translation. + + po/uk.po | 618 ++++++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 354 insertions(+), 264 deletions(-) + +commit ac10b1b3622e70881595586edfb8a3ebdcd76bb6 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-14 17:58:07 +0200 + + Build: Omit x86_64 from --enable-assembler. + + It didn't do anything. There are only 32-bit x86 assembly files + and it feels likely that new files won't be added as intrinsics + in C are more portable across toolchains and OSes. + + configure.ac | 6 ++---- + 1 file changed, 2 insertions(+), 4 deletions(-) + +commit eb0f1450ad9f23dac03050d9c8375980240aee21 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-14 16:00:52 +0200 + + liblzma: Use __attribute__((__constructor__)) if available. + + This uses it for CRC table initializations when using --disable-small. + It avoids mythread_once() overhead. It also means that then + --disable-small --disable-threads is thread-safe if this attribute + is supported. + + CMakeLists.txt | 15 +++++++++++++++ + INSTALL | 4 +++- + configure.ac | 31 ++++++++++++++++++++++++++++--- + src/liblzma/check/crc32_small.c | 7 +++++++ + src/liblzma/check/crc64_small.c | 5 +++++ + src/liblzma/lz/lz_encoder.c | 2 +- + 6 files changed, 59 insertions(+), 5 deletions(-) + +commit 6553f49b11dafad35c73b05f12e14865ea1fd8a1 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-12 21:19:52 +0200 + + Translations: Update the Romanian translation. + + po/ro.po | 651 +++++++++++++++++++++++++++++++++++++-------------------------- + 1 file changed, 380 insertions(+), 271 deletions(-) + +commit db97e69e12393becc29f8febd53133d0d36989bd +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-12 21:17:45 +0200 + + Translations: Update the Hungarian translation. + + po/hu.po | 625 ++++++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 357 insertions(+), 268 deletions(-) + +commit 2bbb9c0f3829a8b121b36998d273a6c6f92000f4 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-11 17:58:57 +0200 + + Translations: Update the Finnish translation. + + po/fi.po | 610 ++++++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 348 insertions(+), 262 deletions(-) + +commit 3c8cbb8137b6f8ed9416c1209d73cdbcb015251f +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-11 17:58:18 +0200 + + Translations: Update the Croatian translation. + + po/hr.po | 680 +++++++++++++++++++++++++++++++++++---------------------------- + 1 file changed, 381 insertions(+), 299 deletions(-) + +commit 26c3359eac0988d6f3986735cd1363bec1678e8e +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-11 17:57:18 +0200 + + Translations: Update the Polish translation. + + po/pl.po | 569 ++++++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 309 insertions(+), 260 deletions(-) + +commit 577e467b137c735afb8de6ae71ac7a73c2960cc4 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-11 17:56:44 +0200 + + Translations: Update the Spanish translation. + + po/es.po | 598 ++++++++++++++++++++++++++++++++++++--------------------------- + 1 file changed, 344 insertions(+), 254 deletions(-) + +commit f9b4ff6e9a0f1678650775582d3e4fe782abce97 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-11 17:16:03 +0200 @@ -221,7 +1135,7 @@ Date: 2022-11-11 17:16:03 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 454f567e58bbadea5cff4f31ef450c1305e9853d +commit a39961ef211e1bf030b17edeea3cff29fe263b67 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-11 17:15:25 +0200 @@ -234,7 +1148,7 @@ Date: 2022-11-11 17:15:25 +0200 src/liblzma/common/common.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 2f01169f5a81e21e7a7e5f799c32472c6277b1d5 +commit c715f683dcb1a817d565da292cddfbceda643e12 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-11 14:35:58 +0200 @@ -248,7 +1162,28 @@ Date: 2022-11-11 14:35:58 +0200 src/liblzma/common/memcmplen.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) -commit fc1358679ed3994f67e5e8c869c281be82370a32 +commit bd334ae56afe7f642ad4d0f1ac19e74e82daa1ce +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-11 13:27:06 +0200 + + Add NEWS for 5.2.7 (forgotten cherry-pick from v5.2). + + NEWS | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 74 insertions(+) + +commit 3c7860cf49de6f81046b3a4034a89f3a4803a576 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-11 13:16:21 +0200 + + xzdiff: Add support for .lz files. + + The other scripts don't need changes for .lz support because + in those scripts it is enough that xz supports .lz. + + src/scripts/xzdiff.in | 10 +++++----- + 1 file changed, 5 insertions(+), 5 deletions(-) + +commit d76c752a6d77052e5ad57ade555082585f7ac5d8 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-11 12:23:58 +0200 @@ -265,7 +1200,228 @@ Date: 2022-11-11 12:23:58 +0200 src/scripts/xzmore.in | 4 ++-- 4 files changed, 8 insertions(+), 7 deletions(-) -commit a08be1c420501b1c75770a8f6103e3959486c895 +commit 6552535afd1fe29d726ab6e68cf14ce3624fd48c +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-10 12:34:43 +0200 + + Translations: Rename poa4/fr_FR.po to po4a/fr.po. + + That's how it is preferred at the Translation Project. + On my system /usr/share/man/fr_FR doesn't contain any + other man pages than XZ Utils while /usr/share/man/fr + has quite a few, so this will fix that too. + + Thanks to Benno Schulenberg from the Translation Project. + + po4a/{fr_FR.po => fr.po} | 0 + po4a/po4a.conf | 2 +- + 2 files changed, 1 insertion(+), 1 deletion(-) + +commit 0918159ce4c75bfb60aff0193b559f8a9f41d25a +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-09 18:48:50 +0200 + + xz: Update the man page about BCJ filters, including upcoming --arm64. + + The --arm64 isn't actually implemented yet in the form + described in this commit. + + Thanks to Jia Tan. + + src/xz/xz.1 | 66 +++++++++++++++++++++++++++---------------------------------- + 1 file changed, 29 insertions(+), 37 deletions(-) + +commit ba2ae3596f6be1587495f33b367488f6e00e56f1 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-09 18:14:14 +0200 + + xz: Add --arm64 to --long-help and omit endianness from ARM(-Thumb). + + Modern 32-bit ARM in big endian mode use little endian for + instruction encoding still, so the filters work on such + executables too. It's likely less confusing for users this way. + + The --arm64 option hasn't been implemented yet (there is + --experimental-arm64 but it's different). The --arm64 option + is added now anyway because this is the likely result and the + strings need to be ready for translators. + + Thanks to Jia Tan. + + src/xz/message.c | 5 +++-- + 1 file changed, 3 insertions(+), 2 deletions(-) + +commit 802d57d9215d9c81dbee86edb43c9e93a7f7ec55 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-09 15:12:13 +0200 + + Windows: Update the VS project files for ARM64 and .lz support. + + windows/vs2013/config.h | 9 +++++++++ + windows/vs2013/liblzma.vcxproj | 5 ++++- + windows/vs2013/liblzma_dll.vcxproj | 5 ++++- + windows/vs2017/config.h | 9 +++++++++ + windows/vs2017/liblzma.vcxproj | 3 +++ + windows/vs2017/liblzma_dll.vcxproj | 3 +++ + windows/vs2019/config.h | 9 +++++++++ + windows/vs2019/liblzma.vcxproj | 5 ++++- + windows/vs2019/liblzma_dll.vcxproj | 5 ++++- + 9 files changed, 49 insertions(+), 4 deletions(-) + +commit 5846aeda05972bc803c6094821ae836229ebe691 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-09 14:57:48 +0200 + + DOS: Update Makefile and config.h to include ARM64 and .lz support. + + dos/Makefile | 2 ++ + dos/config.h | 9 +++++++++ + 2 files changed, 11 insertions(+) + +commit 781da8d6c44de6aa278c916375250668a0b107f2 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-09 14:45:05 +0200 + + CMake: Add lzip decoder files and #define to the build. + + CMakeLists.txt | 3 +++ + 1 file changed, 3 insertions(+) + +commit df8ad4af65a9c4846b108550d0083770a69dee64 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-09 14:41:56 +0200 + + Docs: Update INSTALL and also add new prohibited options to PACKAGERS. + + INSTALL | 49 +++++++++++++++++++++++++++++++++++++++++-------- + PACKAGERS | 2 ++ + 2 files changed, 43 insertions(+), 8 deletions(-) + +commit c8ef089c149afaab413c3a51be827dd1d11afe0e +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-20 17:39:06 +0300 + + Tests: Test the .lz files in test_files.sh. + + tests/test_files.sh | 25 +++++++++++++++++++++++++ + 1 file changed, 25 insertions(+) + +commit c8f70ebb4628ceb6cb29cc9195d9deadf69d2bd7 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-20 15:35:59 +0300 + + Tests: Add .lz (lzip) test files. + + tests/files/README | 109 +++++++++++++++++++++++++++++---- + tests/files/bad-1-v0-uncomp-size.lz | Bin 0 -> 42 bytes + tests/files/bad-1-v1-crc32.lz | Bin 0 -> 50 bytes + tests/files/bad-1-v1-dict-1.lz | Bin 0 -> 50 bytes + tests/files/bad-1-v1-dict-2.lz | Bin 0 -> 50 bytes + tests/files/bad-1-v1-magic-1.lz | Bin 0 -> 50 bytes + tests/files/bad-1-v1-magic-2.lz | Bin 0 -> 50 bytes + tests/files/bad-1-v1-member-size.lz | Bin 0 -> 50 bytes + tests/files/bad-1-v1-trailing-magic.lz | Bin 0 -> 54 bytes + tests/files/bad-1-v1-uncomp-size.lz | Bin 0 -> 50 bytes + tests/files/good-1-v0-trailing-1.lz | Bin 0 -> 59 bytes + tests/files/good-1-v0.lz | Bin 0 -> 42 bytes + tests/files/good-1-v1-trailing-1.lz | Bin 0 -> 67 bytes + tests/files/good-1-v1-trailing-2.lz | Bin 0 -> 70 bytes + tests/files/good-1-v1.lz | Bin 0 -> 50 bytes + tests/files/good-2-v0-v1.lz | Bin 0 -> 78 bytes + tests/files/good-2-v1-v0.lz | Bin 0 -> 78 bytes + tests/files/good-2-v1-v1.lz | Bin 0 -> 86 bytes + tests/files/unsupported-1-v234.lz | Bin 0 -> 50 bytes + 19 files changed, 98 insertions(+), 11 deletions(-) + +commit 731db13e6fa3ad3e3fc786c0ccf6eac4cce6865f +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-19 22:32:51 +0300 + + xz: Remove the commented-out FORMAT_GZIP, gzip, .gz, and .tgz. + + src/xz/args.c | 2 -- + src/xz/coder.h | 1 - + src/xz/suffix.c | 9 --------- + 3 files changed, 12 deletions(-) + +commit 3176f992c55b8d788c4633809aaf9447376a5a12 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-08 21:28:15 +0300 + + xz: Add .lz (lzip) decompression support. + + If configured with --disable-lzip-decoder then --long-help will + still list `lzip' in --format but I left it like that since + due to translations it would be messy to have two help strings. + Features are disabled only in special situations so wrong help + in such a situation shouldn't matter much. + + Thanks to Michał Górny for the original patch. + + src/xz/args.c | 9 ++++++++ + src/xz/coder.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++--- + src/xz/coder.h | 3 +++ + src/xz/message.c | 2 +- + src/xz/suffix.c | 26 ++++++++++++++++++---- + src/xz/xz.1 | 46 +++++++++++++++++++++++++++++++++----- + 6 files changed, 141 insertions(+), 13 deletions(-) + +commit 034086e1ae1459210837a24e04878435c86dc41b +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-08 00:29:20 +0300 + + liblzma: Add .lz support to lzma_auto_decoder(). + + Thanks to Michał Górny for the original patch. + + src/liblzma/api/lzma/container.h | 10 ++++++---- + src/liblzma/common/Makefile.inc | 3 ++- + src/liblzma/common/auto_decoder.c | 23 +++++++++++++++++------ + src/liblzma/common/lzip_decoder.h | 22 ++++++++++++++++++++++ + 4 files changed, 47 insertions(+), 11 deletions(-) + +commit 0538db038f3cdc352007dacb42454aa1806b8e40 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-06 15:50:20 +0300 + + liblzma: Add .lz (lzip) decompression support (format versions 0 and 1). + + Support for format version 0 was removed from lzip 1.18 for some + reason. .lz format version 0 files are rare (and old) but some + source packages were released in this format, and some people might + have personal files in this format too. It's very little extra code + to support it along side format version 1 so this commits adds + support for both. + + The Sync Flush marker extentension to the original .lz format + version 1 isn't supported. It would require changes to the + LZMA decoder itself. Such files are very rare anyway. + + See the API doc for lzma_lzip_decoder() for more details about + the .lz format support. + + Thanks to Michał Górny for the original patch. + + configure.ac | 21 ++ + src/liblzma/api/lzma/container.h | 62 +++++- + src/liblzma/common/Makefile.inc | 5 + + src/liblzma/common/lzip_decoder.c | 413 ++++++++++++++++++++++++++++++++++++++ + src/liblzma/liblzma_generic.map | 1 + + src/liblzma/liblzma_linux.map | 1 + + 6 files changed, 501 insertions(+), 2 deletions(-) + +commit 633d48a075b9ce4b9c08a7a56a7eb4cabc18100c +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-09 14:17:23 +0200 + + liblzma: Add the missing Makefile.inc change for --disable-microlzma. + + This was forgotten from commit 59c4d6e1390f6f4176f43ac1dad1f7ac03c449b8. + + src/liblzma/common/Makefile.inc | 12 ++++++++++-- + 1 file changed, 10 insertions(+), 2 deletions(-) + +commit 724285dadbdc88765c8fb83eab9816575a260966 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-09 14:10:52 +0200 @@ -281,7 +1437,7 @@ Date: 2022-11-09 14:10:52 +0200 src/xz/file_io.c | 4 ++++ 2 files changed, 13 insertions(+) -commit 3ee411cd1cd636bdced6ecede4651394bb4f9cb1 +commit f723eec68b0e44234910f669a29119de33018967 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-09 12:48:22 +0200 @@ -296,7 +1452,7 @@ Date: 2022-11-09 12:48:22 +0200 src/xz/coder.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) -commit aa7fa9d960df4bb21f7727bbf7ce0503f3d211b4 +commit 69265d0f223ddf1d66f799b8b047df22923e376f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-09 11:27:20 +0200 @@ -307,7 +1463,16 @@ Date: 2022-11-09 11:27:20 +0200 src/xz/message.c | 3 +++ 1 file changed, 3 insertions(+) -commit ff49ff84a4ba3fab25ffb6d65af321dee36c1273 +commit fe6b8852a3c6a0eb5a3c33512e0a69af257d3bc7 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-08 23:05:37 +0200 + + xz: Make xz -lvv show that the upcoming --arm64 needs 5.4.0 to decompress. + + src/xz/list.c | 20 +++++++++++++++----- + 1 file changed, 15 insertions(+), 5 deletions(-) + +commit fb3f05ac9f2b4b0e3643401960fbeab31997ac7a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-08 22:26:54 +0200 @@ -316,16 +1481,43 @@ Date: 2022-11-08 22:26:54 +0200 doc/faq.txt | 66 ++++++++++++++++++++++++++++++++++++++++--------------------- 1 file changed, 43 insertions(+), 23 deletions(-) -commit 3489565b75b45ee514cf0a6785bb721c7dc624f3 +commit 05331f091ec3b68eccbfb2a9a7a576072768fb4b +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-08 16:57:17 +0200 + + Translations: Update Turkish translation. + + po/tr.po | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit ed3a4822963b4940d84e6f44d47277c394fc046d +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-08 14:55:32 +0200 + + Translations: Update Croatian translation. + + po/hr.po | 190 ++++++++++++++++++++++++++++++++------------------------------- + 1 file changed, 96 insertions(+), 94 deletions(-) + +commit 4746f5ec721316bc4c6fec9905b2902e0360e0af Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-08 14:13:03 +0200 liblzma: Update API docs about decoder flags. - src/liblzma/api/lzma/container.h | 20 ++++++++++++++++---- - 1 file changed, 16 insertions(+), 4 deletions(-) + src/liblzma/api/lzma/container.h | 21 +++++++++++++++++---- + 1 file changed, 17 insertions(+), 4 deletions(-) + +commit 8779a9db5d0cec00c9dc9e9965dd2dda04f9d80d +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-08 14:01:50 +0200 + + liblzma: Use the return_if_error() macro in alone_decoder.c. -commit e4937710805c4b660fc629c15d9a6164b2d97822 + src/liblzma/common/alone_decoder.c | 6 ++---- + 1 file changed, 2 insertions(+), 4 deletions(-) + +commit 3f4990b6822961e75cd9b4e2e82b1df63f6f8fcc Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-11-08 14:00:58 +0200 @@ -334,7 +1526,56 @@ Date: 2022-11-08 14:00:58 +0200 src/liblzma/common/auto_decoder.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -commit d4674dfbb7d1df1feb841f5dbce6ae1f0b026879 +commit 026a5897c72a2041ae08ceec54ce8b1cdeb51334 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-08 13:43:19 +0200 + + xz: Initialize the pledge(2) sandbox at the very beginning of main(). + + It feels better that the initializations are sandboxed too. + They don't do anything that the pledge() call wouldn't allow. + + src/xz/main.c | 27 ++++++++++++++------------- + 1 file changed, 14 insertions(+), 13 deletions(-) + +commit 49a59f6ca001c3ce9affa2c162b437aad021b4d5 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-07 22:51:16 +0200 + + xz: Extend --robot --info-memory output. + + Now it includes everything that the human-readable --info-memory shows. + + src/xz/hardware.c | 24 +++++++++++++++--------- + src/xz/xz.1 | 47 +++++++++++++++++++++++++++++++++++++++++------ + 2 files changed, 56 insertions(+), 15 deletions(-) + +commit 5e2450c75cbac966c62cf2231c824f2cc91ddba8 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-11-07 17:22:04 +0200 + + liblzma: Include cached memory in reported memusage in threaded decoder. + + This affects lzma_memusage() and lzma_memlimit_set() when used + with the threaded decompressor. Now all allocations are reported + by lzma_memusage() (so it's not misleading) and lzma_memlimit_set() + cannot lower the limit below that value. + + The alternative would have been to allow lowering the limit if + doing so is possible by freeing the cached memory but since + the primary use case of lzma_memlimit_set() is to increase + memlimit after LZMA_MEMLIMIT_ERROR this simple approach + was selected. + + The cached memory was always included when enforcing + the memory usage limit while decoding. + + Thanks to Jia Tan. + + src/liblzma/common/stream_decoder_mt.c | 22 +++++++++++++++++++--- + 1 file changed, 19 insertions(+), 3 deletions(-) + +commit 1fc6e7dd1fabdb60124d449b99273330ccab3ff1 Author: Jia Tan <jiat0218@gmail.com> Date: 2022-11-07 16:24:14 +0200 @@ -345,7 +1586,7 @@ Date: 2022-11-07 16:24:14 +0200 src/xz/message.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) -commit 4ed56d32a91ff8ff21c71db4082a54a72ae08b3d +commit cf118c014683069b5dbe91898acdc40f2f0a1f5d Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-31 16:26:05 +0200 @@ -354,7 +1595,7 @@ Date: 2022-10-31 16:26:05 +0200 configure.ac | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) -commit f930638797f1fec05b272367c052d4fc9de35896 +commit aad3c609ffb72f581a7a2b67be3ad70b2b327840 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-31 16:16:37 +0200 @@ -363,7 +1604,7 @@ Date: 2022-10-31 16:16:37 +0200 configure.ac | 4 ---- 1 file changed, 4 deletions(-) -commit 6930f14733757592a4f8d92513f642a5dcc9ec9c +commit e53e0e2186c6b8ce866bd19aec52f1c318ed31ba Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-31 13:31:58 +0200 @@ -380,7 +1621,16 @@ Date: 2022-10-31 13:31:58 +0200 src/common/mythread.h | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) -commit 1c8cbb5be3a97bf22e933777369c4cc28ef9ed5f +commit 48dde3bab9dc04081acb5aa7cf7c5044b8a49f58 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-31 11:54:44 +0200 + + liblzma: Silence -Wconversion warning from crc64_fast.c. + + src/liblzma/check/crc64_fast.c | 5 +++-- + 1 file changed, 3 insertions(+), 2 deletions(-) + +commit a243c617ff249d915ac123de4f536b80322c1fdb Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-31 11:49:47 +0200 @@ -391,7 +1641,74 @@ Date: 2022-10-31 11:49:47 +0200 cmake/tuklib_cpucores.cmake | 5 +++++ 1 file changed, 5 insertions(+) -commit fa9efb729b1827b25ec71aefcf32f3aceeaf9597 +commit 05c72de06fcaaedc78f8abba7d5ec568ddcf1e75 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-27 15:49:18 +0300 + + Tests: test_files.sh: Make it not fail if features were disabled at build. + + It now tries to test as many files as easily possible. + The exit status indicates skipping if any of the files were + skipped. This way it is easy to notice if something is being + skipped when it isn't expected. + + tests/test_files.sh | 50 ++++++++++++++++++++++++++++++++++++++++++++++---- + 1 file changed, 46 insertions(+), 4 deletions(-) + +commit b3459327a51f4b8239d19e6c34b4e0c6bc2d81de +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-27 15:30:13 +0300 + + Tests: test_files.sh: Suppress an expected warning from the log. + + xz (but not xzdec) will normally warn about unsupported check + but since we are testing specifically such a file, it's better + to silence that warning so that it doesn't look suspicious in + test_files.sh.log. + + The use of -q and -Q in xzdec is just for consistency and + doesn't affect the result at least for now. + + tests/test_files.sh | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit 798c86e4231e0835ab76ccd0810c8ea30833b2ce +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-27 15:27:50 +0300 + + Tests: test_files.sh: Print the reason for skipping if xz & xzdec missing. + + tests/test_files.sh | 1 + + 1 file changed, 1 insertion(+) + +commit c1dd8524e1af07f16b790463899de06a6a5fcc08 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-27 01:12:40 +0300 + + Tests: Keep test_compress_* working when some filters are unavailable. + + tests/test_compress.sh | 34 ++++++++++++++++++++-------------- + 1 file changed, 20 insertions(+), 14 deletions(-) + +commit ce30ada91951d0746879ae438da11f1ee8a90aa0 +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-10-23 21:01:08 +0800 + + Tests: test_bcj_exact_size skips properly now if PowerPC filter disabled. + + tests/test_bcj_exact_size.c | 3 +++ + 1 file changed, 3 insertions(+) + +commit 89c5cfcacaca7130509fac836e2f30c46b824502 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-26 00:05:57 +0300 + + Tests: Test also unsupported-*.xz. + + tests/test_files.sh | 37 +++++++++++++++++++++++++++++++++++++ + 1 file changed, 37 insertions(+) + +commit a4b214b93ac741edef9c41e55865b0b867ca2587 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-25 23:45:03 +0300 @@ -405,7 +1722,7 @@ Date: 2022-10-25 23:45:03 +0300 configure.ac | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit b10ba4bf39edd9c9fef42cade3e64fc3b9e3fd19 +commit 04f299b64e73f50afc188c2590ebebc6b73ed744 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-25 23:31:44 +0300 @@ -414,7 +1731,21 @@ Date: 2022-10-25 23:31:44 +0300 m4/ax_pthread.m4 | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) -commit 01744b280c6e35d331b9dc4cc64716b82bc8b3db +commit 59c4d6e1390f6f4176f43ac1dad1f7ac03c449b8 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-25 23:28:34 +0300 + + Build: Add configure option --disable-microlzma. + + MicroLZMA was made for EROFS and used by erofs-utils. + It might be used by something else in the future but + those wanting a smaller build for specific situations + can now disable this rarely-needed feature. + + configure.ac | 22 ++++++++++++++++++++++ + 1 file changed, 22 insertions(+) + +commit 054ccd6d14b2cc6eddc56897af280d3221414150 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-25 23:09:11 +0300 @@ -432,7 +1763,19 @@ Date: 2022-10-25 23:09:11 +0300 src/xz/coder.c | 9 +++++++++ 1 file changed, 9 insertions(+) -commit a3e4606134058241bf1c396243bebcec42c2a780 +commit 563288ea705e83ff5cb292adf794650c263bca1d +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-25 21:11:58 +0300 + + xz: Add support for OpenBSD's pledge() sandbox. + + configure.ac | 12 +++++++++--- + src/xz/file_io.c | 11 +++++++++++ + src/xz/main.c | 13 +++++++++++++ + src/xz/private.h | 2 +- + 4 files changed, 34 insertions(+), 4 deletions(-) + +commit f9913e8ee2ba0b1e4ff4d0aa4c001aae305ed944 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-25 19:07:17 +0300 @@ -465,7 +1808,7 @@ Date: 2022-10-25 19:07:17 +0300 src/xz/coder.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) -commit 0b5e8c7e073bce07c59112af1e8a7495e3576e6c +commit aa4fe145b9486adc454f44fd3e09be9add808a0f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-25 18:36:19 +0300 @@ -474,7 +1817,21 @@ Date: 2022-10-25 18:36:19 +0300 src/xz/xz.1 | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) -commit 23b7416d5ba0e3579265467bd546865d2ea4cef0 +commit 8b46ae8cdeddfd7dc01fec92971b8696e9a96c5d +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-25 18:30:55 +0300 + + xz: Refactor to remove is_empty_filename(). + + Long ago it was used in list.c too but nowadays it's needed + only in io_open_src() so it's nicer to avoid a separate function. + + src/xz/file_io.c | 4 +++- + src/xz/util.c | 12 ------------ + src/xz/util.h | 4 ---- + 3 files changed, 3 insertions(+), 17 deletions(-) + +commit 85624015978b0de294cff3df79006df987c552b1 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-25 18:23:54 +0300 @@ -488,7 +1845,20 @@ Date: 2022-10-25 18:23:54 +0300 src/xz/file_io.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -commit 5daa40454b63c4eb04ab5caeb339eddc95c94bb1 +commit fda9f85f52c546f7ca0313cf89481da4707fecb3 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-10-24 16:25:09 +0300 + + liblzma: Threaded decoder: Stop the worker threads on errors. + + It's waste of CPU time and electricity to leave the unfinished + worker threads running when it is known that their output will + get ignored. + + src/liblzma/common/stream_decoder_mt.c | 33 ++++++++++++++++++++++++++------- + 1 file changed, 26 insertions(+), 7 deletions(-) + +commit 2611c4d90535652d3eb7ef4a026a6691276fab43 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-10-20 20:22:50 +0300 @@ -504,72 +1874,173 @@ Date: 2022-10-20 20:22:50 +0300 src/common/tuklib_cpucores.c | 9 +++++++++ 2 files changed, 14 insertions(+) -commit 0af861050f375678eafc4e1136ca589ae47d31c0 +commit 424ac91c7e0419393ff2bde4f62e21fa611c776d Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-11-11 13:25:02 +0200 +Date: 2022-10-19 19:39:35 +0300 - NEWS: Omit the extra copy of 5.2.5 NEWS. + Tests: Skip tests in test_*.sh if encoders and/or decoders are disabled. + + This isn't perfect as the scripts can still fail if only + certain filters are disabled. This is still an improvement + as now "make check" has better behavior when all encoders + or decoders are disabled. + + Grepping ../config.h is simple and fairly clean but it only + works if config.h was created. CMake builds don't create + config.h but they don't use these test scripts either. - It was a copy-paste error. + Thanks to Sebastian Andrzej Siewior for reporting the problem. + Thanks to Jia Tan for the original patch which grepped xz + error messages instead of config.h. - NEWS | 105 ------------------------------------------------------------------- - 1 file changed, 105 deletions(-) + tests/test_compress.sh | 12 ++++++++++++ + tests/test_files.sh | 11 +++++++++++ + tests/test_scripts.sh | 11 +++++++++++ + 3 files changed, 34 insertions(+) -commit f0c6a66701afed12c5926c0499b150ecd5b0a63c +commit ca8bf9d7c5a30be8ba1eeb106fd892f19e83ed09 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-11-10 12:34:43 +0200 +Date: 2022-10-19 18:54:34 +0300 - Translations: Rename poa4/fr_FR.po to po4a/fr.po. + Test: Remove the (exit 1) lines. - That's how it is preferred at the Translation Project. - On my system /usr/share/man/fr_FR doesn't contain any - other man pages than XZ Utils while /usr/share/man/fr - has quite a few, so this will fix that too. - - Thanks to Benno Schulenberg from the Translation Project. + I suspect that I used these in the original version because + Autoconf's manual describes that such a trick is needed in + some specific situations for portability reasons. None of those + situations listed on Autoconf 2.71 manual apply to these test + scripts though so this cleans them up. - po4a/{fr_FR.po => fr.po} | 0 - po4a/po4a.conf | 2 +- - 2 files changed, 1 insertion(+), 1 deletion(-) + tests/test_compress.sh | 10 ---------- + tests/test_files.sh | 9 --------- + tests/test_scripts.sh | 6 ------ + 3 files changed, 25 deletions(-) -commit 6bf8b1f870870a1a0c3f555c34decc9ffb9f8d7a +commit 82fcb7cfc17ce62f79ebc7ca2374e1daca5e4d5e Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-11-08 16:57:17 +0200 +Date: 2022-10-19 17:14:57 +0300 - Translations: Update Turkish translation. + Tests: Fix a warning in test_memlimit.c when decoders are disabled. - po/tr.po | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) + tests/test_memlimit.c | 3 +++ + 1 file changed, 3 insertions(+) -commit 9f8e9d3c8189d8c6f587c99b0da1cf83f72330c2 +commit b5f8271b45b9b0e59485ffba3640ca3418835ec4 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-11-08 14:55:32 +0200 +Date: 2022-10-19 17:11:46 +0300 - Translations: Update Croatian translation. + Tests: Add test_memlimit to .gitignore. + + Thanks to Jia Tan. - po/hr.po | 190 ++++++++++++++++++++++++++++++++------------------------------- - 1 file changed, 96 insertions(+), 94 deletions(-) + .gitignore | 1 + + 1 file changed, 1 insertion(+) + +commit 6a86e81cab202d0a812a7b2e9efacaf70c58ba38 +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-10-06 21:53:09 +0300 + + Tests: Refactor test_stream_flags.c. + + Converts test_stream_flags to tuktest. Also the test will now + compile and skip properly if encoders or decoders are disabled. + + Thanks to Sebastian Andrzej Siewior. + + tests/test_stream_flags.c | 533 ++++++++++++++++++++++++++++++++++++---------- + 1 file changed, 416 insertions(+), 117 deletions(-) + +commit 827ac5b4821491fd3afe0d0e1ddac326253aeb66 +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-10-06 17:00:38 +0800 + + Tests: Refactor test_block_header.c. + + test_block_header now achieves higher test coverage. Also the + test will now compile and skip properly if encoders or decoders + are disabled. + + Thanks to Sebastian Andrzej Siewior. + + tests/test_block_header.c | 486 +++++++++++++++++++++++++++++++++++----------- + 1 file changed, 370 insertions(+), 116 deletions(-) -commit d24a57b7fc7e5e9267b84367cb0788d3acf7f569 +commit 84963318952064a93bfc52edd6b0ef70593384ee +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-10-05 23:54:12 +0800 + + Tests: Fix compilation issues. + + test_bcj_exact_size, test_check, test_hardware, and test_index will + all now compile and skip properly if encoders or decoders are disabled. + + Also fixed a small typo (disabed -> disabled). + + Thanks to Sebastian Andrzej Siewior. + + tests/test_bcj_exact_size.c | 20 ++++++++++++++------ + tests/test_check.c | 8 +++++++- + tests/test_hardware.c | 2 +- + tests/test_index.c | 6 ++++++ + tests/test_memlimit.c | 16 +++++++++++++++- + tests/test_vli.c | 13 +++++++++++++ + 6 files changed, 56 insertions(+), 9 deletions(-) + +commit 7dcabeec63d46b436fa5f043c3d1f09d0e15be16 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-09-30 16:41:03 +0300 +Date: 2022-10-05 16:20:47 +0300 - Bump version and soname for 5.2.7. + Tests: Include mythread.h in the tests that use MYTHREAD_ENABLED. - src/liblzma/Makefile.am | 2 +- - src/liblzma/api/lzma/version.h | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) + tests/test_check.c | 1 + + tests/test_hardware.c | 1 + + tests/test_memlimit.c | 1 + + 3 files changed, 3 insertions(+) + +commit 14af758a770c7781af18fb66d6d21ee5b1c27f04 +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-10-05 20:57:16 +0800 + + liblzma: Fix a compilation issue when encoders are disabled. + + When encoders were disabled and threading enabled, outqueue.c and + outqueue.h were not compiled. The multi threaded decoder required + these files, so compilation failed. + + src/liblzma/common/Makefile.inc | 7 ++++--- + 1 file changed, 4 insertions(+), 3 deletions(-) -commit d2003362dd42004355a5550ee7a60ace3b3ea2d8 +commit 6ca5c354bd4620aa7f81da68870eef1b1f26288f +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-10-05 16:41:38 +0800 + + Tests: Fix compilation error when threading support has been disabled. + + Now tests that require threading are skipped when threading + support has been disabled. + + Thanks to Sebastian Andrzej Siewior. + + tests/test_check.c | 4 ++++ + tests/test_hardware.c | 4 ++++ + tests/test_memlimit.c | 4 ++++ + 3 files changed, 12 insertions(+) + +commit fae37ad2affd8fe8871f4ff93d5cab5ec14d5e58 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-09-30 16:40:39 +0300 +Date: 2022-10-05 14:26:00 +0300 - Add NEWS for 5.2.7. + tuklib_integer: Add 64-bit endianness-converting reads and writes. + + Also update the comment in liblzma's memcmplen.h. + + Thanks to Michał Górny for the original patch for the reads. - NEWS | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 1 file changed, 74 insertions(+) + m4/tuklib_integer.m4 | 8 ++++---- + src/common/tuklib_integer.h | 46 ++++++++++++++++++++++++++++++++++++++++-- + src/liblzma/common/memcmplen.h | 9 +++------ + 3 files changed, 51 insertions(+), 12 deletions(-) -commit 369afb51991e91a5c3a23c4ae3872329a403eb31 +commit 508a44372c5b0dede8863fd0d358d4a9d8645c95 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-09-30 12:06:13 +0300 @@ -580,7 +2051,7 @@ Date: 2022-09-30 12:06:13 +0300 src/liblzma/api/lzma/base.h | 11 +++++++++++ 1 file changed, 11 insertions(+) -commit 166431e9951613481b8136f52b570d7bc30b5a03 +commit 8cc9874a7974cd575aee44f218836f7acdbeb0ed Author: Jia Tan <jiat0218@gmail.com> Date: 2022-09-21 16:15:50 +0800 @@ -595,7 +2066,7 @@ Date: 2022-09-21 16:15:50 +0800 src/liblzma/common/index.c | 3 +++ 1 file changed, 3 insertions(+) -commit 5e53a6c28b87a0bb9b0cfeaf708cc99ef8e9bc66 +commit afd5a8bf5374eba82804a999e1ea7af680784086 Author: Jia Tan <jiat0218@gmail.com> Date: 2022-09-21 20:29:28 +0800 @@ -604,7 +2075,7 @@ Date: 2022-09-21 20:29:28 +0800 tests/test_index.c | 43 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) -commit 4ed5fd54c676b735e3f58158f214af68755d59b5 +commit 3d5a99ca373a4e86faf671226ca6487febb9eeac Author: Jia Tan <jiat0218@gmail.com> Date: 2022-09-21 19:28:53 +0800 @@ -630,7 +2101,7 @@ Date: 2022-09-21 19:28:53 +0800 src/liblzma/common/index.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) -commit c4476f695291a2d7376d5fa406d32e354f858858 +commit a61d32172789735350a941e23baf6b587c49e5d9 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-09-28 12:20:41 +0300 @@ -641,7 +2112,17 @@ Date: 2022-09-28 12:20:41 +0300 src/common/tuklib_physmem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 976f897bbba9890a99f01a8d59c6d59c230142e6 +commit 5a4d3548ab214fdca364d5c734baf1d1fab47308 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-09-28 11:12:07 +0300 + + Tests: Add test_memlimit.c to test restarting after LZMA_MEMLIMIT_ERROR. + + tests/Makefile.am | 2 + + tests/test_memlimit.c | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 153 insertions(+) + +commit 660739f99ab211edec4071de98889fb32ed04e98 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-09-28 11:05:15 +0300 @@ -656,7 +2137,7 @@ Date: 2022-09-28 11:05:15 +0300 src/liblzma/common/stream_decoder.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) -commit 2caa9580e5f7c5cc6668e9427a2c21f3f63bb684 +commit 7e68fda58c74ad9e5b876cc22fcbe80fc0e4747b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-09-28 11:00:23 +0300 @@ -665,7 +2146,114 @@ Date: 2022-09-28 11:00:23 +0300 src/liblzma/common/stream_decoder.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) -commit 51882fec5b96cf7760389b670d86c1e7a5a0ced2 +commit f664cb25841fc1c478b819034a224a558e2ac6e7 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-09-20 16:58:22 +0300 + + liblzma: ARM64: Add comments. + + src/liblzma/simple/arm64.c | 13 +++++++++++++ + 1 file changed, 13 insertions(+) + +commit b557b4a0eea05470fae8ba5ef4ad5a6dfb36ac41 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-09-20 16:27:50 +0300 + + liblzma: ARM64: Fix wrong comment in API doc. + + Thanks to Jia Tan. + + src/liblzma/api/lzma/bcj.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit d5b0906fa55157f48c200188a3951d80df9cb308 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-09-19 20:24:26 +0300 + + xz: Add --experimental-arm64[=width=WIDTH]. + + It will be renamed to --arm64 once it is stable. + + Man page or --long-help weren't updated yet. + + src/xz/args.c | 7 +++++++ + src/xz/message.c | 7 +++++++ + src/xz/options.c | 39 +++++++++++++++++++++++++++++++++++++++ + src/xz/options.h | 7 +++++++ + 4 files changed, 60 insertions(+) + +commit ecb966de308c255bb4735a7307ef9901c643a9de +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-09-19 19:34:56 +0300 + + liblzma: Add experimental ARM64 BCJ filter with a temporary Filter ID. + + That is, the Filter ID will be changed once the design is final. + The current version will be removed. So files created with the + tempoary Filter ID won't be supported in the future. + + CMakeLists.txt | 3 + + configure.ac | 4 +- + src/liblzma/api/lzma/bcj.h | 35 +++++- + src/liblzma/common/filter_common.c | 9 ++ + src/liblzma/common/filter_decoder.c | 8 ++ + src/liblzma/common/filter_encoder.c | 11 ++ + src/liblzma/simple/Makefile.inc | 4 + + src/liblzma/simple/arm64.c | 227 ++++++++++++++++++++++++++++++++++++ + src/liblzma/simple/simple_coder.h | 9 ++ + src/liblzma/simple/simple_decoder.h | 4 + + src/liblzma/simple/simple_encoder.h | 2 + + 11 files changed, 313 insertions(+), 3 deletions(-) + +commit 177bdc922cb17bd0fd831ab8139dfae912a5c2b8 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-09-17 22:42:18 +0300 + + liblzma: Simple/BCJ filters: Allow disabling generic BCJ options. + + This will be needed for the ARM64 BCJ filter as it will use + its own options struct. + + src/liblzma/simple/arm.c | 2 +- + src/liblzma/simple/armthumb.c | 2 +- + src/liblzma/simple/ia64.c | 2 +- + src/liblzma/simple/powerpc.c | 2 +- + src/liblzma/simple/simple_coder.c | 4 ++-- + src/liblzma/simple/simple_private.h | 2 +- + src/liblzma/simple/sparc.c | 2 +- + src/liblzma/simple/x86.c | 3 ++- + 8 files changed, 10 insertions(+), 9 deletions(-) + +commit c3592d0a55114144686ecf960cb516d6b31c98e9 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-09-16 17:08:53 +0300 + + Tests: Add a test file for lzma_index_append() integer overflow bug. + + This test fails before commit 18d7facd3802b55c287581405c4d49c98708c136. + + test_files.sh now runs xz -l for bad-3-index-uncomp-overflow.xz + because only then the previously-buggy code path gets tested. + Normal decompression doesn't use lzma_index_append() at all. + Instead, lzma_index_hash functions are used and those already + did the overflow check. + + tests/files/README | 10 ++++++++++ + tests/files/bad-3-index-uncomp-overflow.xz | Bin 0 -> 132 bytes + tests/test_files.sh | 8 ++++++++ + 3 files changed, 18 insertions(+) + +commit 982b29f828079a2a26253a40e975127a40a7d2bd +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-09-16 15:10:07 +0300 + + Translations: Add Turkish translation. + + po/LINGUAS | 1 + + po/tr.po | 977 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 978 insertions(+) + +commit 1fc088d9f6d1697924aaeac8cd1fb9918d1532e2 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-09-16 14:09:07 +0300 @@ -674,7 +2262,7 @@ Date: 2022-09-16 14:09:07 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 974186f7cd839f1dbf937b98e112dff670d2bd5c +commit 097c7b67ce86ff16a7cef7631b39e5ca4ee3d420 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-09-16 14:07:03 +0300 @@ -709,7 +2297,24 @@ Date: 2022-09-16 14:07:03 +0300 src/scripts/xzgrep.in | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) -commit f94da15120c3d3c363ca12c2262ac6cb9f321f4f +commit f2d084fe3f0d6d71488bfc6987f26542f67bfd99 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-09-09 14:12:30 +0300 + + Tests: Silence warnings about unused functions from tuktest.h. + + Warnings about unused tuktest_run_test conveniently tell which + test programs haven't been converted to tuktest.h yet but I + silenced that warning too for now anyway. + + It is fine to use __attribute__((__unused__)) even when the + function is actually used because the attribute only means + that the function might be unused. + + tests/tuktest.h | 10 ++++++++++ + 1 file changed, 10 insertions(+) + +commit f8ee61e74eb40600445fdb601c374d582e1e9c8a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-09-09 13:51:57 +0300 @@ -733,26 +2338,7 @@ Date: 2022-09-09 13:51:57 +0300 src/liblzma/common/filter_common.c | 18 ++++++++++++------ 2 files changed, 15 insertions(+), 7 deletions(-) -commit ea57b9aa2c3e1cdb667f8dd698314b1c36047018 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-09-16 17:08:53 +0300 - - Tests: Add a test file for lzma_index_append() integer overflow bug. - - This test fails before commit 18d7facd3802b55c287581405c4d49c98708c136. - - test_files.sh now runs xz -l for bad-3-index-uncomp-overflow.xz - because only then the previously-buggy code path gets tested. - Normal decompression doesn't use lzma_index_append() at all. - Instead, lzma_index_hash functions are used and those already - did the overflow check. - - tests/files/README | 10 ++++++++++ - tests/files/bad-3-index-uncomp-overflow.xz | Bin 0 -> 132 bytes - tests/test_files.sh | 8 ++++++++ - 3 files changed, 18 insertions(+) - -commit 72e1645a439a999638a63ec9abb7210671ea5415 +commit 18d7facd3802b55c287581405c4d49c98708c136 Author: Jia Tan <jiat0218@gmail.com> Date: 2022-09-02 20:18:55 +0800 @@ -767,7 +2353,7 @@ Date: 2022-09-02 20:18:55 +0800 src/liblzma/common/index.c | 4 ++++ 1 file changed, 4 insertions(+) -commit 20d82bc90781aa9d2373efe872d90ba64dc7dc62 +commit 9ac06cb5b85274d18f9f70d82cf2d8c9c1151bd4 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-09-08 15:11:08 +0300 @@ -776,7 +2362,31 @@ Date: 2022-09-08 15:11:08 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 31d80c6b261b24220776dfaeb8a04f80f80e0a24 +commit ba3e4ba2de034ae93a513f9c3a0823b80cdb66dc +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-09-08 15:07:00 +0300 + + CMake: Clarify a comment about Windows symlinks without file extension. + + CMakeLists.txt | 7 +++---- + 1 file changed, 3 insertions(+), 4 deletions(-) + +commit 17485e884ce5c74315f29a8a1507bc706cd5cd1d +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-09-08 15:02:41 +0300 + + CMake: Update for liblzma_*.map files and fix wrong common_w32res.rc dep. + + The previous commit split liblzma.map into liblzma_linux.map and + liblzma_generic.map. This commit updates the CMake build for those. + + common_w32res.rc dependency was listed under Linux/FreeBSD while + obviously it belongs to Windows when building a DLL. + + CMakeLists.txt | 24 +++++++++++++++++++----- + 1 file changed, 19 insertions(+), 5 deletions(-) + +commit 913ddc5572b9455fa0cf299be2e35c708840e922 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-09-04 23:23:00 +0300 @@ -897,43 +2507,19 @@ Date: 2022-09-04 23:23:00 +0300 Thanks to Boud Roukema for reporting the problem and discussing the details and testing the fix. - configure.ac | 23 ++++- + configure.ac | 23 +++- src/liblzma/Makefile.am | 10 +- src/liblzma/common/block_buffer_encoder.c | 18 ++++ src/liblzma/common/common.c | 14 +++ - src/liblzma/common/common.h | 28 ++++++ + src/liblzma/common/common.h | 28 +++++ src/liblzma/common/hardware_cputhreads.c | 12 +++ src/liblzma/common/stream_encoder_mt.c | 42 ++++++++ src/liblzma/{liblzma.map => liblzma_generic.map} | 6 +- - src/liblzma/liblzma_linux.map | 123 +++++++++++++++++++++++ - src/liblzma/validate_map.sh | 113 +++++++++++++++++++-- - 10 files changed, 374 insertions(+), 15 deletions(-) - -commit e7a7ac744eb0f890ef52388de838596ef566c73f -Author: Jia Tan <jiat0218@gmail.com> -Date: 2022-09-08 15:07:00 +0300 - - CMake: Clarify a comment about Windows symlinks without file extension. - - CMakeLists.txt | 7 +++---- - 1 file changed, 3 insertions(+), 4 deletions(-) - -commit a273a0cb77c6000e4da627b07f28c5af41f0908c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-09-08 15:02:41 +0300 - - CMake: Update for liblzma_*.map files and fix wrong common_w32res.rc dep. - - The previous commit split liblzma.map into liblzma_linux.map and - liblzma_generic.map. This commit updates the CMake build for those. - - common_w32res.rc dependency was listed under Linux/FreeBSD while - obviously it belongs to Windows when building a DLL. - - CMakeLists.txt | 24 +++++++++++++++++++----- - 1 file changed, 19 insertions(+), 5 deletions(-) + src/liblzma/liblzma_linux.map | 131 +++++++++++++++++++++++ + src/liblzma/validate_map.sh | 113 +++++++++++++++++-- + 10 files changed, 382 insertions(+), 15 deletions(-) -commit 5875a45be0ab399e37acc31165725a9b5291f67e +commit 80a1a8bb838842a2be343bd88ad1462c21c5e2c9 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-08-31 16:42:04 +0300 @@ -948,7 +2534,7 @@ Date: 2022-08-31 16:42:04 +0300 CMakeLists.txt | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) -commit 3523b6ebb56c447f21736f01bfe91602741aa3e7 +commit a4193bb6d85d7765b1b87faeab3e50106d3ab2e9 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-08-31 16:29:38 +0300 @@ -959,17 +2545,293 @@ Date: 2022-08-31 16:29:38 +0300 CMakeLists.txt | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) -commit 5af9e8759f748a11a408bfc5aef829df3811fc23 +commit c1555b1a221a1427d4d650647531016d754bc4da Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-09-16 15:10:07 +0300 +Date: 2022-08-22 18:16:40 +0300 - Translations: Add Turkish translation. + Bump version number for 5.3.3alpha. + + src/liblzma/api/lzma/version.h | 2 +- + src/liblzma/liblzma.map | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) + +commit 44fedddc11c3f6ec2f7fe35a3e38f15ca93f90eb +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-08-22 18:13:56 +0300 + + Add NEWS for 5.3.3alpha. + + NEWS | 119 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 119 insertions(+) + +commit a93e235d7ca764cc19f8f9f9063b40ff361c3cfa +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-07-12 17:59:41 +0300 + + Translations: Add Portuguese translation. + + Jia Tan made white-space changes and also changed "Language: pt_BR\n" + to pt. The translator wasn't reached so I'm hoping these changes + are OK and will commit it without translator's approval. + + Thanks to Pedro Albuquerque and Jia Tan. + + po/LINGUAS | 1 + + po/pt.po | 1001 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 1002 insertions(+) + +commit e7cf5a946f25e40d77f45e41f0dee7d42a04e9ae +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-07-10 21:16:40 +0300 + + Translations: Add Serbian translation. + + Quite a few white-space changes were made by Jia Tan to make + this look good. Contacting the translator didn't succeed so + I'm committing this without getting translator's approval. + + Thanks to Мирослав Николић (Miroslav Nikolic) and Jia Tan. po/LINGUAS | 1 + - po/tr.po | 977 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 978 insertions(+) + po/sr.po | 987 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 988 insertions(+) + +commit cc8617ab53b1f2a4da212fa76c92fe538269c5de +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-07-04 23:51:36 +0300 + + Translations: Add Swedish translation. + + Thanks to Sebastian Rasmussen and Jia Tan. + + po/LINGUAS | 1 + + po/sv.po | 983 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 984 insertions(+) + +commit c613598c358b640682d0ca2aed38fa7df763e8c7 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-07-04 23:40:27 +0300 + + Translations: Add Esperanto translation. + + Thanks to Keith Bowes and Jia Tan. + + po/LINGUAS | 1 + + po/eo.po | 984 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 985 insertions(+) + +commit 659a587d678f21e98e91d2751c31d4ce050c081a +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-07-01 00:22:33 +0300 + + Translations: Add Catalan translation. + + Thanks to Jordi Mas and Jia Tan. + + po/LINGUAS | 1 + + po/ca.po | 1076 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 1077 insertions(+) + +commit 739fd8c9bdf1d85f57d56642aad87148d4779530 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-30 17:47:08 +0300 + + Translations: Add Ukrainian translation. + + Thanks to Yuri Chornoivan and Jia Tan. + + po/LINGUAS | 1 + + po/uk.po | 996 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 997 insertions(+) + +commit 73280550b111930c62a667e56add8fd574f80bc8 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-30 17:45:26 +0300 + + Translators: Add Romanian translation. + + Thanks to Remus-Gabriel Chelu and Jia Tan. + + po/LINGUAS | 1 + + po/ro.po | 1016 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 1017 insertions(+) + +commit 2465f5b825152714b2c357d96c2422c31109d320 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-29 18:33:32 +0300 + + Translations: Update Brazilian Portuguese translation. + + One msgstr was changed. The diff is long due to changes + in the source code line numbers in the comments. + + Thanks to Rafael Fontenelle. + + po/pt_BR.po | 186 ++++++++++++++++++++++++++++++------------------------------ + 1 file changed, 92 insertions(+), 94 deletions(-) + +commit 434e1ffd3e62719d509da10b646216f5ef53fe4d +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-29 18:04:44 +0300 + + Translations: Add Croatian translation. + + Thanks to Božidar Putanec and Jia Tan. + + po/LINGUAS | 1 + + po/hr.po | 987 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 988 insertions(+) + +commit 0732d0f7065c9bd48bfe4f5983144ae970c4a499 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-29 17:58:48 +0300 + + Translations: Add Spanish translation. + + Thanks to Cristian Othón Martínez Vera and Jia Tan. + + po/LINGUAS | 1 + + po/es.po | 984 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 985 insertions(+) + +commit 9899b0f82bc130998d1f1f618a6ab805b73f2696 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-29 17:49:43 +0300 + + Translations: Add Korean translation. + + Thanks to Seong-ho Cho and Jia Tan. + + po/LINGUAS | 1 + + po/ko.po | 972 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 973 insertions(+) + +commit 65217eaf6bd195f3ef027d4ac55d57b7d133d69f +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2020-03-16 21:57:21 +0200 + + Translations: Rebuild cs.po to avoid incorrect fuzzy strings. + + "make dist" updates the .po files and the fuzzy strings would + result in multiple very wrong translations. + + po/cs.po | 592 ++++++++++++++++++++++++++++++++++----------------------------- + 1 file changed, 322 insertions(+), 270 deletions(-) + +commit e56ff423ee7af9e648e16b132f0d835d2cb4db26 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2020-03-16 17:30:39 +0200 + + Translations: Add partial Danish translation. + + I made a few minor white space changes without getting them + approved by the Danish translation team. + + po/LINGUAS | 1 + + po/da.po | 896 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 897 insertions(+) + +commit 43e09c62e77cb8807d932c81de4decbdb956e184 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2020-03-11 22:37:54 +0200 + + Translations: Add hu, zh_CN, and zh_TW. + + I made a few white space changes to these without getting them + approved by the translation teams. (I tried to contact the hu and + zh_TW teams but didn't succeed. I didn't contact the zh_CN team.) + + po/LINGUAS | 3 + + po/hu.po | 985 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + po/zh_CN.po | 963 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + po/zh_TW.po | 956 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 4 files changed, 2907 insertions(+) + +commit 982b6b198ae1ffe6093236dd8a3d03d1415b912b +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2020-03-11 14:33:30 +0200 + + Translations: Update vi.po to match the file from the TP. + + The translated strings haven't been updated but word wrapping + is different. + + po/vi.po | 407 ++++++++++++++++++++++++++++----------------------------------- + 1 file changed, 179 insertions(+), 228 deletions(-) + +commit 801f39691fc4abc6dd33d7653d498781b593f3eb +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2020-03-11 14:18:03 +0200 + + Translations: Add fi and pt_BR, and update de, fr, it, and pl. + + The German translation isn't identical to the file in + the Translation Project but the changes (white space changes + only) were approved by the translator Mario Blättermann. + + po/LINGUAS | 2 + + po/de.po | 476 ++++++++++++++-------------- + po/fi.po | 974 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + po/fr.po | 272 ++++++++-------- + po/it.po | 479 ++++++++++++---------------- + po/pl.po | 239 +++++++------- + po/pt_BR.po | 1001 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 7 files changed, 2697 insertions(+), 746 deletions(-) + +commit 311e4f85ede5d2f0bb71f3ad70b5b7db1b5adf33 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-08-22 17:27:19 +0300 + + xz: Try to clarify --memlimit-mt-decompress vs. --memlimit-compress. + + src/xz/xz.1 | 31 +++++++++++++++++++------------ + 1 file changed, 19 insertions(+), 12 deletions(-) + +commit df23c31000283c00e5ef1ca32a0bc3bb757bd707 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-08-22 16:46:18 +0300 + + CMake: Add liblzma tests. + + Thanks to Jia Tan for the patch. + + CMakeLists.txt | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++++- + 1 file changed, 52 insertions(+), 1 deletion(-) + +commit 02a777f9c422d3b5ec895078530bb1a2b6f7bdf5 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-08-19 23:32:22 +0300 + + xz: Revise --info-memory output. + + The strings could be more descriptive but it's good + to have some version of this committed now. + + --robot mode wasn't changed yet. + + src/xz/hardware.c | 32 ++++++++++++++++++++++++++------ + src/xz/xz.1 | 1 + + 2 files changed, 27 insertions(+), 6 deletions(-) + +commit f864f6d42eab57ea8ed82cc2dd19a03b51377442 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-08-19 23:12:02 +0300 + + xz: Update the man page for threaded decompression and memlimits. + + This documents the changes made in commits + 6c6da57ae2aa962aabde6892442227063d87e88c, + cad299008cf73ec566f0662a9cf2b94f86a99659, and + 898faa97287a756231c663a3ed5165672b417207. + + The --info-memory bit hasn't been finished yet + even though it's already mentioned in this commit + under --memlimit-mt-decompress and --threads. -commit f05a69685e9c7f7157d751abfbd2d8ea6e3bacc7 + src/xz/xz.1 | 148 +++++++++++++++++++++++++++++++++++++++++++++++++----------- + 1 file changed, 121 insertions(+), 27 deletions(-) + +commit d13bfcc0056617dd648f655a01653932fad7067f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-08-18 17:49:16 +0300 @@ -984,7 +2846,7 @@ Date: 2022-08-18 17:49:16 +0300 Makefile.am | 2 ++ 1 file changed, 2 insertions(+) -commit ad5ef6d3c30bb749113c51aa0bbe8bcb5e193fbe +commit e66787bcfefdb93f19c974f895f65969a77937b0 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-08-18 17:38:05 +0300 @@ -1006,26 +2868,32 @@ Date: 2022-08-18 17:38:05 +0300 windows/vs2019/liblzma_dll.vcxproj | 6 ++++++ 3 files changed, 18 insertions(+) -commit 8dfed05bdaa4873833ba24279f02ad2db25effea +commit c4e8e5fb311225b8b48d34157891a640b2535e0c Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-08-12 14:30:13 +0300 +Date: 2022-08-18 17:16:49 +0300 - Bump version and soname for 5.2.6. + liblzma: Threaded decoder: Improve LZMA_FAIL_FAST when LZMA_FINISH is used. + + It will now return LZMA_DATA_ERROR (not LZMA_OK or LZMA_BUF_ERROR) + if LZMA_FINISH is used and there isn't enough input to finish + decoding the Block Header or the Block. The use of LZMA_DATA_ERROR + is simpler and the less risky than LZMA_BUF_ERROR but this might + be changed before 5.4.0. - src/liblzma/Makefile.am | 2 +- - src/liblzma/api/lzma/version.h | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) + src/liblzma/api/lzma/container.h | 6 +++++ + src/liblzma/common/stream_decoder_mt.c | 42 ++++++++++++++++++++++++++++++++++ + 2 files changed, 48 insertions(+) -commit 09b4af4e04def5777b06124200b64b0b941eea7f +commit 6dcf606e7efa2b259f0262f9e2f61e00116842d3 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-08-12 14:29:28 +0300 +Date: 2022-08-12 18:31:47 +0300 Add NEWS for 5.2.6. - NEWS | 226 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 1 file changed, 226 insertions(+) + NEWS | 121 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 121 insertions(+) -commit 692de534fae6c38f92416793addcf5f60ffe2bf6 +commit 413b86fcf8934fae5a004f378a9483d37d8fcaab Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-08-12 14:28:41 +0300 @@ -1034,7 +2902,18 @@ Date: 2022-08-12 14:28:41 +0300 AUTHORS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 275de376a645f406be7c453bb68427aa9466cf83 +commit 352672732c346c546ff3c26d0605bc0ed1c8b7c7 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-07-25 19:28:26 +0300 + + Build: Start the generated ChangeLog from around 5.2.0 instead of 5.0.0. + + This makes ChangeLog smaller. + + Makefile.am | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 6f6d11225d6016be2bbb55d845b66f6b04d048df Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-25 19:11:05 +0300 @@ -1047,7 +2926,7 @@ Date: 2022-07-25 19:11:05 +0300 po4a/update-po | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 76a5a752b8467ff591dd028deb61e9bf2c274c7e +commit 61f8ec804abdb4c5dac01e8ae9b90c7be58a5c24 Author: Jia Tan <jiat0218@gmail.com> Date: 2022-07-25 18:30:05 +0300 @@ -1056,7 +2935,7 @@ Date: 2022-07-25 18:30:05 +0300 src/liblzma/lz/lz_encoder.c | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-) -commit 749b86c2c18ab61a07f19ec8fefc67325da97397 +commit 4d80b463a1251aa22eabc87d2732fec13b1adda6 Author: Jia Tan <jiat0218@gmail.com> Date: 2022-07-25 18:20:01 +0300 @@ -1068,28 +2947,34 @@ Date: 2022-07-25 18:20:01 +0300 configure.ac | 4 ++++ 1 file changed, 4 insertions(+) -commit 63e3cdef805a4d9ea94c76c36cbabe4dc76abe36 +commit 9cc721af5436908f2d5a828aebbc4050a32a3487 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2021-01-11 23:41:16 +0200 +Date: 2022-07-24 13:27:48 +0300 - xz: Make --keep accept symlinks, hardlinks, and setuid/setgid/sticky. - - Previously this required using --force but that has other - effects too which might be undesirable. Changing the behavior - of --keep has a small risk of breaking existing scripts but - since this is a fairly special corner case I expect the - likehood of breakage to be low enough. - - I think the new behavior is more logical. The only reason for - the old behavior was to be consistent with gzip and bzip2. - - Thanks to Vincent Lefevre and Sebastian Andrzej Siewior. + xz: Update the man page that change to --keep will be in 5.2.6. - src/xz/file_io.c | 9 +++++---- - src/xz/xz.1 | 16 +++++++++++++++- - 2 files changed, 20 insertions(+), 5 deletions(-) + src/xz/xz.1 | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit b81bf0c7d1873e52a4086a9abb494471d652cb55 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-07-19 23:23:54 +0300 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) -commit 9055584be0ae47032b7fb3ba8531aa56b9ed572a +commit 340cf1ec3927767046b8293a49da3db4e393f426 +Author: Nicholas Jackson <nickajacks1@gmail.com> +Date: 2022-07-17 17:39:23 -0700 + + CMake: Add missing source file to liblzma build + + CMakeLists.txt | 1 + + 1 file changed, 1 insertion(+) + +commit d796b6d7fdb8b7238b277056cf9146cce25db604 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-19 23:19:49 +0300 @@ -1098,7 +2983,7 @@ Date: 2022-07-19 23:19:49 +0300 src/scripts/xzgrep.1 | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) -commit 57e1ccbb7ce30e5298ea7ec3934e462fa8d32237 +commit 923bf96b55e5216a6c8df9d8331934f54784390e Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-19 23:13:24 +0300 @@ -1131,7 +3016,7 @@ Date: 2022-07-19 23:13:24 +0300 src/scripts/xzgrep.in | 72 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 53 insertions(+), 19 deletions(-) -commit 6351ea1afb261ec9be95f808bcbe724e0da4ae09 +commit a648978b20495b7aa4a8b029c5a810b5ad9d08ff Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-19 00:10:55 +0300 @@ -1145,7 +3030,7 @@ Date: 2022-07-19 00:10:55 +0300 src/scripts/xzgrep.in | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) -commit 2c1ff2ed6b9f17b3913866a15d776ebbbeb824d2 +commit bd7b290f3fe4faeceb7d3497ed9bf2e6ed5e7dc5 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-18 21:52:31 +0300 @@ -1166,7 +3051,7 @@ Date: 2022-07-18 21:52:31 +0300 src/scripts/xzgrep.in | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) -commit 8b0be38a795fa1cdffdada64341a34209e630060 +commit b56729af9f1a596e57aeefd7570d8d7dce5c9f52 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-18 21:10:25 +0300 @@ -1178,7 +3063,7 @@ Date: 2022-07-18 21:10:25 +0300 src/scripts/xzgrep.in | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) -commit 4a61867a874c60b31448d2d4f3c276efba85a243 +commit bad61b5997e6647911974022bfb72f3d4818a594 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-18 19:18:48 +0300 @@ -1193,7 +3078,7 @@ Date: 2022-07-18 19:18:48 +0300 src/scripts/xzmore.in | 8 ++++---- 4 files changed, 11 insertions(+), 11 deletions(-) -commit 0e222bf7d79a9792025c5ffeaa66c8e14eb47615 +commit 6a4a4a7d2667837dc824c26fcb19ed6ca5aff645 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-17 21:36:25 +0300 @@ -1219,7 +3104,7 @@ Date: 2022-07-17 21:36:25 +0300 src/scripts/xzgrep.in | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) -commit 62c1d2bc2da66e182c52bb1e3aa791f6d8a9e099 +commit b48f9d615f2c2e8d2f6e253d0e48ee66d0652b68 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-17 20:55:16 +0300 @@ -1249,7 +3134,7 @@ Date: 2022-07-17 20:55:16 +0300 src/scripts/xzgrep.in | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) -commit 372a0d12c95fad5e5477b4f79646bf66d28e0c31 +commit 2d2201bc6315deda4d43625aa510972467bd51d4 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-14 20:33:05 +0300 @@ -1258,7 +3143,7 @@ Date: 2022-07-14 20:33:05 +0300 tests/test_files.sh | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) -commit b8e3d0c45b62c81dfc6b350556d77cf2c7401edd +commit ce5549a591bf126300618879f5b24023351baff6 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-14 19:37:42 +0300 @@ -1275,7 +3160,7 @@ Date: 2022-07-14 19:37:42 +0300 tests/files/good-unknown_size-with_eopm.lzma | Bin 0 -> 37 bytes 9 files changed, 50 insertions(+), 13 deletions(-) -commit e96bdf7189916bffec0dcc933cd809fac898587c +commit 107c93ee5cad51a6ea0cee471209bfe8d76deaa3 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-14 18:12:38 +0300 @@ -1284,7 +3169,7 @@ Date: 2022-07-14 18:12:38 +0300 src/liblzma/lzma/lzma_decoder.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) -commit 2d54fdf58e80bb2b0852633838576fd1ae7961c5 +commit 511feb5eadb988d641b025d597f4fac7502003b8 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-13 22:24:41 +0300 @@ -1293,7 +3178,7 @@ Date: 2022-07-13 22:24:41 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit ff54b557fe3b06b59a24e4028c1464c35dd5b142 +commit 9595a3119b9faf0ce01375329cad8bbf85c35ea2 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-07-13 22:24:07 +0300 @@ -1313,15 +3198,183 @@ Date: 2022-07-13 22:24:07 +0300 Thanks to Karl Beldan for reporting the problem. - doc/lzma-file-format.txt | 11 ++++- - src/liblzma/common/alone_decoder.c | 2 +- - src/liblzma/lz/lz_decoder.c | 10 +++- - src/liblzma/lz/lz_decoder.h | 8 +-- - src/liblzma/lzma/lzma2_decoder.c | 2 +- - src/liblzma/lzma/lzma_decoder.c | 99 +++++++++++++++++++++++++++----------- - 6 files changed, 94 insertions(+), 38 deletions(-) + doc/lzma-file-format.txt | 11 +++- + src/liblzma/common/alone_decoder.c | 2 +- + src/liblzma/common/microlzma_decoder.c | 2 +- + src/liblzma/lz/lz_decoder.c | 10 +++- + src/liblzma/lz/lz_decoder.h | 8 +-- + src/liblzma/lzma/lzma2_decoder.c | 2 +- + src/liblzma/lzma/lzma_decoder.c | 99 ++++++++++++++++++++++++---------- + 7 files changed, 95 insertions(+), 39 deletions(-) + +commit 0c0f8e9761eb6eaf199082cf144db7ac5f9d8cb2 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-07-12 18:53:04 +0300 + + xz: Document the special memlimit case of 2000 MiB on MIPS32. + + See commit fc3d3a7296ef58bb799a73943636b8bfd95339f7. + + src/xz/xz.1 | 8 ++++++-- + 1 file changed, 6 insertions(+), 2 deletions(-) + +commit d1bfa3dc703325ecd974167e864a8712fdfe936e +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-07-01 21:19:26 +0800 + + Created script to generate code coverage reports. + + The script uses lcov and genhtml after running the tests + to show the code coverage statistics. The script will create + a coverage directory where it is run. It can be run both in + and out of the source directory. + + .gitignore | 4 +++ + tests/code_coverage.sh | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 83 insertions(+) + +commit 86a30b0255d8064169fabfd213d907016d2f9f2a +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-06-16 17:32:19 +0300 -commit bb795fe835cf2b3ae7ed68b155784fbfc14e2faf + Tests: Add more tests into test_check. + + tests/test_check.c | 279 +++++++++++++++++++++++++++++++++++++++++++++++++++-- + tests/tests.h | 23 +++++ + 2 files changed, 295 insertions(+), 7 deletions(-) + +commit 82e30fed66a89706388a8c15dc954d84e63f38fa +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-16 15:02:57 +0300 + + Tests: Use char[][24] array for enum_strings_lzma_ret. + + Array of pointers to short strings is a bit pointless here + and now it's fully const. + + tests/tests.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 5ba9459e6c4a29f6870ca78ce8ac6e519d59c41e +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-16 14:12:14 +0300 + + Tests: tuktest.h: Add tuktest_error_impl to help with error conditions. + + tests/tuktest.h | 72 +++++++++++++++++++++++++-------------------------------- + 1 file changed, 32 insertions(+), 40 deletions(-) + +commit b339892668da20aea22a93668c82b87a38e4a97f +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-16 13:29:59 +0300 + + Tests: tuktest.h: Rename file_from_* and use tuktest_malloc there. + + tests/test_bcj_exact_size.c | 4 +--- + tests/tuktest.h | 52 +++++++++++++++++++++------------------------ + 2 files changed, 25 insertions(+), 31 deletions(-) + +commit d8b63a0ad68d1c461eb373466679ebc41fbc207d +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-16 13:08:19 +0300 + + Tests: tuktest.h: Add malloc wrapper with automatic freeing. + + tests/tuktest.h | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 124 insertions(+) + +commit 1d51536a4b19a8fae768f8eb462fc2238cb36d53 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-16 11:47:37 +0300 + + Tests: tuktest.h: Move a function. + + tests/tuktest.h | 44 ++++++++++++++++++++++---------------------- + 1 file changed, 22 insertions(+), 22 deletions(-) + +commit 70c7555f6403553ee35539e869de0025592d8564 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-14 22:21:15 +0300 + + Tests: test_vli: Remove an invalid test-assertion. + + lzma_vli is unsigned so trying a signed value results in + a compiler warning from -Wsign-conversion. (lzma_vli)-1 + equals to LZMA_VLI_UNKNOWN anyway which is the next assertion. + + tests/test_vli.c | 2 -- + 1 file changed, 2 deletions(-) + +commit 154b73c5a1092c3f785e01666b564ad7ff1be555 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-14 22:10:10 +0300 + + Tests: test_vli: Add const where appropriate. + + tests/test_vli.c | 53 ++++++++++++++++++++++++++++------------------------- + 1 file changed, 28 insertions(+), 25 deletions(-) + +commit 0354d6cce3ff98ea6f927107baf216253f6ce2bb +Author: Jia Tan <jiat75@gmail.com> +Date: 2022-06-13 20:27:03 +0800 + + Added vli tests to .gitignore + + .gitignore | 1 + + 1 file changed, 1 insertion(+) + +commit a08f5ccf6bdc20ef70e41f6f3321618ef146f96e +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-06-12 11:31:40 +0800 + + Created tests for all functions exported in vli.h + + Achieved 100% code coverage vli_encoder.c, vli_decoder.c, and vli_size.c + + tests/Makefile.am | 4 +- + tests/test_vli.c | 308 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 311 insertions(+), 1 deletion(-) + +commit 1e3eb61815a91c0a1bfbb802e2d95593f523491f +Author: jiat75 <jiat0218@gmail.com> +Date: 2022-06-03 21:24:54 +0800 + + Added parallel test artifacts to .gitignore + + .gitignore | 2 ++ + 1 file changed, 2 insertions(+) + +commit 00e3613f1212eaa84e721585fdb4de6967cf2476 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-14 21:29:21 +0300 + + Tests: Use good-1-empty-bcj-lzma2.xz in test_bcj_exact_size. + + It's much nicer this way so that the test data isn't a hardcoded + table inside the C file. + + tests/test_bcj_exact_size.c | 19 +++++++------------ + 1 file changed, 7 insertions(+), 12 deletions(-) + +commit 86bab755be252bfd3e0a9aee8e7b83a9bbb23ed0 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-14 21:26:13 +0300 + + Tests: Add file reading helpers to tuktest.h. + + tests/tuktest.h | 183 +++++++++++++++++++++++++++++++++++++++++++++++++++++--- + 1 file changed, 176 insertions(+), 7 deletions(-) + +commit 83d2337b72dbf391c6f3b41889eea99e51679105 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-14 18:21:57 +0300 + + Tests: tuktest.h: Move a printf from a macro to a helper function. + + tests/tuktest.h | 26 ++++++++++++++++++-------- + 1 file changed, 18 insertions(+), 8 deletions(-) + +commit f9e8176ea7d520797a2db2d49a5a632c285674a8 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-06-14 17:20:49 +0300 @@ -1334,7 +3387,371 @@ Date: 2022-06-14 17:20:49 +0300 tests/files/good-1-empty-bcj-lzma2.xz | Bin 0 -> 52 bytes 2 files changed, 5 insertions(+) -commit dbd8b0bf45c10f991635f585a2ea59fb5ae28ff3 +commit aa75c5563a760aea3aa23d997d519e702e82726b +Author: Jia Tan <jiat0218@gmail.com> +Date: 2022-06-10 21:35:18 +0800 + + Tests: Created tests for hardware functions. + + Created tests for all API functions exported in + src/liblzma/api/lzma/hardware.h. The tests are fairly trivial + but are helpful because they will inform users if their machines + cannot support these functions. They also improve the code + coverage metrics. + + .gitignore | 1 + + tests/Makefile.am | 2 ++ + tests/test_hardware.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ + 3 files changed, 48 insertions(+) + +commit 5c8ffdca20813939818843476fb212dfae8838a3 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-02 21:01:45 +0300 + + Tests: Convert test_check to tuktest. + + Thanks to Jia Tan for help with all the tests. + + tests/test_check.c | 67 +++++++++++++++++++++++++++--------------------------- + 1 file changed, 33 insertions(+), 34 deletions(-) + +commit faf5ff8899d539b4dcd2a7e5280cb820a4746c86 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-02 20:31:03 +0300 + + Tests: Convert test_block_header to tuktest. + + tests/test_block_header.c | 89 +++++++++++++++++++++++++++-------------------- + 1 file changed, 52 insertions(+), 37 deletions(-) + +commit 754d39fbebee3782258d42f154a223d3c5770ec7 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-02 20:28:23 +0300 + + Tests: Convert test_bcj_exact_size to tuktest. + + The compress() and decompress() functions were merged because + the later depends on the former so they need to be a single + test case. + + tests/test_bcj_exact_size.c | 75 +++++++++++++++++++++++++-------------------- + 1 file changed, 41 insertions(+), 34 deletions(-) + +commit 96da21470f9570cd08286906a050a7c22631775b +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-02 20:27:00 +0300 + + Tests: Include tuktest.h in tests.h. + + This breaks -Werror because none of the tests so far use + tuktest.h and thus there are warnings about unused variables + and functions. + + tests/tests.h | 47 +++++++++++++++++++++++------------------------ + 1 file changed, 23 insertions(+), 24 deletions(-) + +commit df71ba1c991f60c3269aaadd398247e632714626 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-06-02 20:25:21 +0300 + + Tests: Add tuktest.h mini-test-framework. + + tests/Makefile.am | 1 + + tests/tuktest.h | 752 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 2 files changed, 753 insertions(+) + +commit 4773608554d1b684a05ff9c1d879cf5c42266d33 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-05-23 21:31:36 +0300 + + Build: Enable Automake's parallel test harness. + + It has been the default for quite some time already and + the old serial harness isn't discouraged. The downside is + that with parallel tests one cannot print progress info or + other diagnostics to the terminal; all output from the tests + will be in the log files only. But now that the compression + tests are separated the parallel tests will speed things up. + + configure.ac | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 9a6dd6d46f7f256a5573e5d874c1052033ed7b05 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-05-23 21:17:47 +0300 + + Tests: Split test_compress.sh into separate test unit for each file. + + test_compress.sh now takes one command line argument: + a filename to be tested. If it begins with "compress_generated_" + the file will be created with create_compress_files. + + This will allow parallel execution of the slow tests. + + tests/Makefile.am | 11 +++- + tests/test_compress.sh | 91 +++++++++++++++++++--------------- + tests/test_compress_generated_abc | 3 ++ + tests/test_compress_generated_random | 3 ++ + tests/test_compress_generated_text | 3 ++ + tests/test_compress_prepared_bcj_sparc | 3 ++ + tests/test_compress_prepared_bcj_x86 | 3 ++ + 7 files changed, 77 insertions(+), 40 deletions(-) + +commit c7758ac9c734707514dd34f254173ebac5eea7f8 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-05-23 20:32:49 +0300 + + Test: Make create_compress_files.c a little more flexible. + + If a command line argument is given, then only the test file + of that type is created. It's quite dumb in sense that unknown + names don't give an error but it's good enough here. + + Also use EXIT_FAILURE instead of 1 as exit status for errors. + + tests/create_compress_files.c | 25 +++++++++++++++---------- + 1 file changed, 15 insertions(+), 10 deletions(-) + +commit 4a8e4a7b0ad4b03c0ac6862716c3457452cdaf8c +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-05-23 20:17:42 +0300 + + Tests: Remove unneeded commented lines from test_compress.sh. + + tests/test_compress.sh | 13 ------------- + 1 file changed, 13 deletions(-) + +commit 2ee50d150ee009f36135540b459e6ff328549725 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-05-23 20:16:00 +0300 + + Tests: Remove progress indicator from test_compress.sh. + + It will be useless with Automake's parallel tests. + + tests/test_compress.sh | 9 --------- + 1 file changed, 9 deletions(-) + +commit 2ce4f36f179a81d0c6e182a409f363df759d1ad0 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-05-23 19:37:18 +0300 + + liblzma: Silence a warning. + + The actual initialization is done via mythread_sync and seems + that GCC doesn't necessarily see that it gets initialized there. + + src/liblzma/common/stream_decoder_mt.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 5d8f3764ef43c35910e6d7003c0900a961ef6544 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-04-14 20:53:16 +0300 + + xz: Fix build with --disable-threads. + + src/xz/hardware.c | 4 ++++ + 1 file changed, 4 insertions(+) + +commit 1d592897278b172d8549aa29c3a1f3a4f432a9b9 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-04-14 14:50:17 +0300 + + xz: Change the cap of the default -T0 memlimit for 32-bit xz. + + The SIZE_MAX / 3 was 1365 MiB. 1400 MiB gives little more room + and it looks like a round (artificial) number in --info-memory + once --info-memory is made to display it. + + Also, using #if avoids useless code on 64-bit builds. + + src/xz/hardware.c | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +commit c77fe55ddb7752ed0fec46967c5ec9a72632ea0c +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-04-14 14:20:46 +0300 + + xz: Add a default soft memory usage limit for --threads=0. + + This is a soft limit in sense that it only affects the number of + threads. It never makes xz fail and it never makes xz change + settings that would affect the compressed output. + + The idea is to make -T0 have more reasonable behavior when + the system has very many cores or when a memory-hungry + compression options are used. This also helps with 32-bit xz, + preventing it from running out of address space. + + The downside of this commit is that now the number of threads + might become too low compared to what the user expected. I + hope this to be an acceptable compromise as the old behavior + has been a source of well-argued complaints for a long time. + + src/xz/coder.c | 28 ++++++++++++++++++++++++++-- + src/xz/hardware.c | 38 +++++++++++++++++++++++++++++--------- + src/xz/hardware.h | 27 +++++++++++++++++++++++++++ + 3 files changed, 82 insertions(+), 11 deletions(-) + +commit 0adc13bfe32c14f3e4c6ce9f2d4fdf4112ab53f4 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-04-14 12:59:09 +0300 + + xz: Make -T0 use multithreaded mode on single-core systems. + + The main problem withi the old behavior is that the compressed + output is different on single-core systems vs. multicore systems. + This commit fixes it by making -T0 one thread in multithreaded mode + on single-core systems. + + The downside of this is that it uses more memory. However, if + --memlimit-compress is used, xz can (thanks to the previous commit) + drop to the single-threaded mode still. + + src/xz/coder.c | 18 +++++++++--------- + src/xz/hardware.c | 14 ++++++++++++++ + src/xz/hardware.h | 4 ++++ + 3 files changed, 27 insertions(+), 9 deletions(-) + +commit 898faa97287a756231c663a3ed5165672b417207 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-04-14 12:38:00 +0300 + + xz: Changes to --memlimit-compress and --no-adjust. + + In single-threaded mode, --memlimit-compress can make xz scale down + the LZMA2 dictionary size to meet the memory usage limit. This + obviously affects the compressed output. However, if xz was in + threaded mode, --memlimit-compress could make xz reduce the number + of threads but it wouldn't make xz switch from multithreaded mode + to single-threaded mode or scale down the LZMA2 dictionary size. + This seemed illogical and there was even a "FIXME?" about it. + + Now --memlimit-compress can make xz switch to single-threaded + mode if one thread in multithreaded mode uses too much memory. + If memory usage is still too high, then the LZMA2 dictionary + size can be scaled down too. + + The option --no-adjust was also changed so that it no longer + prevents xz from scaling down the number of threads as that + doesn't affect compressed output (only performance). After + this commit --no-adjust only prevents adjustments that affect + compressed output, that is, with --no-adjust xz won't switch + from multithreaded mode to single-threaded mode and won't + scale down the LZMA2 dictionary size. + + The man page wasn't updated yet. + + src/xz/coder.c | 63 +++++++++++++++++++++++++++++++++++++++------------------- + 1 file changed, 43 insertions(+), 20 deletions(-) + +commit cad299008cf73ec566f0662a9cf2b94f86a99659 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-04-11 22:20:49 +0300 + + xz: Add --memlimit-mt-decompress along with a default limit value. + + --memlimit-mt-decompress allows specifying the limit for + multithreaded decompression. This matches memlimit_threading in + liblzma. This limit can only affect the number of threads being + used; it will never prevent xz from decompressing a file. The + old --memlimit-decompress option is still used at the same time. + + If the value of --memlimit-decompress (the default value or + one specified by the user) is less than the value of + --memlimit-mt-decompress , then --memlimit-mt-decompress is + reduced to match --memlimit-decompress. + + Man page wasn't updated yet. + + src/xz/args.c | 24 +++++++++++++++------- + src/xz/coder.c | 34 ++++++++++--------------------- + src/xz/hardware.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++----- + src/xz/hardware.h | 17 ++++++++++------ + src/xz/message.c | 4 +++- + 5 files changed, 97 insertions(+), 42 deletions(-) + +commit fe87b4cd5364f5bbb6a75a0299f1500c852d7c9a +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-04-06 23:11:59 +0300 + + liblzma: Threaded decoder: Improve setting of pending_error. + + It doesn't need to be done conditionally. The comments try + to explain it. + + src/liblzma/common/stream_decoder_mt.c | 51 +++++++++++++++++++++++----------- + 1 file changed, 35 insertions(+), 16 deletions(-) + +commit 90621da7f6e1bfd6d91d60415eae04b2bca274c2 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-04-06 12:39:49 +0300 + + liblzma: Add a new flag LZMA_FAIL_FAST for threaded decoder. + + In most cases if the input file is corrupt the application won't + care about the uncompressed content at all. With this new flag + the threaded decoder will return an error as soon as any thread + has detected an error; it won't wait to copy out the data before + the location of the error. + + I don't plan to use this in xz to keep the behavior consistent + between single-threaded and multi-threaded modes. + + src/liblzma/api/lzma/container.h | 25 ++++++++++++++++- + src/liblzma/common/common.h | 7 +++-- + src/liblzma/common/stream_decoder_mt.c | 50 +++++++++++++++++++--------------- + 3 files changed, 56 insertions(+), 26 deletions(-) + +commit 64b6d496dc815a176d8307f418f6834a26783484 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-04-05 12:24:57 +0300 + + liblzma: Threaded decoder: Always wait for output if LZMA_FINISH is used. + + This makes the behavior consistent with the single-threaded + decoder when handling truncated .xz files. + + Thanks to Jia Tan for finding this issue. + + src/liblzma/common/stream_decoder_mt.c | 26 ++++++++++++++++++++++++-- + 1 file changed, 24 insertions(+), 2 deletions(-) + +commit e671bc8828b9c0c5406c3a22c541301d0eb54518 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-04-02 21:49:59 +0300 + + liblzma: Threaded decoder: Support zpipe.c-style decoding loop. + + This makes it possible to call lzma_code() in a loop that only + reads new input when lzma_code() didn't fill the output buffer + completely. That isn't the calling style suggested by the + liblzma example program 02_decompress.c so perhaps the usefulness + of this feature is limited. + + Also, it is possible to write such a loop so that it works + with the single-threaded decoder but not with the threaded + decoder even after this commit, or so that it works only if + lzma_mt.timeout = 0. + + The zlib tutorial <https://zlib.net/zlib_how.html> is a well-known + example of a loop where more input is read only when output isn't + full. Porting this as is to liblzma would work with the + single-threaded decoder (if LZMA_CONCATENATED isn't used) but it + wouldn't work with threaded decoder even after this commit because + the loop assumes that no more output is possible when it cannot + read more input ("if (strm.avail_in == 0) break;"). This cannot + be fixed at liblzma side; the loop has to be modified at least + a little. + + I'm adding this in any case because the actual code is simple + and short and should have no harmful side-effects in other + situations. + + src/liblzma/common/stream_decoder_mt.c | 77 +++++++++++++++++++++++++++++----- + 1 file changed, 67 insertions(+), 10 deletions(-) + +commit 2ba8173e27be4793edb46497e499ac2ae753a316 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-03-31 00:05:07 +0300 @@ -1343,7 +3760,7 @@ Date: 2022-03-31 00:05:07 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit bb66a98ded39c40df202da17cb2055356dcd4168 +commit 69d1b3fc29677af8ade8dc15dba83f0589cb63d6 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-03-29 19:19:12 +0300 @@ -1396,7 +3813,27 @@ Date: 2022-03-29 19:19:12 +0300 src/scripts/xzgrep.in | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) -commit fa3af4e4c6a8f692785b87ba0f165f34d154ae0f +commit bd93b776c1bd15e90661033c918cdeb354dbcc38 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-03-26 01:02:44 +0200 + + liblzma: Fix a deadlock in threaded decoder. + + If a worker thread has consumed all input so far and it's + waiting on thr->cond and then the main thread enables + partial update for that thread, the code used to deadlock. + This commit allows one dummy decoding pass to occur in this + situation which then also does the partial update. + + As part of the fix, this moves thr->progress_* updates to + avoid the second thr->mutex locking. + + Thanks to Jia Tan for finding, debugging, and reporting the bug. + + src/liblzma/common/stream_decoder_mt.c | 71 +++++++++++++++++++++++++--------- + 1 file changed, 52 insertions(+), 19 deletions(-) + +commit e0394e94230f208682ac1e1f4c41f22f9ad79916 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-03-23 16:34:00 +0200 @@ -1405,7 +3842,63 @@ Date: 2022-03-23 16:34:00 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit f12ce0f23a0496daebc77e8359309bf724ad5e8a +commit 487c77d48760564b1949c5067630b675b87be4de +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-03-23 16:28:55 +0200 + + liblzma: Threaded decoder: Don't stop threads on LZMA_TIMED_OUT. + + LZMA_TIMED_OUT is not an error and thus stopping threads on + LZMA_TIMED_OUT breaks the decoder badly. + + Thanks to Jia Tan for finding the bug and for the patch. + + src/liblzma/common/stream_decoder_mt.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 6c6da57ae2aa962aabde6892442227063d87e88c +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-03-07 00:36:16 +0200 + + xz: Add initial support for threaded decompression. + + If threading support is enabled at build time, this will + use lzma_stream_decoder_mt() even for single-threaded mode. + With memlimit_threading=0 the behavior should be identical. + + This needs some work like adding --memlimit-threading=LIMIT. + + The original patch from Sebastian Andrzej Siewior included + a method to get currently available RAM on Linux. It might + be one way to go but as it is Linux-only, the available-RAM + approach needs work for portability or using a fallback method + on other OSes. + + The man page wasn't updated yet. + + src/xz/coder.c | 36 +++++++++++++++++++++++++++++++++++- + 1 file changed, 35 insertions(+), 1 deletion(-) + +commit 4cce3e27f529af33e0e7749a8cbcec59954946b5 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-03-06 23:36:20 +0200 + + liblzma: Add threaded .xz decompressor. + + I realize that this is about a decade late. + + Big thanks to Sebastian Andrzej Siewior for the original patch. + I made a bunch of smaller changes but after a while quite a few + things got rewritten. So any bugs in the commit were created by me. + + src/liblzma/api/lzma/container.h | 90 +- + src/liblzma/common/Makefile.inc | 5 + + src/liblzma/common/common.h | 4 + + src/liblzma/common/stream_decoder_mt.c | 1814 ++++++++++++++++++++++++++++++++ + src/liblzma/liblzma.map | 1 + + 5 files changed, 1907 insertions(+), 7 deletions(-) + +commit 717631b9788dc9c100ee0c87d3c14a2782638ff4 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-03-06 16:54:23 +0200 @@ -1417,7 +3910,34 @@ Date: 2022-03-06 16:54:23 +0200 src/liblzma/api/lzma/block.h | 3 --- 1 file changed, 3 deletions(-) -commit 412566731111edc28f0b68bc966d01ab24d17082 +commit 1a4bb97a00936535e30ac61945aeee38882b5d1a +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-03-06 16:41:19 +0200 + + liblzma: Add new output queue (lzma_outq) features. + + Add lzma_outq_clear_cache2() which may leave one buffer allocated + in the cache. + + Add lzma_outq_outbuf_memusage() to get the memory needed for + a single lzma_outbuf. This is now used internally in outqueue.c too. + + Track both the total amount of memory allocated and the amount of + memory that is in active use (not in cache). + + In lzma_outbuf, allow storing the current input position that + matches the current output position. This way the main thread + can notice when no more output is possible without first providing + more input. + + Allow specifying return code for lzma_outq_read() in a finished + lzma_outbuf. + + src/liblzma/common/outqueue.c | 43 +++++++++++++++++++++++++++++++++++------- + src/liblzma/common/outqueue.h | 44 ++++++++++++++++++++++++++++++++++++++++++- + 2 files changed, 79 insertions(+), 8 deletions(-) + +commit ddbc6f58c2de388eed24cd7ea91b523d397da5f4 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-03-06 15:18:58 +0200 @@ -1426,7 +3946,7 @@ Date: 2022-03-06 15:18:58 +0200 src/liblzma/common/index_hash.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) -commit 7c3ce02df001701a26c2fa41b2dc8b530743f454 +commit 20e7a33e2d59c6a814447d3991f21e2702174b20 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-02-22 03:42:57 +0200 @@ -1437,7 +3957,7 @@ Date: 2022-02-22 03:42:57 +0200 src/liblzma/api/lzma/vli.h | 2 ++ 1 file changed, 2 insertions(+) -commit b8f667fe0c3e17fd2c559901d2aaf19b1dfd51f2 +commit 4f78f5fcf63592f2d77e921cfe0d5de300867374 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-02-22 02:04:18 +0200 @@ -1452,7 +3972,7 @@ Date: 2022-02-22 02:04:18 +0200 src/liblzma/common/stream_encoder_mt.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) -commit 2356d53edd8f44ff8f411972ca62d63079ff59af +commit 5313ad66b40aab822ddca3e9905254cb99a4080d Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-02-22 01:37:39 +0200 @@ -1461,7 +3981,7 @@ Date: 2022-02-22 01:37:39 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 748ef0833821f1ea4ff5d57abc3cab1ae76266d3 +commit 865e0a3689a25a7ee8eecae1a34c1775e3aa676e Author: Ed Maste <emaste@FreeBSD.org> Date: 2022-02-11 15:25:46 +0000 @@ -1471,7 +3991,7 @@ Date: 2022-02-11 15:25:46 +0000 src/liblzma/check/crc64_x86.S | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) -commit 068a6e3286c9ca7d67f13f4c78646c4f4a3c3c5b +commit 1c9a5786d206b4abc8e427326651c8174baea753 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-02-20 20:36:27 +0200 @@ -1500,7 +4020,7 @@ Date: 2022-02-20 20:36:27 +0200 src/liblzma/common/block_decoder.c | 79 ++++++++++++++++++++++++++------------ 1 file changed, 54 insertions(+), 25 deletions(-) -commit 766df4f62c165ba96d60af8fadbada51349fd140 +commit 555de11873eb00c9b94a8be70645db502e5a9dbd Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-02-20 19:38:55 +0200 @@ -1510,7 +4030,7 @@ Date: 2022-02-20 19:38:55 +0200 tests/files/bad-1-lzma2-11.xz | Bin 0 -> 64 bytes 2 files changed, 5 insertions(+) -commit 12a6d6ce2a9850e32140e542decfd9c8a33c87fc +commit f0da507f22e7f4e3edb75b45b74d344244ca03fb Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-02-18 18:51:10 +0200 @@ -1521,7 +4041,7 @@ Date: 2022-02-18 18:51:10 +0200 po4a/fr_FR.po | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) -commit 00e6aad836cf301e95575ea53790f5ab96fe8ed6 +commit f7711d228c3c32395460c82498c60a9f730d0239 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2022-02-07 01:14:37 +0200 @@ -1537,7 +4057,7 @@ Date: 2022-02-07 01:14:37 +0200 po4a/po4a.conf | 2 +- 2 files changed, 3542 insertions(+), 1 deletion(-) -commit e20ce2b12251a246c50fb5b7fa2204c11611b407 +commit 6468f7e41a8e9c611e4ba8d34e2175c5dacdbeb4 Author: jiat75 <jiat0218@gmail.com> Date: 2022-01-28 20:47:55 +0800 @@ -1552,7 +4072,23 @@ Date: 2022-01-28 20:47:55 +0800 src/liblzma/lzma/lzma_encoder.c | 3 +++ 2 files changed, 6 insertions(+) -commit feb80ace866c3033cefbc39f6b07ada83e316e6a +commit 2523c30705f49eabd27b854aa656ae87cc224808 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2022-02-06 23:19:32 +0200 + + liblzma: Fix uint64_t vs. size_t confusion. + + This broke 32-bit builds due to a pointer type mismatch. + + This bug was introduced with the output-size-limited encoding + in 625f4c7c99b2fcc4db9e7ab2deb4884790e2e17c. + + Thanks to huangqinjin for the bug report. + + src/liblzma/rangecoder/range_encoder.h | 11 +++++++---- + 1 file changed, 7 insertions(+), 4 deletions(-) + +commit 2bd36c91d03e03b31a4f12fd0afc100ae32d66e2 Author: huangqinjin <huangqinjin@gmail.com> Date: 2021-12-13 20:49:21 +0800 @@ -1561,7 +4097,7 @@ Date: 2021-12-13 20:49:21 +0800 CMakeLists.txt | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) -commit 725f2e0522c4ef8dc5fdd69b5017bac6322ebce2 +commit 2024fbf2794885277d05378d40b2b8015a7c3b40 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-11-13 21:04:05 +0200 @@ -1570,7 +4106,7 @@ Date: 2021-11-13 21:04:05 +0200 src/scripts/xzgrep.1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 7955669d423724e69a7f2dae1173dae248f7a596 +commit 400e7a239a53282cedaad927a41f3463d7f542e5 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-11-13 18:23:24 +0200 @@ -1579,7 +4115,7 @@ Date: 2021-11-13 18:23:24 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 671673a7a209e6131491e6a424758bbc13fb7aa1 +commit 3a512c7787b2642ca946f4adc6e9a0a5d9b0d5a0 Author: Ville Skyttä <ville.skytta@iki.fi> Date: 2021-11-13 10:11:57 +0200 @@ -1593,7 +4129,26 @@ Date: 2021-11-13 10:11:57 +0200 src/scripts/xzgrep.in | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) -commit 45e538257e85050f67a6d363961e31b71e3ec565 +commit edf525e2b1840dcaf377df472c67d8f11f8ace1b +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-10-28 23:02:11 +0300 + + Bump the version number for 5.3.2alpha. + + src/liblzma/api/lzma/version.h | 2 +- + src/liblzma/liblzma.map | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) + +commit ea8c948655a86290524efe59cff067e06a886709 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-10-28 22:59:52 +0300 + + Add NEWS for 5.3.2alpha. + + NEWS | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 86 insertions(+) + +commit 52435f302f4724157ec50b4210cbe42b285c3cbc Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-10-27 23:27:48 +0300 @@ -1602,7 +4157,7 @@ Date: 2021-10-27 23:27:48 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit ca21733d24a4f65b46a5efa124a89c859ac5592e +commit f2aea1d5a504b2021bf47a238390e4f12bdd518d Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-10-27 23:23:11 +0300 @@ -1614,7 +4169,7 @@ Date: 2021-10-27 23:23:11 +0300 src/xz/file_io.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) -commit 906b990b151b93750958371e39ad00e8d1daada9 +commit 892b16cc282f5b4e1c49871b326f4db25c5b4d81 Author: Alexander Bluhm <alexander.bluhm@gmx.net> Date: 2021-10-05 23:33:16 +0200 @@ -1631,7 +4186,20 @@ Date: 2021-10-05 23:33:16 +0200 src/xz/file_io.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) -commit ca83df96c48371ce0094c73c6f4582899ae4b62b +commit 2b509c868cae3988bf21cd693fbf9021cdc85628 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-09-17 17:31:11 +0300 + + liblzma: Fix liblzma.map for the lzma_microlzma_* symbols. + + This should have been part of d267d109c370a40b502e73f8664b154b15e4f253. + + Thanks to Gao Xiang. + + src/liblzma/liblzma.map | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) + +commit cacb06a954b58255dfc084a0bc9708f43a0fd6d6 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-09-09 22:21:07 +0300 @@ -1640,7 +4208,7 @@ Date: 2021-09-09 22:21:07 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit d8b294af03840c7c47d6b7b0d4c0d6d2f6357dfa +commit 6928aac9da6ba612780b9f72ba1d6ecbe1e8b54e Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-09-09 21:41:51 +0300 @@ -1657,7 +4225,23 @@ Date: 2021-09-09 21:41:51 +0300 src/liblzma/api/lzma.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -commit c2fde22befe1f5256ad3d7c4cb35794aa8e64c67 +commit d267d109c370a40b502e73f8664b154b15e4f253 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-09-05 20:38:12 +0300 + + liblzma: Rename EROFS LZMA to MicroLZMA. + + It still exists primarily for EROFS but MicroLZMA is + a more generic name (that hopefully doesn't clash with + something that already exists). + + src/liblzma/api/lzma/container.h | 33 +++++++++++++--------- + src/liblzma/common/Makefile.inc | 4 +-- + .../{erofs_decoder.c => microlzma_decoder.c} | 32 ++++++++++----------- + .../{erofs_encoder.c => microlzma_encoder.c} | 30 ++++++++++---------- + 4 files changed, 52 insertions(+), 47 deletions(-) + +commit 3247e95115acb95bc27f41e8cf4501db5b0b4309 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-06-04 19:02:38 +0300 @@ -1668,7 +4252,7 @@ Date: 2021-06-04 19:02:38 +0300 src/scripts/xzdiff.1 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -commit 8d0fd42fbedd01fd3c847b56d19841e603500dea +commit 96f5a28a46fc93ac4e296808ac0f8631d05498bc Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-06-04 18:52:48 +0300 @@ -1684,18 +4268,7 @@ Date: 2021-06-04 18:52:48 +0300 src/scripts/xzless.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 6e2cab8579fc6f832b5954510b27fd7e74563436 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-07-12 18:53:04 +0300 - - xz: Document the special memlimit case of 2000 MiB on MIPS32. - - See commit 95806a8a52ae57bddf6c77dfd19cf7938a92e040. - - src/xz/xz.1 | 8 ++++++-- - 1 file changed, 6 insertions(+), 2 deletions(-) - -commit 38b311462bd8974888ba3cd9312de132c76ec2e0 +commit 5fb5212d816addbc523d0798cb482fdd0484f8fa Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-04-11 19:58:10 +0300 @@ -1704,7 +4277,7 @@ Date: 2021-04-11 19:58:10 +0300 THANKS | 2 ++ 1 file changed, 2 insertions(+) -commit 95806a8a52ae57bddf6c77dfd19cf7938a92e040 +commit fc3d3a7296ef58bb799a73943636b8bfd95339f7 Author: Ivan A. Melnikov <iv@altlinux.org> Date: 2021-04-09 11:45:10 +0300 @@ -1721,7 +4294,7 @@ Date: 2021-04-09 11:45:10 +0300 src/xz/hardware.c | 6 ++++++ 1 file changed, 6 insertions(+) -commit a79bd30a6fbfe614f47ad77498c46e218dbe2b1d +commit e7da44d5151e21f153925781ad29334ae0786101 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-02-13 23:31:27 +0200 @@ -1734,7 +4307,7 @@ Date: 2021-02-13 23:31:27 +0200 CMakeLists.txt | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) -commit 64d98147616c48f2df37fb46b49350f3d5febd34 +commit a61dd82ada39030f41b4ffca9ea551714908bedc Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-01-30 18:36:04 +0200 @@ -1781,7 +4354,30 @@ Date: 2021-01-30 18:36:04 +0200 CMakeLists.txt | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) -commit 9e2f9e2d0841dd0417125ca3c7a05418541fb230 +commit 5b7bc1b8ae766a76710ca1b99f909cf52c697f05 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-29 21:19:43 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 6c6f0db340dcb8bb424411cedba713405d55f6b8 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-29 21:19:08 +0200 + + liblzma: Fix unitialized variable. + + This was introduced two weeks ago in the commit + 625f4c7c99b2fcc4db9e7ab2deb4884790e2e17c. + + Thanks to Nathan Moinvaziri. + + src/liblzma/lzma/lzma_encoder.c | 1 + + 1 file changed, 1 insertion(+) + +commit bb1d5c1fdd30550d4221ecd336e0be1206132a5c Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-01-24 22:32:41 +0200 @@ -1792,7 +4388,16 @@ Date: 2021-01-24 22:32:41 +0200 tests/files/bad-1-lzma2-9.xz | Bin 72 -> 72 bytes 3 files changed, 9 insertions(+), 2 deletions(-) -commit 00a4c69bbbf302dd6323fd6970890ee22ddd4f96 +commit 6b8abc84a5469792e0355d0bfc0784d41cfdfef7 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-24 19:22:35 +0200 + + liblzma: Fix a wrong comment in stream_encoder_mt.c. + + src/liblzma/common/stream_encoder_mt.c | 10 +++++++--- + 1 file changed, 7 insertions(+), 3 deletions(-) + +commit 939fc5ed654aac25fe0c8684b2df8dbeadb2de1e Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-01-24 18:51:51 +0200 @@ -1802,7 +4407,7 @@ Date: 2021-01-24 18:51:51 +0200 tests/files/bad-1-lzma2-9.xz | Bin 0 -> 72 bytes 2 files changed, 4 insertions(+) -commit 1da2269b2e9b952d165ad7bec823de9dfe9a9e9e +commit fdd30032f8531ac89519b48c21d810ecf06825f6 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-01-24 17:02:00 +0200 @@ -1812,7 +4417,99 @@ Date: 2021-01-24 17:02:00 +0200 tests/files/bad-1-check-crc32-2.xz | Bin 0 -> 72 bytes 2 files changed, 7 insertions(+) -commit 11ceecb5e2999ff96989915120e63a2e7e130057 +commit db465419ae26ec7fb9b9472183911ff521620c77 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-17 19:20:50 +0200 + + liblzma: In EROFS LZMA decoder, verify that comp_size matches at the end. + + When the uncompressed size is known to be exact, after decompressing + the stream exactly comp_size bytes of input must have been consumed. + This is a minor improvement to error detection. + + src/liblzma/common/erofs_decoder.c | 7 ++++++- + 1 file changed, 6 insertions(+), 1 deletion(-) + +commit 774cc0118ba2496581cb2621505a04bb6598cc75 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-17 18:53:34 +0200 + + liblzma: Make EROFS LZMA decoder work when exact uncomp_size isn't known. + + The caller must still not specify an uncompressed size bigger + than the actual uncompressed size. + + As a downside, this now needs the exact compressed size. + + src/liblzma/api/lzma/container.h | 23 ++++++++--- + src/liblzma/common/erofs_decoder.c | 80 ++++++++++++++++++++++++++++++++++---- + 2 files changed, 91 insertions(+), 12 deletions(-) + +commit 421b0aa352da244075db10205cf33712f91b9835 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-14 20:57:11 +0200 + + liblzma: Fix missing normalization in rc_encode_dummy(). + + Without this fix it could attempt to create too much output. + + src/liblzma/rangecoder/range_encoder.h | 7 ++++++- + 1 file changed, 6 insertions(+), 1 deletion(-) + +commit 601ec0311e769fc704daaaa7dac0ca840aff080e +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-14 20:07:01 +0200 + + liblzma: Add EROFS LZMA encoder and decoder. + + Right now this is just a planned extra-compact format for use + in the EROFS file system in Linux. At this point it's possible + that the format will either change or be abandoned and removed + completely. + + The special thing about the encoder is that it uses the + output-size-limited encoding added in the previous commit. + EROFS uses fixed-sized blocks (e.g. 4 KiB) to hold compressed + data so the compressors must be able to create valid streams + that fill the given block size. + + src/liblzma/api/lzma/container.h | 76 +++++++++++++++++++ + src/liblzma/common/Makefile.inc | 2 + + src/liblzma/common/erofs_decoder.c | 148 +++++++++++++++++++++++++++++++++++++ + src/liblzma/common/erofs_encoder.c | 139 ++++++++++++++++++++++++++++++++++ + src/liblzma/liblzma.map | 2 + + 5 files changed, 367 insertions(+) + +commit 625f4c7c99b2fcc4db9e7ab2deb4884790e2e17c +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-13 19:16:32 +0200 + + liblzma: Add rough support for output-size-limited encoding in LZMA1. + + With this it is possible to encode LZMA1 data without EOPM so that + the encoder will encode as much input as it can without exceeding + the specified output size limit. The resulting LZMA1 stream will + be a normal LZMA1 stream without EOPM. The actual uncompressed size + will be available to the caller via the uncomp_size pointer. + + One missing thing is that the LZMA layer doesn't inform the LZ layer + when the encoding is finished and thus the LZ may read more input + when it won't be used. However, this doesn't matter if encoding is + done with a single call (which is the planned use case for now). + For proper multi-call encoding this should be improved. + + This commit only adds the functionality for internal use. + Nothing uses it yet. + + src/liblzma/common/common.h | 11 +++ + src/liblzma/lz/lz_encoder.c | 16 ++++ + src/liblzma/lz/lz_encoder.h | 4 + + src/liblzma/lzma/lzma_encoder.c | 127 +++++++++++++++++++++++--------- + src/liblzma/lzma/lzma_encoder_private.h | 12 +++ + src/liblzma/rangecoder/range_encoder.h | 111 ++++++++++++++++++++++++++++ + 6 files changed, 246 insertions(+), 35 deletions(-) + +commit 9cdabbeea891e8f1e7741b076f7db6ac05ae392a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-01-11 23:57:11 +0200 @@ -1822,7 +4519,37 @@ Date: 2021-01-11 23:57:11 +0200 src/scripts/xzdiff.in | 16 +++++++++++----- 2 files changed, 15 insertions(+), 7 deletions(-) -commit d655b8c9cb5a4e596c0fa2290fa91e995b734f97 +commit d9ec3add97cf4c999a7f594c6529680227b6c274 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-11 23:41:30 +0200 + + Update THANKS. + + THANKS | 2 ++ + 1 file changed, 2 insertions(+) + +commit 074259f4f3966aeac6edb205fecbc1a8d2b58bb2 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-11 23:41:16 +0200 + + xz: Make --keep accept symlinks, hardlinks, and setuid/setgid/sticky. + + Previously this required using --force but that has other + effects too which might be undesirable. Changing the behavior + of --keep has a small risk of breaking existing scripts but + since this is a fairly special corner case I expect the + likehood of breakage to be low enough. + + I think the new behavior is more logical. The only reason for + the old behavior was to be consistent with gzip and bzip2. + + Thanks to Vincent Lefevre and Sebastian Andrzej Siewior. + + src/xz/file_io.c | 9 +++++---- + src/xz/xz.1 | 16 +++++++++++++++- + 2 files changed, 20 insertions(+), 5 deletions(-) + +commit 73c555b3077c19dda29b6f4592ced2af876f8333 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-01-11 23:28:52 +0200 @@ -1849,7 +4576,7 @@ Date: 2021-01-11 23:28:52 +0200 src/scripts/xzgrep.in | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) -commit 09c331b03c801414d547fc25711daf099cef6fb5 +commit 194029ffaf74282a81f0c299c07f73caca3232ca Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2021-01-11 22:01:51 +0200 @@ -1884,7 +4611,30 @@ Date: 2021-01-11 22:01:51 +0200 src/scripts/xzdiff.in | 35 +++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-) -commit b33a345cbab90e18b601d26cc3a0dea083b7d8d9 +commit f7fa309e1f7178d04c7bedc03b73077639371e97 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2021-01-09 21:14:36 +0200 + + liblzma: Make lzma_outq usable for threaded decompression too. + + Before this commit all output queue buffers were allocated as + a single big allocation. Now each buffer is allocated separately + when needed. Used buffers are cached to avoid reallocation + overhead but the cache will keep only one buffer size at a time. + This should make things work OK in the decompression where most + of the time the buffer sizes will be the same but with some less + common files the buffer sizes may vary. + + While this should work fine, it's still a bit preliminary + and may even get reverted if it turns out to be useless for + decompression. + + src/liblzma/common/outqueue.c | 268 +++++++++++++++++++++------------ + src/liblzma/common/outqueue.h | 138 ++++++++++++----- + src/liblzma/common/stream_encoder_mt.c | 52 ++++--- + 3 files changed, 301 insertions(+), 157 deletions(-) + +commit a35a69d693ce37d4ba7c1855bda7d9cfa13d1778 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-12-23 17:15:49 +0200 @@ -1893,7 +4643,7 @@ Date: 2020-12-23 17:15:49 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit c01e29a933baba737217a3549eaaced7e499edf5 +commit 4fd79b90c52396d70e0b1206ceb1a873a0ad2589 Author: H.J. Lu <hjl.tools@gmail.com> Date: 2020-12-23 06:49:04 -0800 @@ -1908,7 +4658,7 @@ Date: 2020-12-23 06:49:04 -0800 src/liblzma/check/crc64_x86.S | 9 +++++++++ 2 files changed, 18 insertions(+) -commit 0983682f87ae1a9efcbe4e45579d53c9bca16500 +commit bb3b8c6a23e25db79f862b1de325c56052e0354b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-12-16 18:33:29 +0200 @@ -1917,7 +4667,7 @@ Date: 2020-12-16 18:33:29 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 880596e4b8731b24f6c19cb737ed7d06fe92dda2 +commit 21588ca34af98738954fc12ded1b89d7294ef646 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-12-16 18:30:14 +0200 @@ -1928,7 +4678,7 @@ Date: 2020-12-16 18:30:14 +0200 CMakeLists.txt | 3 +++ 1 file changed, 3 insertions(+) -commit 29050c79f1f83bf73714146a44dfbb4d26e27920 +commit d05b0c42dd8b38d8c6b8193c8af50e9bd3d16f28 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-12-05 22:44:03 +0200 @@ -1937,7 +4687,7 @@ Date: 2020-12-05 22:44:03 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 94fd72474973978f8369be7c23f77c3b741953a6 +commit 1890351f3423627ba5c4c495402f32d7e9ed90b7 Author: Adam Borowski <kilobyte@angband.pl> Date: 2020-09-25 03:35:18 +0200 @@ -1949,7 +4699,7 @@ Date: 2020-09-25 03:35:18 +0200 src/scripts/xzgrep.in | 1 + 2 files changed, 7 insertions(+), 3 deletions(-) -commit 13c58ac13e1a84c30de24191931af425286d0550 +commit 2f108abb3d82e4e2313b438dae9c0c7c7a6366f2 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-11-17 20:51:48 +0200 @@ -1967,7 +4717,7 @@ Date: 2020-11-17 20:51:48 +0200 cmake/tuklib_physmem.cmake | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) -commit 8f7d3345a7c31a20a544dc04232715b1bcdc91f7 +commit 5af726a79273fafa5de5745b117e567f21c90e49 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-11-01 22:56:43 +0200 @@ -1976,7 +4726,7 @@ Date: 2020-11-01 22:56:43 +0200 THANKS | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) -commit ca7bcdb30f14e7cd66fe357d10046442eee648a0 +commit 4575d9d365c756ec189899f9f743e0b3515ce72d Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-11-01 22:34:25 +0200 @@ -1991,7 +4741,7 @@ Date: 2020-11-01 22:34:25 +0200 src/xz/xz.1 | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) -commit 3b40a0792ee5ba0dd4d9ab129b4443585bbfdd14 +commit 620b32f5339f86710cb4435e01ecdac972ccac73 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-11-01 19:09:53 +0200 @@ -2005,7 +4755,7 @@ Date: 2020-11-01 19:09:53 +0200 src/xz/xz.1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit d85699c36df5f839b82800a6e63151eff25677eb +commit cb1f34988c8a4130485091b2f8b641303d8f701b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-11-01 18:49:37 +0200 @@ -2018,7 +4768,7 @@ Date: 2020-11-01 18:49:37 +0200 src/xz/xz.1 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -commit d996ae66176206de9b4d485b250a88b2d9ded151 +commit 5d224da3da87400f2fab313abbd7c710e7169ef9 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-11-01 18:41:21 +0200 @@ -2034,7 +4784,7 @@ Date: 2020-11-01 18:41:21 +0200 src/xz/xz.1 | 66 ++++++++++++++++++++++++++++++------------------------------- 1 file changed, 33 insertions(+), 33 deletions(-) -commit d16d0d198ade6820ed9d39ba6a7068b55b8bf6dc +commit 90457dbe3e5717660f5b81f8c604860fc5137c0c Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-07-12 23:10:03 +0300 @@ -2050,7 +4800,7 @@ Date: 2020-07-12 23:10:03 +0300 src/xz/xz.1 | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) -commit 3acf1adfc768c7d2290e138bcb1109145e4d8160 +commit 352ba2d69af2136bc814aa1df1a132559d445616 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-07-12 20:46:24 +0300 @@ -2063,7 +4813,7 @@ Date: 2020-07-12 20:46:24 +0300 src/common/common_w32res.rc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) -commit adba06e649875b45b5b671f5ab45f694144938d4 +commit a9e2a87f1d61dcf684d809bf08c8ebea93f8a480 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-04-06 19:31:50 +0300 @@ -2074,23 +4824,7 @@ Date: 2020-04-06 19:31:50 +0300 src/scripts/xzgrep.1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 7be1dcf8585e150bffb57a6ad499a63edd0d3494 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-07-12 17:59:41 +0300 - - Translations: Add Portuguese translation. - - Jia Tan made white-space changes and also changed "Language: pt_BR\n" - to pt. The translator wasn't reached so I'm hoping these changes - are OK and will commit it without translator's approval. - - Thanks to Pedro Albuquerque and Jia Tan. - - po/LINGUAS | 1 + - po/pt.po | 1001 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 1002 insertions(+) - -commit 3f94d2a568290bf770a762ee69403955fdb0a228 +commit a7ba275d9b855d186abb29eb7a4f4cb6d9ca6fe0 Author: Bjarni Ingi Gislason <bjarniig@rhi.hi.is> Date: 2020-03-26 22:17:31 +0000 @@ -2118,7 +4852,7 @@ Date: 2020-03-26 22:17:31 +0000 src/scripts/xzgrep.1 | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) -commit 725d9791c90b7cef19a3ab4d7469b567101c2a87 +commit 133d498db0f4b14f066d192d64dbcade45deae6b Author: Bjarni Ingi Gislason <bjarniig@rhi.hi.is> Date: 2020-03-30 21:56:36 +0000 @@ -2136,7 +4870,7 @@ Date: 2020-03-30 21:56:36 +0000 src/scripts/xzgrep.1 | 1 - 1 file changed, 1 deletion(-) -commit 55c2555c5d420e9251a97828022e5c8c4d786dac +commit 057839ca982f886387b66746bffe749cb14fd8cd Author: Bjarni Ingi Gislason <bjarniig@rhi.hi.is> Date: 2020-03-26 21:16:18 +0000 @@ -2162,142 +4896,7 @@ Date: 2020-03-26 21:16:18 +0000 src/xz/xz.1 | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) -commit 55e3a8e0e437279223c108085dccddc60b06036c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-07-10 21:16:40 +0300 - - Translations: Add Serbian translation. - - Quite a few white-space changes were made by Jia Tan to make - this look good. Contacting the translator didn't succeed so - I'm committing this without getting translator's approval. - - Thanks to Мирослав Николић (Miroslav Nikolic) and Jia Tan. - - po/LINGUAS | 1 + - po/sr.po | 987 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 988 insertions(+) - -commit 4e04279cbe8c1af25dde42ebf740722deded27a0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-07-04 23:51:36 +0300 - - Translations: Add Swedish translation. - - Thanks to Sebastian Rasmussen and Jia Tan. - - po/LINGUAS | 1 + - po/sv.po | 983 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 984 insertions(+) - -commit 27f55e96b0cdae3fc51b409663b6c017a88ea8bc -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-07-04 23:40:27 +0300 - - Translations: Add Esperanto translation. - - Thanks to Keith Bowes and Jia Tan. - - po/LINGUAS | 1 + - po/eo.po | 984 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 985 insertions(+) - -commit be31be9e8f12cf6fce4be1ab822660ad411e803b -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-07-01 00:22:33 +0300 - - Translations: Add Catalan translation. - - Thanks to Jordi Mas and Jia Tan. - - po/LINGUAS | 1 + - po/ca.po | 1076 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 1077 insertions(+) - -commit 30e3ad040f4e7ba1a0e97179d6fd62a1201c863b -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-06-30 17:47:08 +0300 - - Translations: Add Ukrainian translation. - - Thanks to Yuri Chornoivan and Jia Tan. - - po/LINGUAS | 1 + - po/uk.po | 996 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 997 insertions(+) - -commit f2e2cce49f7b07e8985ba6add4b4ca79b22eddd0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-06-30 17:45:26 +0300 - - Translators: Add Romanian translation. - - Thanks to Remus-Gabriel Chelu and Jia Tan. - - po/LINGUAS | 1 + - po/ro.po | 1016 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 1017 insertions(+) - -commit a12fd5bad2aa922c9b4cf96d16632696f3e0584e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-06-29 18:33:32 +0300 - - Translations: Update Brazilian Portuguese translation. - - One msgstr was changed. The diff is long due to changes - in the source code line numbers in the comments. - - Thanks to Rafael Fontenelle. - - po/pt_BR.po | 186 ++++++++++++++++++++++++++++++------------------------------ - 1 file changed, 92 insertions(+), 94 deletions(-) - -commit dd713288d27a0d64426d74effe9187087496333c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-06-29 18:04:44 +0300 - - Translations: Add Croatian translation. - - Thanks to Božidar Putanec and Jia Tan. - - po/LINGUAS | 1 + - po/hr.po | 987 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 988 insertions(+) - -commit 016980f26974623a4bcd889b6ba5a0caa4b2f57f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-06-29 17:58:48 +0300 - - Translations: Add Spanish translation. - - Thanks to Cristian Othón Martínez Vera and Jia Tan. - - po/LINGUAS | 1 + - po/es.po | 984 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 985 insertions(+) - -commit 4fe9578c3aab21e6ceb8f0ea470bc87821f5907f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2022-06-29 17:49:43 +0300 - - Translations: Add Korean translation. - - Thanks to Seong-ho Cho and Jia Tan. - - po/LINGUAS | 1 + - po/ko.po | 972 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 973 insertions(+) - -commit cf1ec5518e6e3d89f57685ac83b81ab228e48bcd -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-03-23 18:09:40 +0200 - - v5.2-specific typo fix from fossies.org. - - README | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 968bbfea09447a4f52e856bb97301aca80e40dcb +commit b8e12f5ab4c9fd3cb09a4330b2861f6b979ababd Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-03-23 18:07:50 +0200 @@ -2313,38 +4912,16 @@ Date: 2020-03-23 18:07:50 +0200 src/xz/xz.1 | 4 ++-- 6 files changed, 7 insertions(+), 7 deletions(-) -commit 2327a461e1afce862c22269b80d3517801103c1b +commit 869b9d1b4edd6df07f819d360d306251f8147353 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-03-17 16:27:42 +0200 - - Bump version and soname for 5.2.5. - - src/liblzma/Makefile.am | 2 +- - src/liblzma/api/lzma/version.h | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) - -commit 3be82d2f7dc882258caf0f0a69214e5916b2bdda -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-03-17 16:26:04 +0200 +Date: 2020-03-17 16:24:28 +0200 Update NEWS for 5.2.5. NEWS | 105 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) -commit ab3e57539c7337f0653b13b75dbc5d03ade9700e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-03-16 21:57:21 +0200 - - Translations: Rebuild cs.po to avoid incorrect fuzzy strings. - - "make dist" updates the .po files and the fuzzy strings would - result in multiple very wrong translations. - - po/cs.po | 592 ++++++++++++++++++++++++++++++++++----------------------------- - 1 file changed, 322 insertions(+), 270 deletions(-) - -commit 3a6f38309dc5d44d8a63ebb337b6b2028561c93e +commit a048e3a92d238c65f050a765174d9c75417231d4 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-03-16 20:01:37 +0200 @@ -2353,16 +4930,7 @@ Date: 2020-03-16 20:01:37 +0200 README | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) -commit 9cc0901798217e258e91c13cf6fda7ad42ba108c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-03-16 19:46:27 +0200 - - README: Mention that translatable strings will change after 5.2.x. - - README | 74 +++--------------------------------------------------------------- - 1 file changed, 3 insertions(+), 71 deletions(-) - -commit cc163574249f6a4a66f3dc09d6fe5a71bee24fab +commit 29aed815ad4f98f3e4d355faa76a244ecd8ce716 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-03-16 19:39:45 +0200 @@ -2371,20 +4939,7 @@ Date: 2020-03-16 19:39:45 +0200 README | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) -commit ca261994edc3f2d03d5589c037171c63471ee9dc -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-03-16 17:30:39 +0200 - - Translations: Add partial Danish translation. - - I made a few minor white space changes without getting them - approved by the Danish translation team. - - po/LINGUAS | 1 + - po/da.po | 896 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 897 insertions(+) - -commit 51cd5d051fc730d61411dee292e863582784e189 +commit 7fa7653940cc9dcfcbce2fbc5166ea343ad4e3c1 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-03-16 16:43:29 +0200 @@ -2393,7 +4948,7 @@ Date: 2020-03-16 16:43:29 +0200 INSTALL.generic | 321 ++++++++++++++++++++++++++++---------------------------- 1 file changed, 162 insertions(+), 159 deletions(-) -commit 69d694e5f1beae2bbfa3b6c348ec0ec5f14b5cd0 +commit 9bd317ef03ab9b3e6a927c27c2e9c4ac041182f0 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-03-15 15:27:22 +0200 @@ -2402,7 +4957,7 @@ Date: 2020-03-15 15:27:22 +0200 INSTALL | 51 +++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 41 insertions(+), 10 deletions(-) -commit 2c3b1bb80a3ca7e09728fe4d7a1d8648a5cb9bca +commit a3148c0446dc7fa96363752df414d22539c9007b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-03-15 15:26:20 +0200 @@ -2411,7 +4966,7 @@ Date: 2020-03-15 15:26:20 +0200 m4/ax_pthread.m4 | 219 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 117 insertions(+), 102 deletions(-) -commit 74a5af180a6a6c4b8c90cefb37ee900d3fea7dc6 +commit 7812002dd3ed319e42a14662a8531802cca8ca67 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-03-11 21:15:35 +0200 @@ -2425,7 +4980,7 @@ Date: 2020-03-11 21:15:35 +0200 src/xz/util.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) -commit ceba0d25e826bcdbf64bb4cb03385a2a66f8cbcb +commit 7c8f688bf7fccd65d396e0130cbf4ea5dff5c56f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-03-11 19:38:08 +0200 @@ -2436,7 +4991,7 @@ Date: 2020-03-11 19:38:08 +0200 dos/Makefile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) -commit 29e5bd71612253281fb22bbaa0a566990a74dcc3 +commit 319ca928d73de87940c54e30bffe69f9fa65efdf Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-03-11 19:36:07 +0200 @@ -2445,7 +5000,7 @@ Date: 2020-03-11 19:36:07 +0200 dos/INSTALL.txt | 59 ++++++++++++++++++++++++++++----------------------------- 1 file changed, 29 insertions(+), 30 deletions(-) -commit 00a037ee9c8ee5a03cf9744e05570ae93d56b875 +commit cb6b227ce39932824812ccd8a0647bd968de27d2 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-03-11 17:58:51 +0200 @@ -2456,54 +5011,43 @@ Date: 2020-03-11 17:58:51 +0200 dos/config.h | 8 ++++++++ 1 file changed, 8 insertions(+) -commit 4ec2feaefa310b4249eb41893caf526e5c51ee39 +commit 4572d53e16e87eee375bc5624de2fd59bb0ae9cd Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-03-11 22:37:54 +0200 +Date: 2020-03-02 13:54:33 +0200 - Translations: Add hu, zh_CN, and zh_TW. + liblzma: Fix a comment and RC_SYMBOLS_MAX. - I made a few white space changes to these without getting them - approved by the translation teams. (I tried to contact the hu and - zh_TW teams but didn't succeed. I didn't contact the zh_CN team.) + The comment didn't match the value of RC_SYMBOLS_MAX and the value + itself was slightly larger than actually needed. The only harm + about this was that memory usage was a few bytes larger. - po/LINGUAS | 3 + - po/hu.po | 985 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - po/zh_CN.po | 963 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - po/zh_TW.po | 956 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 4 files changed, 2907 insertions(+) + src/liblzma/rangecoder/range_encoder.h | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) -commit b6ed09729ae408be4533a0ddbc7df3d6f566846a +commit 265daa873c0d871f5f23f9b56e133a6f20045a0a Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-03-11 14:33:30 +0200 +Date: 2020-02-27 20:58:52 +0200 - Translations: Update vi.po to match the file from the TP. - - The translated strings haven't been updated but word wrapping - is different. + Build: Make CMake build fail if tuklib_cpucores or tuklib_physmem fails. - po/vi.po | 407 ++++++++++++++++++++++++++++----------------------------------- - 1 file changed, 179 insertions(+), 228 deletions(-) + CMakeLists.txt | 18 ++++++++++++++++++ + 1 file changed, 18 insertions(+) -commit 7c85e8953ced204c858101872a15183e4639e9fb +commit 7c8b904527cdbe61248c80edcc2e20d840c4fef9 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-03-11 14:18:03 +0200 +Date: 2020-02-27 20:24:27 +0200 - Translations: Add fi and pt_BR, and update de, fr, it, and pl. + Build: Add support for --no-po4a option to autogen.sh. - The German translation isn't identical to the file in - the Translation Project but the changes (white space changes - only) were approved by the translator Mario Blättermann. + Normally, if po4a isn't available, autogen.sh will return + with non-zero exit status. The option --no-po4a can be useful + when one knows that po4a isn't available but wants autogen.sh + to still return with zero exit status. - po/LINGUAS | 2 + - po/de.po | 476 ++++++++++++++-------------- - po/fi.po | 974 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - po/fr.po | 272 ++++++++-------- - po/it.po | 479 ++++++++++++---------------- - po/pl.po | 239 +++++++------- - po/pt_BR.po | 1001 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 7 files changed, 2697 insertions(+), 746 deletions(-) + autogen.sh | 11 ++++++++++- + 1 file changed, 10 insertions(+), 1 deletion(-) -commit 7da3ebc67fb5414034685ec16c7a29dad03dfa9b +commit 292a5c0f9c9b3a66f5a5c652dc46381836d4537f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-25 21:35:14 +0200 @@ -2512,42 +5056,70 @@ Date: 2020-02-25 21:35:14 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 1acc48794364606c9091cae6fa56db75a1325114 +commit 474320e9908786ba2021035f9013191e16cde08a Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-03-11 13:05:29 +0200 +Date: 2020-02-25 20:42:31 +0200 - Build: Add very limited experimental CMake support. + Build: Fix bugs in the CMake files. + + Seems that the phrase "add more quotes" from sh/bash scripting + applies to CMake as well. E.g. passing an unquoted list ${FOO} + to a function that expects one argument results in only the + first element of the list being passed as an argument and + the rest get ignored. Adding quotes helps ("${FOO}"). - This version matches CMake files in the master branch (commit - 265daa873c0d871f5f23f9b56e133a6f20045a0a) except that this omits - two source files that aren't in v5.2 and in the beginning of - CMakeLists.txt the first paragraph in the comment is slightly - different to point out possible issues in building shared liblzma. + list(INSERT ...) is weird. Inserting an empty string to an empty + variable results in empty list, but inserting it to a non-empty + variable does insert an empty element to the list. + + Since INSERT requires at least one element, + "${CMAKE_THREAD_LIBS_INIT}" needs to be quoted in CMakeLists.txt. + It might result in an empty element in the list. It seems to not + matter as empty elements consistently get ignored in that variable. + In fact, calling cmake_check_push_state() and cmake_check_pop_state() + will strip the empty elements from CMAKE_REQUIRED_LIBRARIES! + + In addition to quoting fixes, this fixes checks for the cache + variables in tuklib_cpucores.cmake and tuklib_physmem.cmake. + + Thanks to Martin Matuška for testing and reporting the problems. + These fixes aren't tested yet but hopefully they soon will be. - CMakeLists.txt | 659 ++++++++++++++++++++++++++++++++++++++++++++ - cmake/tuklib_common.cmake | 49 ++++ - cmake/tuklib_cpucores.cmake | 175 ++++++++++++ - cmake/tuklib_integer.cmake | 102 +++++++ - cmake/tuklib_mbstr.cmake | 20 ++ - cmake/tuklib_physmem.cmake | 150 ++++++++++ - cmake/tuklib_progname.cmake | 19 ++ - 7 files changed, 1174 insertions(+) + CMakeLists.txt | 52 ++++++++++++++++++++++----------------------- + cmake/tuklib_common.cmake | 8 ++++--- + cmake/tuklib_cpucores.cmake | 30 ++++++++++++++------------ + cmake/tuklib_integer.cmake | 34 +++++++++++++++-------------- + cmake/tuklib_mbstr.cmake | 6 +++--- + cmake/tuklib_physmem.cmake | 29 +++++++++++++------------ + cmake/tuklib_progname.cmake | 4 ++-- + 7 files changed, 85 insertions(+), 78 deletions(-) -commit 9acc6abea1552803c74c1486fbb10af119550772 +commit 7e3493d40eac0c3fa3d5124097745a70e15c41f6 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-02-27 20:24:27 +0200 +Date: 2020-02-24 23:38:16 +0200 - Build: Add support for --no-po4a option to autogen.sh. + Build: Add very limited experimental CMake support. - Normally, if po4a isn't available, autogen.sh will return - with non-zero exit status. The option --no-po4a can be useful - when one knows that po4a isn't available but wants autogen.sh - to still return with zero exit status. + This does *NOT* replace the Autotools-based build system in + the foreseeable future. See the comment in the beginning + of CMakeLists.txt. + + So far this has been tested only on GNU/Linux but I commit + it anyway to make it easier for others to test. Since I + haven't played much with CMake before, it's likely that + there are things that have been done in a silly or wrong + way and need to be fixed. - autogen.sh | 11 ++++++++++- - 1 file changed, 10 insertions(+), 1 deletion(-) + CMakeLists.txt | 643 ++++++++++++++++++++++++++++++++++++++++++++ + cmake/tuklib_common.cmake | 47 ++++ + cmake/tuklib_cpucores.cmake | 173 ++++++++++++ + cmake/tuklib_integer.cmake | 100 +++++++ + cmake/tuklib_mbstr.cmake | 20 ++ + cmake/tuklib_physmem.cmake | 149 ++++++++++ + cmake/tuklib_progname.cmake | 19 ++ + 7 files changed, 1151 insertions(+) -commit c8853b31545db7bd0551be85949624b1261efd47 +commit 21bd4701fca3e9002ce78bc135debca369ed8545 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-24 23:37:07 +0200 @@ -2556,7 +5128,22 @@ Date: 2020-02-24 23:37:07 +0200 m4/.gitignore | 1 + 1 file changed, 1 insertion(+) -commit 901eb4a8c992354c3ea482f5bad60a1f8ad6fcc8 +commit e094d1d0f196a91ec703e8d0055948feef349ae8 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2020-02-24 23:29:35 +0200 + + tuklib: Omit an unneeded <sys/types.h> from a tests. + + tuklib_cpucores.c and tuklib_physmem.c don't include <sys/types.h> + even via other files in this package, so clearly that header isn't + needed in the tests either (no one has reported build problems due + to a missing header in a .c file). + + m4/tuklib_cpucores.m4 | 1 - + m4/tuklib_physmem.m4 | 1 - + 2 files changed, 2 deletions(-) + +commit b3ed19a55fe99a45bd77614e149d39d18498075c Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-24 23:01:00 +0200 @@ -2568,7 +5155,7 @@ Date: 2020-02-24 23:01:00 +0200 src/liblzma/lzma/fastpos_tablegen.c | 1 - 1 file changed, 1 deletion(-) -commit ac35c9585fb734b7a19785d490c152e0b8cd4663 +commit 7b8982b29179b3c586e0456dc9ecbd4f58dcea59 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-22 14:15:07 +0200 @@ -2581,7 +5168,7 @@ Date: 2020-02-22 14:15:07 +0200 src/liblzma/api/lzma.h | 5 +++-- 2 files changed, 5 insertions(+), 3 deletions(-) -commit fb9cada7cfade1156d6277717280e05b5cd342d6 +commit 43dfe04e6209c691cf4fbe3072d4ee91271748f1 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-21 17:40:02 +0200 @@ -2594,7 +5181,7 @@ Date: 2020-02-21 17:40:02 +0200 src/liblzma/lzma/lzma_encoder_optimum_normal.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) -commit 6117955af0b9cef5acde7859e86f773692b5f43c +commit 59e6eb4840b9f52fa3a61544974017279b448216 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-21 17:01:15 +0200 @@ -2608,7 +5195,7 @@ Date: 2020-02-21 17:01:15 +0200 m4/visibility.m4 | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 1 deletion(-) -commit c2cc64d78c098834231f9cfd7d852c9cd8950d74 +commit 7fe3ef2eaa53d439cec043727ea1998f4ff0e22a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-21 16:10:44 +0200 @@ -2619,7 +5206,7 @@ Date: 2020-02-21 16:10:44 +0200 src/xz/signals.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit b6314aa275b35c714e0a191d0b2e9b6106129ea9 +commit b0a2a77d10940c42b449d47a005bfc2e50ab5db8 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-21 15:59:26 +0200 @@ -2628,7 +5215,7 @@ Date: 2020-02-21 15:59:26 +0200 src/xz/signals.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit f772a1572f723e5dc7d2d32e1d4287ac7a0da55e +commit 524c2f12c762032b819757aeda8af7c47c4cabce Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-21 01:24:18 +0200 @@ -2639,7 +5226,7 @@ Date: 2020-02-21 01:24:18 +0200 m4/tuklib_integer.m4 | 56 +++++++++++++++++++++++++++------------------------- 1 file changed, 29 insertions(+), 27 deletions(-) -commit 641042e63f665f3231c2fd1241fd3dddda3fb313 +commit 57360bb4fd79b358b36d2877db26ac828d1fdfcb Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-20 18:54:04 +0200 @@ -2653,7 +5240,33 @@ Date: 2020-02-20 18:54:04 +0200 src/common/tuklib_exit.c | 1 + 1 file changed, 1 insertion(+) -commit dbd55a69e530fec9ae866aaf6c3ccc0b4daf1f1f +commit fddd31175e74a538997a939d930462fde17d2dd4 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2020-02-18 19:12:35 +0200 + + Revert the previous commit and add a comment. + + The previous commit broke crc32_tablegen.c. + + If the whole package is built without config.h (with defines + set on the compiler command line) this should still work fine + as long as these headers conform to C99 well enough. + + src/common/tuklib_config.h | 17 ++++++++++------- + 1 file changed, 10 insertions(+), 7 deletions(-) + +commit 4e4e9fbb7e66d45319525ac224bff48fbdd0cf6e +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2020-02-17 23:37:20 +0200 + + Do not check for HAVE_CONFIG_H in tuklib_config.h. + + In XZ Utils sysdefs.h takes care of it and the required headers. + + src/common/tuklib_config.h | 15 +++++++-------- + 1 file changed, 7 insertions(+), 8 deletions(-) + +commit 2d4cef954feba82073951358466a1d614141cf33 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-16 11:18:28 +0200 @@ -2667,7 +5280,7 @@ Date: 2020-02-16 11:18:28 +0200 src/common/sysdefs.h | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) -commit 9294909861e6d22b32418467e0e988f953a82264 +commit feb9c1969bc3eb33d4ecb72cfa897f92dae84939 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-15 15:07:11 +0200 @@ -2684,7 +5297,7 @@ Date: 2020-02-15 15:07:11 +0200 configure.ac | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -commit bd09081bbdf552f730030d2fd0e5e39ccb3936af +commit 3d576cf92158d62790017ad7f2dd6dc1dd6b42bb Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-15 03:08:32 +0200 @@ -2701,7 +5314,7 @@ Date: 2020-02-15 03:08:32 +0200 configure.ac | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) -commit 1e5e08d86534aec7ca957982c7f6e90203c19e9f +commit fa792b8befaf7cb3960b655e0a9410da866d756f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-14 20:42:06 +0200 @@ -2713,7 +5326,7 @@ Date: 2020-02-14 20:42:06 +0200 po4a/po4a.conf | 2 +- 2 files changed, 5533 insertions(+), 1 deletion(-) -commit 4b1447809ffbc0d77c0ad456bd6b3afcf0b8623e +commit 6f7211b6bb47a895b47f533282dba9ee9a1b0c8b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-07 15:32:21 +0200 @@ -2749,16 +5362,16 @@ Date: 2020-02-07 15:32:21 +0200 src/xzdec/Makefile.am | 55 ++++++++++++++++++++++++++++++++---------- 8 files changed, 197 insertions(+), 45 deletions(-) -commit 882fcfdcd86525cc5c6f6d0bf0230d0089206d13 +commit 426f9e5819ff7710a5ff573a96c02940be65d52f Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-02-06 00:04:42 +0200 +Date: 2020-02-06 17:31:38 +0200 - Update THANKS (sync with the master branch). + Update THANKS. - THANKS | 3 +++ - 1 file changed, 3 insertions(+) + THANKS | 1 + + 1 file changed, 1 insertion(+) -commit 134bb7765815d5f265eb0bc9e6ebacd9ae4a52bc +commit e3a4481d020e4de89efa037f335cf50f3ca55592 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-05 22:35:06 +0200 @@ -2767,7 +5380,7 @@ Date: 2020-02-05 22:35:06 +0200 .gitignore | 4 ++++ 1 file changed, 4 insertions(+) -commit 6912472fafb656be8f4c5b4ac9ea28fea3065de4 +commit 07208de92f2d5bca764f6d0ebe9d9866051dc4ef Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-05 22:28:51 +0200 @@ -2776,7 +5389,7 @@ Date: 2020-02-05 22:28:51 +0200 m4/.gitignore | 1 + 1 file changed, 1 insertion(+) -commit 68c60735bbb6e51d4205ba8a9fde307bcfb22f8c +commit c91fbf223db46c3b3cb9df769863a1a60cd9c908 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-05 20:47:38 +0200 @@ -2785,7 +5398,20 @@ Date: 2020-02-05 20:47:38 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit e1beaa74bc7cb5a409d59b55870e01ae7784ce3a +commit 15a133b6d1a3eab4faf6eb52a71fdc56bd65846f +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2020-02-05 20:40:14 +0200 + + xz: Make it a fatal error if enabling the sandbox fails. + + Perhaps it's too drastic but on the other hand it will let me + learn about possible problems if people report the errors. + This won't be backported to the v5.2 branch. + + src/xz/file_io.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit af0fb386ef55db66654ae39e2deec6e04190c4ff Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-05 20:33:50 +0200 @@ -2794,7 +5420,7 @@ Date: 2020-02-05 20:33:50 +0200 src/xz/file_io.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) -commit 8238192652290df78bd728b20e3f6542d1a2819e +commit 986d8c9b52b824474088e5bb3b6940651660f0e2 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-05 19:33:37 +0200 @@ -2810,7 +5436,7 @@ Date: 2020-02-05 19:33:37 +0200 configure.ac | 10 ++++++++++ 1 file changed, 10 insertions(+) -commit 93a1f61e892e145607dd938e3b30098af19a1672 +commit 6629ed929cc7d45a11e385f357ab58ec15e7e4ad Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-03 22:03:50 +0200 @@ -2819,7 +5445,7 @@ Date: 2020-02-03 22:03:50 +0200 m4/ax_pthread.m4 | 398 ++++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 279 insertions(+), 119 deletions(-) -commit d0daa21792ff861e5423bbd82aaa6c8ba9fa0462 +commit 353970510895f6a80adfe60cf71b70a95adfa8bc Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-02-01 19:56:18 +0200 @@ -2855,7 +5481,7 @@ Date: 2020-02-01 19:56:18 +0200 src/xz/xz.1 | 21 ++++++++++++++++++++- 2 files changed, 51 insertions(+), 2 deletions(-) -commit 4433c2dc5727ee6aef570e001a5a024e0d94e609 +commit ba76d67585f88677af9f48b48e7bdc3bb7687def Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-01-26 20:53:25 +0200 @@ -2889,7 +5515,7 @@ Date: 2020-01-26 20:53:25 +0200 src/xz/mytime.c | 1 - 3 files changed, 6 insertions(+), 7 deletions(-) -commit acc0ef3ac80f18e349c6d0252177707105c0a29c +commit fd47fd62bbb1bfd13ab63869137971d8b390025f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-01-26 20:19:19 +0200 @@ -2902,7 +5528,7 @@ Date: 2020-01-26 20:19:19 +0200 src/xz/mytime.h | 4 ---- 5 files changed, 7 insertions(+), 9 deletions(-) -commit 4afe69d30b66812682a2016ee18441958019cbb2 +commit 815035681063d5774d3640fc20b8ede783dd574e Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-01-26 14:49:22 +0200 @@ -2916,7 +5542,7 @@ Date: 2020-01-26 14:49:22 +0200 src/xz/coder.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) -commit ec26f3ace5f9b260ca91508030f07465ae2f9f78 +commit 5a49e081a098455bcdbd95cefb90e9b18780fe58 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-01-26 14:13:42 +0200 @@ -2932,7 +5558,7 @@ Date: 2020-01-26 14:13:42 +0200 src/xz/file_io.h | 4 ++++ 3 files changed, 19 insertions(+), 4 deletions(-) -commit 38915703241e69a64f133ff9a02ec9100c6019c6 +commit dcca70fe9fa3c4bec56cf9c79e966166c9a9cf6a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-01-26 13:47:31 +0200 @@ -2941,7 +5567,7 @@ Date: 2020-01-26 13:47:31 +0200 src/xz/file_io.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) -commit f6d24245349cecfae6ec0d2366fa80716c9f6d37 +commit 4ae9ab70cd3214395756435d13d8d000368ca2cb Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-01-26 13:37:08 +0200 @@ -2950,7 +5576,7 @@ Date: 2020-01-26 13:37:08 +0200 src/xz/file_io.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) -commit 15b55d5c63d27f81776edb1abc05872a751fc674 +commit 3333ba4a6795a55cf0375329ba08152bd7fcbd46 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2020-01-26 13:27:51 +0200 @@ -2959,23 +5585,22 @@ Date: 2020-01-26 13:27:51 +0200 src/xz/file_io.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) -commit 609c7067859146ffc62ac655f6ba53599c891801 +commit cf2df0f05ac98c1158c6e48145900b773223605d Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2020-02-05 19:56:09 +0200 +Date: 2020-01-19 21:54:33 +0200 - xz: Enable Capsicum sandboxing by default if available. + Use $(LIB_FUZZING_ENGINE) in tests/ossfuzz/Makefile. - It has been enabled in FreeBSD for a while and reported to work fine. + https://github.com/google/oss-fuzz/pull/3219#issuecomment-573751048 - Thanks to Xin Li. + Thanks to Bhargava Shastry for sending the patch. - INSTALL | 6 ------ - configure.ac | 8 ++++---- - 2 files changed, 4 insertions(+), 10 deletions(-) + tests/ossfuzz/Makefile | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) -commit 00517d125cc26ecece0eebb84c1c3975cd19bee0 +commit 7136f1735c60ac6967c4b8e277fcde53d485234f Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2019-12-31 22:41:45 +0200 +Date: 2019-12-31 00:41:28 +0200 Rename unaligned_read32ne to read32ne, and similarly for the others. @@ -2996,7 +5621,7 @@ Date: 2019-12-31 22:41:45 +0200 tests/test_stream_flags.c | 6 +-- 15 files changed, 54 insertions(+), 62 deletions(-) -commit 52d89d8443c4a31a69c0701062f2c7711d82bbed +commit 5e78fcbf2eb21936022c9c5c3625d4da76f4b241 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-12-31 00:29:48 +0200 @@ -3013,7 +5638,7 @@ Date: 2019-12-31 00:29:48 +0200 src/liblzma/check/crc64_fast.c | 4 +-- 3 files changed, 32 insertions(+), 32 deletions(-) -commit 850620468b57d49f16093e5870d1050886fcb37a +commit 77bc5bc6dd67056cfd5888520ac930cfc57b4516 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-12-31 00:18:24 +0200 @@ -3040,11 +5665,11 @@ Date: 2019-12-31 00:18:24 +0200 - Cleaner bswap fallback macros - Minor cleanups - m4/tuklib_integer.m4 | 43 ++++ + m4/tuklib_integer.m4 | 46 ++++- src/common/tuklib_integer.h | 488 ++++++++++++++++++++++++-------------------- - 2 files changed, 314 insertions(+), 217 deletions(-) + 2 files changed, 316 insertions(+), 218 deletions(-) -commit a45badf0342666462cc6a7107a071418570ab773 +commit 8b72950a6b2e2a36c2d8fdc8857564b57191b088 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-12-29 22:51:58 +0200 @@ -3055,7 +5680,7 @@ Date: 2019-12-29 22:51:58 +0200 tests/test_check.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) -commit c9a8071e6690a8db8a485c075920df254e7c70ea +commit 43ce4ea7c762238d3df9717b34126d3e0d7cd51c Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-09-24 23:02:40 +0300 @@ -3078,7 +5703,7 @@ Date: 2019-09-24 23:02:40 +0300 src/scripts/xzmore.in | 1 + 6 files changed, 65 insertions(+), 8 deletions(-) -commit aba140e2df3ff63ad124ae997de16d517b98ca50 +commit 6a89e656ebedd53a10cd1a063a32a9e4ade0da1f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-07-12 18:57:43 +0300 @@ -3089,7 +5714,7 @@ Date: 2019-07-12 18:57:43 +0300 src/common/tuklib_mbstr_width.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) -commit 710f5bd769a5d2bd8684256c2727d15350ee2ab8 +commit ac0b4212656a48ef0c187c0c941d40ac9489ae36 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-07-12 18:30:46 +0300 @@ -3101,7 +5726,30 @@ Date: 2019-07-12 18:30:46 +0300 src/common/tuklib_mbstr_width.c | 1 + 1 file changed, 1 insertion(+) -commit 0e491aa8cd72e0100cd15c1b9469cd57fae500b0 +commit 72a443281fb0b91aebf8cdff2ab1f7c07b081240 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-07-12 18:10:57 +0300 + + Update tuklib base headers to include stdbool.h. + + src/common/tuklib_common.h | 2 +- + src/common/tuklib_config.h | 1 + + 2 files changed, 2 insertions(+), 1 deletion(-) + +commit de1f47b2b40e960b7bc3acba754f66dd19705921 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-06-28 00:54:31 +0300 + + xz: Automatically align the strings in --info-memory. + + This makes it easier to translate the strings. + + Also, the string for amount of RAM was shortened. + + src/xz/hardware.c | 45 ++++++++++++++++++++++++++++++++++----------- + 1 file changed, 34 insertions(+), 11 deletions(-) + +commit 8ce679125dbd0e2058d8f886e738d7f19a45cab5 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-25 23:15:21 +0300 @@ -3110,7 +5758,7 @@ Date: 2019-06-25 23:15:21 +0300 src/liblzma/lz/lz_encoder_mf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit bfc245569f340a75bd71ad32a6beba786712683b +commit fc77929e92e869f6869bf88931066103fd75f376 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-25 00:16:06 +0300 @@ -3119,7 +5767,7 @@ Date: 2019-06-25 00:16:06 +0300 configure.ac | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit f18eee9d15a22c8449ef395a05f0eb637c4ad253 +commit e873902641794210ad7db59743f98e3e1cd6139f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-25 00:08:13 +0300 @@ -3132,7 +5780,7 @@ Date: 2019-06-25 00:08:13 +0300 tests/test_stream_flags.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) -commit 25f74554723e8deabc66fed1abf0ec27a4ed19d5 +commit d499e467d99efeaae688564eedc4548837c1416a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-24 23:52:17 +0300 @@ -3141,7 +5789,7 @@ Date: 2019-06-24 23:52:17 +0300 src/liblzma/common/stream_encoder_mt.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 44eb961f2a51d02420d017bc5ff470360663650c +commit a12b13c5f0d54c684fa8446f93fdac08ab2a716b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-24 23:45:21 +0300 @@ -3151,7 +5799,28 @@ Date: 2019-06-24 23:45:21 +0300 src/liblzma/check/crc64_table.c | 3 +++ 2 files changed, 6 insertions(+) -commit 267afcd9955e668c1532b069230c21c348eb4f82 +commit 1b4675cebf7471f7cc9b7072c950e3de97147063 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-06-24 23:25:41 +0300 + + Add LZMA_RET_INTERNAL1..8 to lzma_ret and use one for LZMA_TIMED_OUT. + + LZMA_TIMED_OUT is *internally* used as a value for lzma_ret + enumeration. Previously it was #defined to 32 and cast to lzma_ret. + That way it wasn't visible in the public API, but this was hackish. + + Now the public API has eight LZMA_RET_INTERNALx members and + LZMA_TIMED_OUT is #defined to LZMA_RET_INTERNAL1. This way + the code is cleaner overall although the public API has a few + extra mysterious enum members. + + src/liblzma/api/lzma/base.h | 15 ++++++++++++++- + src/liblzma/common/common.c | 4 +--- + src/liblzma/common/common.h | 5 ++--- + src/xz/message.c | 8 ++++++++ + 4 files changed, 25 insertions(+), 7 deletions(-) + +commit 159c43875eb25deea626ed651274464bae3e32ef Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-24 22:57:43 +0300 @@ -3160,7 +5829,18 @@ Date: 2019-06-24 22:57:43 +0300 src/xz/main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 0e3c4002f809311ecef239b05e556d9c462b5703 +commit 466cfcd3e52f6750ce28a635997f3dd84fb18515 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-06-24 22:52:20 +0300 + + xz: Make "headings" static in list.c. + + Caught by clang -Wmissing-variable-declarations. + + src/xz/list.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 608517b9b76c41fac6613dbda1193d6f41338e19 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-24 22:47:39 +0300 @@ -3173,7 +5853,7 @@ Date: 2019-06-24 22:47:39 +0300 src/liblzma/lz/lz_decoder.c | 3 +-- 3 files changed, 3 insertions(+), 6 deletions(-) -commit cb708e8fa3405ec13a0ebfebbbf2793f927deab1 +commit c2d2ab6a9d41a2b55d047c5b710aacf80d219255 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-24 20:53:55 +0300 @@ -3182,7 +5862,7 @@ Date: 2019-06-24 20:53:55 +0300 tests/create_compress_files.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) -commit c8cace3d6e965c0fb537591372bf71b9357dd76c +commit 2402f7873dcae719d0ebddd23bb579074519ac52 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-24 20:45:49 +0300 @@ -3196,7 +5876,18 @@ Date: 2019-06-24 20:45:49 +0300 src/xz/file_io.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) -commit 65a42741e290fbcd85dfc5db8a62c4bce5f7712c +commit 4fd3a8dd0b60f029e1c66a0ee634f9e9fda3caa9 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-06-24 01:24:17 +0300 + + xz: Cleanup io_seek_src() a bit. + + lseek() returns -1 on error and checking for -1 is nicer. + + src/xz/file_io.c | 4 +--- + 1 file changed, 1 insertion(+), 3 deletions(-) + +commit dfda7cf6afa486e10df035327d68753896dfb48a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-24 00:57:23 +0300 @@ -3210,7 +5901,30 @@ Date: 2019-06-24 00:57:23 +0300 tests/tests.h | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) -commit 5c4fb60e8df026e933afab0cfe0a8b55be20036c +commit 1d4a904d8fb634bd5a04f7fbdd17d3739f3d8866 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-06-24 00:40:45 +0300 + + xz: Change io_seek_src and io_pread arguments from off_t to uint64_t. + + This helps fixing warnings from -Wsign-conversion and makes the + code look better too. + + src/xz/file_io.c | 16 ++++++++++++---- + src/xz/file_io.h | 4 ++-- + src/xz/list.c | 9 ++++----- + 3 files changed, 18 insertions(+), 11 deletions(-) + +commit 50120deb0159fcb53ee1a6caffb2bb81a1ecd990 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-06-24 00:12:38 +0300 + + xz: list.c: Fix some warnings from -Wsign-conversion. + + src/xz/list.c | 7 ++++--- + 1 file changed, 4 insertions(+), 3 deletions(-) + +commit d0a78751eb54fb1572002746c533936a118e4e42 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-23 23:22:45 +0300 @@ -3219,7 +5933,7 @@ Date: 2019-06-23 23:22:45 +0300 src/common/tuklib_mbstr_width.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 37df03ce52ce53710e1513387648763f8a744154 +commit 7883d73530b4b2a701ddd7d50c35676cbc158039 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-23 23:19:34 +0300 @@ -3234,7 +5948,7 @@ Date: 2019-06-23 23:19:34 +0300 src/xz/util.c | 4 ++-- 7 files changed, 14 insertions(+), 13 deletions(-) -commit 7c65ae0f5f2e2431f88621e8fe6d1dc7907e30c1 +commit c2b994fe3d35e9e575c28869a2f7f534f2495d05 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-23 22:27:45 +0300 @@ -3243,7 +5957,7 @@ Date: 2019-06-23 22:27:45 +0300 src/common/tuklib_cpucores.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) -commit a502dd1d000b598406637d452f535f4bbd43e2a4 +commit 07c4fa9e1a195e0543f271380c8de22a3ab145ff Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-23 21:40:47 +0300 @@ -3252,7 +5966,7 @@ Date: 2019-06-23 21:40:47 +0300 src/xzdec/xzdec.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit a45d1a5374ceb22e23255b0b595b9e641e9860af +commit dfac2c9a1d7d4a2b8a5d7c9c6d567dee48318bcf Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-23 21:38:56 +0300 @@ -3275,7 +5989,7 @@ Date: 2019-06-23 21:38:56 +0300 src/liblzma/simple/x86.c | 2 +- 11 files changed, 31 insertions(+), 29 deletions(-) -commit 4ff87ddf80ed7cb233444cddd86ab1940b5b55ec +commit 41838dcc26375f6aa393a63e4d81e2f4d223de07 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-23 19:33:55 +0300 @@ -3284,7 +5998,7 @@ Date: 2019-06-23 19:33:55 +0300 src/common/tuklib_integer.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) -commit ed1a9d33984a3a37ae9a775a46859850d98ea4d0 +commit 3ce05d235f736d392347a05086b8033416874b87 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-20 19:40:30 +0300 @@ -3297,7 +6011,7 @@ Date: 2019-06-20 19:40:30 +0300 src/common/tuklib_integer.h | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) -commit 612c88dfc08e2c572623954ecfde541d21c84882 +commit b525b0c0ef40cd89b69294c9b8d57f4a8db58e1f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-03 20:44:19 +0300 @@ -3306,7 +6020,7 @@ Date: 2019-06-03 20:44:19 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 85da31d8b882b8b9671ab3e3d74d88bd945cd0bb +commit 039a168e8cf201d5104a25ec41f0cf25eda6cc53 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-03 20:41:54 +0300 @@ -3322,7 +6036,7 @@ Date: 2019-06-03 20:41:54 +0300 src/liblzma/common/filter_flags_encoder.c | 2 +- 6 files changed, 7 insertions(+), 7 deletions(-) -commit 6a73a7889587aa394e236c7e9e4f870b44851036 +commit c460f6defebc5a81bbca90adc2476154ca244f69 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-02 00:50:59 +0300 @@ -3331,7 +6045,7 @@ Date: 2019-06-02 00:50:59 +0300 src/liblzma/lz/lz_encoder_hash.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit ce59b34ec9ac344d62a57cad5f94f695f42cdaee +commit c81d77c537f0b8c8672868e1dc6cf7290ce4a25b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-01 21:41:55 +0300 @@ -3340,7 +6054,7 @@ Date: 2019-06-01 21:41:55 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 94aa3fb568fe41dd4925a961966ed5cf8213bd1f +commit 386394fc9fcde2615391f804eaa466749f96f4ef Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-01 21:36:13 +0300 @@ -3351,7 +6065,7 @@ Date: 2019-06-01 21:36:13 +0300 src/liblzma/common/memcmplen.h | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) -commit 412791486dfb430219d8e30bcbebbfc57a99484a +commit 264ab971ce2994baac41b1579c9c35aba7743fc8 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-01 21:30:03 +0300 @@ -3360,7 +6074,7 @@ Date: 2019-06-01 21:30:03 +0300 src/common/tuklib_integer.h | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) -commit efbf6e5f0932e6c1a4250f91ee99059f449f2470 +commit 33773c6f2a8711d4aa6656795db52c59a28580ec Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-01 19:01:21 +0300 @@ -3374,7 +6088,19 @@ Date: 2019-06-01 19:01:21 +0300 src/liblzma/lzma/lzma_encoder_private.h | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) -commit 29afef03486d461c23f57150ac5436684bff7811 +commit e5f13a66567b1987e0aae42c6fdcd277bb5810ba +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-06-01 18:46:54 +0300 + + tuklib_integer: Autodetect support for unaligned access on ARM. + + The result is used as the default for --enable-unaligned-access. + The test should work with GCC and Clang. + + m4/tuklib_integer.m4 | 11 +++++++++++ + 1 file changed, 11 insertions(+) + +commit 3bc112c2d38d5f348bce7bc2422286b1692c7490 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-06-01 18:41:16 +0300 @@ -3392,7 +6118,7 @@ Date: 2019-06-01 18:41:16 +0300 src/common/tuklib_integer.h | 180 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 168 insertions(+), 12 deletions(-) -commit 596ed3de4485a4b1d83b5fe506ae9d0a172139b4 +commit 2a22de439ec63da1927b640eda309296a1e8dce5 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-05-13 20:05:17 +0300 @@ -3427,7 +6153,7 @@ Date: 2019-05-13 20:05:17 +0300 src/liblzma/simple/simple_coder.c | 10 +++++++++- 3 files changed, 23 insertions(+), 5 deletions(-) -commit b4b83555c576e1d845a2b98a193b23c021437804 +commit d3fc850cfedc058247d9e334ce59bbc8f2286d8a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-05-11 20:56:08 +0300 @@ -3436,7 +6162,7 @@ Date: 2019-05-11 20:56:08 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 8d4906262b45557ed164cd74adb270e6ef7f6f03 +commit 4adb8288ab61d5f14e212007b8742df0710baf73 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-05-11 20:54:12 +0300 @@ -3445,7 +6171,7 @@ Date: 2019-05-11 20:54:12 +0300 src/xz/xz.1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 0d318402f8a022f707622c72f8f1894ea476cf89 +commit 2fb0ddaa557ce86e38fe06439930fa8665f092fd Author: Antoine Cœur <antoine.coeur@ef.com> Date: 2019-05-08 13:30:57 +0800 @@ -3453,10 +6179,12 @@ Date: 2019-05-08 13:30:57 +0800 Doxyfile.in | 2 +- NEWS | 2 +- + doc/examples/11_file_info.c | 2 +- src/liblzma/api/lzma/block.h | 2 +- src/liblzma/api/lzma/hardware.h | 2 +- src/liblzma/api/lzma/lzma12.h | 2 +- src/liblzma/api/lzma/vli.h | 2 +- + src/liblzma/common/file_info.c | 4 ++-- src/liblzma/common/hardware_physmem.c | 2 +- src/liblzma/common/index.c | 4 ++-- src/liblzma/common/stream_encoder_mt.c | 2 +- @@ -3465,14 +6193,86 @@ Date: 2019-05-08 13:30:57 +0800 src/scripts/xzgrep.in | 2 +- src/xz/args.c | 2 +- src/xz/coder.c | 4 ++-- + src/xz/list.c | 4 ++-- src/xz/main.c | 2 +- src/xz/mytime.h | 2 +- src/xz/private.h | 2 +- src/xz/xz.1 | 2 +- windows/build.bash | 2 +- - 19 files changed, 21 insertions(+), 21 deletions(-) + 22 files changed, 26 insertions(+), 26 deletions(-) + +commit 4ed339606156bd313ed99237485cb8ed0362d64f +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-05-01 18:43:10 +0300 + + xz: In xz -lvv look at the widths of the check names too. + + Now the widths of the check names is used to adjust the width + of the Check column. This way there no longer is a need to restrict + the widths of the check names to be at most ten terminal-columns. + + src/xz/list.c | 32 ++++++++++++++++++++++++++------ + 1 file changed, 26 insertions(+), 6 deletions(-) + +commit 2f4281a1001dcf7fdf1418c0c0d246c16561fb65 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-05-01 18:33:25 +0300 + + xz: Fix xz -lvv column alignment to look at the translated strings. + + src/xz/list.c | 4 ++-- + 1 file changed, 2 insertions(+), 2 deletions(-) -commit aeb3be8ac4c4b06a745c3799b80b38159fb78b1a +commit 01d01b7c7c0b8eaf7f780a5584ec52c22d10fa4a +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-05-01 16:52:36 +0300 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 64030c6b17f7743df03a9948a0ccfcdf40c6b97c +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-05-01 16:43:16 +0300 + + Windows: Update VS version in windows/vs2019/config.h. + + windows/vs2019/config.h | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit 2dc9117f5fbfab31444a3ca1e55795ccfa8a9f51 +Author: Julien Marrec <julien.marrec@gmail.com> +Date: 2019-04-25 17:44:06 +0200 + + Windows: Upgrade solution itself + + windows/vs2019/xz_win.sln | 7 +++++-- + 1 file changed, 5 insertions(+), 2 deletions(-) + +commit ac31413916fa9b11bab17f0f0aa63e2869360f6f +Author: Julien Marrec <julien.marrec@gmail.com> +Date: 2019-04-25 17:40:24 +0200 + + Windows: Upgrade solution with VS2019 + + windows/vs2019/liblzma.vcxproj | 15 ++++++++------- + windows/vs2019/liblzma_dll.vcxproj | 15 ++++++++------- + 2 files changed, 16 insertions(+), 14 deletions(-) + +commit be25a0c37ba92a20c390b4d17fe502457fe96b71 +Author: Julien Marrec <julien.marrec@gmail.com> +Date: 2019-04-25 17:39:32 +0200 + + Windows: Duplicate windows/vs2017 before upgrading + + windows/vs2019/config.h | 148 ++++++++++++++ + windows/vs2019/liblzma.vcxproj | 356 ++++++++++++++++++++++++++++++++++ + windows/vs2019/liblzma_dll.vcxproj | 385 +++++++++++++++++++++++++++++++++++++ + windows/vs2019/xz_win.sln | 48 +++++ + 4 files changed, 937 insertions(+) + +commit d0e58b3a51e8e616f3dc26ec7b7e4aa0fa6991ad Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2019-03-04 22:49:04 +0200 @@ -3484,7 +6284,51 @@ Date: 2019-03-04 22:49:04 +0200 README | 32 +++++++++++++------------------- 1 file changed, 13 insertions(+), 19 deletions(-) -commit 0c238dc3feb0a3eea1e713feb8d338c8dfba9f74 +commit a750c35a7d45a16c11c1d40fecee8443c32a9996 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-03-04 21:20:39 +0200 + + xz: Automatically align column headings in xz -lvv. + + src/xz/list.c | 263 ++++++++++++++++++++++++++++++++++++++++++++++------------ + 1 file changed, 212 insertions(+), 51 deletions(-) + +commit 6cb42e8aa1dc37bf403a9f5acbd07e86036b7e77 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-03-04 01:07:59 +0200 + + xz: Automatically align strings ending in a colon in --list output. + + This should avoid alignment errors in translations with these + strings. + + src/xz/list.c | 114 +++++++++++++++++++++++++++++++++++++++++++++++++++------- + 1 file changed, 102 insertions(+), 12 deletions(-) + +commit 1e3f29b62f2c03e50fc9ebea7b83c1497dd35484 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2019-01-13 17:29:23 +0200 + + Windows/VS2017: Omit WindowsTargetPlatformVersion from project files. + + I understood that if a WTPV is specified, it's often wrong + because different VS installations have different SDK version + installed. Omitting the WTPV tag makes VS2017 default to + Windows SDK 8.1 which often is also missing, so in any case + people may need to specify the WTPV before building. But some + day in the future a missing WTPV tag will start to default to + the latest installed SDK which sounds reasonable: + + https://developercommunity.visualstudio.com/content/problem/140294/windowstargetplatformversion-makes-it-impossible-t.html + + Thanks to "dom". + + windows/INSTALL-MSVC.txt | 4 ++++ + windows/vs2017/liblzma.vcxproj | 1 - + windows/vs2017/liblzma_dll.vcxproj | 1 - + 3 files changed, 4 insertions(+), 2 deletions(-) + +commit 4d86076332aece6314063d3712a5f364172bbb0f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-12-20 20:42:29 +0200 @@ -3493,7 +6337,7 @@ Date: 2018-12-20 20:42:29 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 3ca432d9cce4bf7e793de173dd22025b68611c42 +commit b55d79461d1f6aeaac03c7dae84481e5eb8bea4c Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-12-14 20:34:30 +0200 @@ -3517,7 +6361,7 @@ Date: 2018-12-14 20:34:30 +0200 src/xz/message.h | 3 ++- 3 files changed, 25 insertions(+), 7 deletions(-) -commit fcc419e3c3f77a8b6fc5056a86b1b8abbe266e62 +commit 4ae5526de013efd1021686fa80bdd10cf1cb9c56 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-11-22 17:20:31 +0200 @@ -3526,7 +6370,7 @@ Date: 2018-11-22 17:20:31 +0200 src/xz/xz.1 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 5a2fc3cd0194e55df329dd29f805299aaca5f32f +commit 6a36d0d5f49e0080ff64dd9ef250abd489bea2ad Author: Pavel Raiskup <praiskup@redhat.com> Date: 2018-11-22 15:14:34 +0100 @@ -3536,7 +6380,52 @@ Date: 2018-11-22 15:14:34 +0100 src/xz/xz.1 | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -commit 7143b04fe49390807f355b1dad686a3d8c4dbdcf +commit 9eca51ef805ed8002a851df1b4995d71826c8b6e +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2018-11-02 20:40:48 +0200 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 106d1a663d4ba42b63231caa289f531548df81c1 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2018-11-02 20:18:45 +0200 + + Tests: Add a fuzz test program and a config file for OSS-Fuzz. + + Thanks to Bhargava Shastry and Github user pdknsk. + + tests/Makefile.am | 1 + + tests/ossfuzz/Makefile | 7 ++++ + tests/ossfuzz/config/fuzz.dict | 2 + + tests/ossfuzz/config/fuzz.options | 2 + + tests/ossfuzz/fuzz.c | 82 +++++++++++++++++++++++++++++++++++++++ + 5 files changed, 94 insertions(+) + +commit a18ae42a79a19b1394b41eb3e238139fd28012ec +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2018-10-26 22:49:10 +0300 + + liblzma: Don't verify header CRC32s if building for fuzz testing. + + FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION is #defined when liblzma + is being built for fuzz testing. + + Most fuzzed inputs would normally get rejected because of incorrect + CRC32 and the actual header decoding code wouldn't get fuzzed. + Disabling CRC32 checks avoids this problem. The fuzzer program + must still use LZMA_IGNORE_CHECK flag to disable verification of + integrity checks of uncompressed data. + + src/liblzma/common/block_header_decoder.c | 5 ++++- + src/liblzma/common/index_decoder.c | 5 ++++- + src/liblzma/common/index_hash.c | 5 ++++- + src/liblzma/common/stream_flags_decoder.c | 10 ++++++++-- + 4 files changed, 20 insertions(+), 5 deletions(-) + +commit f76f7516d6a1c832f61810c82e92d151cc80966c Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-07-27 18:10:44 +0300 @@ -3550,7 +6439,7 @@ Date: 2018-07-27 18:10:44 +0300 src/scripts/xzless.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 273c33297bb69621045ed19665eaf8338bcf4a50 +commit 3cbcaeb07eb7543735befd6f507fdb5fa4363cff Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-07-27 16:02:58 +0300 @@ -3564,7 +6453,7 @@ Date: 2018-07-27 16:02:58 +0300 src/liblzma/common/index.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -commit 65b4aba6d06d2cd24ba9ad01fa389c238ad8f352 +commit 76762ae6098ec55c326f4b4b4a42e8c1918ee81f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-05-19 21:23:25 +0300 @@ -3573,98 +6462,44 @@ Date: 2018-05-19 21:23:25 +0300 src/liblzma/api/lzma/filter.h | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) -commit 531e78e5a253a3e2c4d4dd1505acaccee48f4083 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2019-05-01 16:52:36 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 905de7e93528ca5a47039e7e1e5270163f9fc67e +commit 2267f5b0d20a5d24e93fcd9f72ea7eeb0d89708c Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2019-05-01 16:43:16 +0300 - - Windows: Update VS version in windows/vs2019/config.h. - - windows/vs2019/config.h | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 0ffd30e172fd18cc619823b2a86448bf56a67e22 -Author: Julien Marrec <julien.marrec@gmail.com> -Date: 2019-04-25 17:44:06 +0200 - - Windows: Upgrade solution itself - - windows/vs2019/xz_win.sln | 7 +++++-- - 1 file changed, 5 insertions(+), 2 deletions(-) - -commit c2ef96685fc7ca36311649eeb2284b9808292040 -Author: Julien Marrec <julien.marrec@gmail.com> -Date: 2019-04-25 17:40:24 +0200 - - Windows: Upgrade solution with VS2019 - - windows/vs2019/liblzma.vcxproj | 15 ++++++++------- - windows/vs2019/liblzma_dll.vcxproj | 15 ++++++++------- - 2 files changed, 16 insertions(+), 14 deletions(-) - -commit 25fccaf00bea399d8aa026e5b8fa254ce196e6e0 -Author: Julien Marrec <julien.marrec@gmail.com> -Date: 2019-04-25 17:39:32 +0200 +Date: 2018-04-29 18:58:19 +0300 - Windows: Duplicate windows/vs2017 before upgrading + Bump the version number to 5.3.1alpha. - windows/vs2019/config.h | 148 ++++++++++++++ - windows/vs2019/liblzma.vcxproj | 354 ++++++++++++++++++++++++++++++++++ - windows/vs2019/liblzma_dll.vcxproj | 383 +++++++++++++++++++++++++++++++++++++ - windows/vs2019/xz_win.sln | 48 +++++ - 4 files changed, 933 insertions(+) + src/liblzma/api/lzma/version.h | 2 +- + src/liblzma/liblzma.map | 2 +- + 2 files changed, 2 insertions(+), 2 deletions(-) -commit 1424078d6328291c7c524b64328ce9660617cb24 +commit cee3021d30704858e4bdd22240e7d28e570d7451 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2019-01-13 17:29:23 +0200 +Date: 2018-04-29 18:48:00 +0300 - Windows/VS2017: Omit WindowsTargetPlatformVersion from project files. - - I understood that if a WTPV is specified, it's often wrong - because different VS installations have different SDK version - installed. Omitting the WTPV tag makes VS2017 default to - Windows SDK 8.1 which often is also missing, so in any case - people may need to specify the WTPV before building. But some - day in the future a missing WTPV tag will start to default to - the latest installed SDK which sounds reasonable: - - https://developercommunity.visualstudio.com/content/problem/140294/windowstargetplatformversion-makes-it-impossible-t.html - - Thanks to "dom". + extra/scanlzma: Fix compiler warnings. - windows/INSTALL-MSVC.txt | 4 ++++ - windows/vs2017/liblzma.vcxproj | 1 - - windows/vs2017/liblzma_dll.vcxproj | 1 - - 3 files changed, 4 insertions(+), 2 deletions(-) + extra/scanlzma/scanlzma.c | 6 +++++- + 1 file changed, 5 insertions(+), 1 deletion(-) -commit b5be61cc06088bb07f488f9baf7d447ff47b37c1 +commit c5c7ceb08a011b97d261798033e2c39613a69eb7 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2018-04-29 19:00:06 +0300 +Date: 2018-04-29 18:44:47 +0300 - Bump version and soname for 5.2.4. + DOS: Add file_info.c to the list of files to build. - src/liblzma/Makefile.am | 2 +- - src/liblzma/api/lzma/version.h | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) + dos/Makefile | 1 + + 1 file changed, 1 insertion(+) -commit c47fa6d06745bb2e99866e76b81ac7a9c5a8bfec +commit 114cab97af766b21e0fc8620479202fb1e7a5e41 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2018-04-29 18:48:00 +0300 +Date: 2018-04-29 18:33:10 +0300 - extra/scanlzma: Fix compiler warnings. + Update NEWS for 5.3.1alpha. - extra/scanlzma/scanlzma.c | 6 +++++- - 1 file changed, 5 insertions(+), 1 deletion(-) + NEWS | 11 +++++++++++ + 1 file changed, 11 insertions(+) -commit 7b350fe21aa4fd6495a3b6188a40e3f1ae7c0edf +commit b8139e11c512bbf32bf58ab0689f9bb6c52819da Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-04-29 18:15:37 +0300 @@ -3673,7 +6508,7 @@ Date: 2018-04-29 18:15:37 +0300 NEWS | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) -commit 5801591162a280aa52d156dfde42c531ec7fd8b6 +commit 47b59d47cfd904a420fbd45629d168ca1973721d Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-02-06 19:36:30 +0200 @@ -3682,7 +6517,7 @@ Date: 2018-02-06 19:36:30 +0200 THANKS | 2 ++ 1 file changed, 2 insertions(+) -commit c4a616f4536146f8906e1b4412eefeec07b28fae +commit bc197991690ede24ab143665b5b0f0f9cb35cc46 Author: Ben Boeckel <mathstuf@gmail.com> Date: 2018-01-29 13:58:18 -0500 @@ -3694,7 +6529,7 @@ Date: 2018-01-29 13:58:18 -0500 src/liblzma/api/lzma.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) -commit 0b8947782ff3c5ef830a7f85412e44dcf3cdeb77 +commit fb6d4f83cb6e144734f2a4216bb117bd56dc3cb5 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-02-06 18:02:48 +0200 @@ -3708,7 +6543,7 @@ Date: 2018-02-06 18:02:48 +0200 src/liblzma/rangecoder/range_common.h | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) -commit 48f3b9f73ffea7f55d5678997aba0e79d2e82168 +commit bc577d35c2d0ed17f554d2d8107b2a2a9abbac76 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-01-10 22:10:39 +0200 @@ -3717,7 +6552,7 @@ Date: 2018-01-10 22:10:39 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit a3ce3e902342be37c626a561ce3d9ffcf27d0f94 +commit 713bbc1a80f26d34c96ed3dbb9887362204de3a1 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2018-01-10 21:54:27 +0200 @@ -3728,7 +6563,7 @@ Date: 2018-01-10 21:54:27 +0200 src/common/tuklib_integer.h | 11 +++++++++++ 1 file changed, 11 insertions(+) -commit 4505ca483985f88c6923c05a43b4327feaab83b1 +commit a0ee1afbd99da138b559cb27fa2022e7f1ab44f3 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-09-24 20:04:24 +0300 @@ -3737,7 +6572,7 @@ Date: 2017-09-24 20:04:24 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 1ef3cc226e3ce173575c218238b71a4eecabc470 +commit a1e2c568de29c0b57d873eab40a2879b749da429 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-09-16 20:36:20 +0300 @@ -3752,7 +6587,17 @@ Date: 2017-09-16 20:36:20 +0300 windows/vs2017/liblzma_dll.vcxproj | 24 ++++++++++++------------ 4 files changed, 36 insertions(+), 36 deletions(-) -commit e775d2a8189d24f60470e6e49d8af881df3a1680 +commit cea5cf8d26c9d1dc30a808614d79c0b25640e15e +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2017-09-16 12:56:20 +0300 + + Windows: Update VS2017 project files to include file info decoder. + + windows/vs2017/liblzma.vcxproj | 2 ++ + windows/vs2017/liblzma_dll.vcxproj | 2 ++ + 2 files changed, 4 insertions(+) + +commit 95d563db3ee497b223e522b699c4d4c29943eef0 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-09-16 12:54:23 +0300 @@ -3766,7 +6611,17 @@ Date: 2017-09-16 12:54:23 +0300 windows/vs2017/xz_win.sln | 48 +++++ 4 files changed, 935 insertions(+) -commit 10e02e0fbb6e2173f8b41f6e39b7b570f47dd74d +commit ab72416d62ea8f50ad31d5b8545fcb6a2bf96b73 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2017-09-16 12:45:50 +0300 + + Windows: Update VS2013 project files to include file info decoder. + + windows/vs2013/liblzma.vcxproj | 2 ++ + windows/vs2013/liblzma_dll.vcxproj | 2 ++ + 2 files changed, 4 insertions(+) + +commit 82388980187b0e3794d187762054200bbdcc9a53 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-09-16 12:39:43 +0300 @@ -3778,7 +6633,7 @@ Date: 2017-09-16 12:39:43 +0300 windows/{ => vs2013}/xz_win.sln | 0 4 files changed, 279 insertions(+), 279 deletions(-) -commit 06eebd4543196ded36fa9b8b9544195b38b24ef2 +commit 94e3f986aa4e14b4ff01ac24857f499630d6d180 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-08-14 20:08:33 +0300 @@ -3788,7 +6643,7 @@ Date: 2017-08-14 20:08:33 +0300 src/xz/list.c | 2 ++ 2 files changed, 8 insertions(+) -commit ea4ea1dffafebaa8b2770bf3eca46900e4dd22dc +commit 0b0e1e6803456aac641a59332200f8e95e2b7ea8 Author: Alexey Tourbin <alexey.tourbin@gmail.com> Date: 2017-05-16 23:56:35 +0300 @@ -3797,7 +6652,7 @@ Date: 2017-05-16 23:56:35 +0300 doc/examples/02_decompress.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit eb2ef4c79bf405ea0d215f3b1df3d0eaf5e1d27b +commit a015cd1f90116e655be4eaf4aad42c4c911c2807 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-05-23 18:34:43 +0300 @@ -3815,7 +6670,17 @@ Date: 2017-05-23 18:34:43 +0300 src/xz/list.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) -commit 3ea5dbd9b0d79048e336e40cef3b6d814fb74e13 +commit c2e29f06a7d1e3ba242ac2fafc69f5d6e92f62cd +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2017-04-24 20:20:11 +0300 + + Docs: Add doc/examples/11_file_info.c. + + doc/examples/11_file_info.c | 206 ++++++++++++++++++++++++++++++++++++++++++++ + doc/examples/Makefile | 3 +- + 2 files changed, 208 insertions(+), 1 deletion(-) + +commit 1520f6ec808896375ac7bf778c449e0f7dea5f46 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-04-24 19:48:47 +0300 @@ -3826,7 +6691,28 @@ Date: 2017-04-24 19:48:47 +0300 Makefile.am | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -commit bae24675936df99064de1502593c006bd902594b +commit 8269782283806c90a8509c2ac2a308344f70e171 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2017-04-24 19:48:23 +0300 + + xz: Use lzma_file_info_decoder() for --list. + + src/xz/list.c | 254 ++++++++++------------------------------------------------ + 1 file changed, 44 insertions(+), 210 deletions(-) + +commit e353d0b1cc0d3997ae5048faa8e6786414953e06 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2017-04-24 19:35:50 +0300 + + liblzma: Add lzma_file_info_decoder(). + + src/liblzma/api/lzma/index.h | 66 ++++ + src/liblzma/common/Makefile.inc | 1 + + src/liblzma/common/file_info.c | 855 ++++++++++++++++++++++++++++++++++++++++ + src/liblzma/liblzma.map | 7 +- + 4 files changed, 928 insertions(+), 1 deletion(-) + +commit 144ef9e19e9496c995b21505dd1e111c442968d1 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-04-24 19:30:22 +0300 @@ -3835,7 +6721,18 @@ Date: 2017-04-24 19:30:22 +0300 ChangeLog | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 70f479211973b5361f4d7cb08ba5be69b4266e7a +commit 8c9842c265993d7dd4039f732d3546267fb5ecc4 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2017-04-21 15:05:16 +0300 + + liblzma: Rename LZMA_SEEK to LZMA_SEEK_NEEDED and seek_in to seek_pos. + + src/liblzma/api/lzma/base.h | 18 +++++++++--------- + src/liblzma/common/common.c | 2 +- + src/xz/message.c | 2 +- + 3 files changed, 11 insertions(+), 11 deletions(-) + +commit 662b27c417cab248cb365dd7682121bdec4d5ae7 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-04-19 22:17:35 +0300 @@ -3853,7 +6750,17 @@ Date: 2017-04-19 22:17:35 +0300 windows/config.h | 2 +- 10 files changed, 14 insertions(+), 14 deletions(-) -commit 2a4b2fa75d06a097261a02ecd3cf2b6d449bf754 +commit c28f0b3d00af87b92dda229831548d8eb0067d1d +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2017-04-05 18:47:22 +0300 + + xz: Add io_seek_src(). + + src/xz/file_io.c | 20 +++++++++++++++++--- + src/xz/file_io.h | 13 +++++++++++++ + 2 files changed, 30 insertions(+), 3 deletions(-) + +commit bba477257d7319c8764890f3669175b866d24944 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-03-30 22:01:54 +0300 @@ -3865,7 +6772,34 @@ Date: 2017-03-30 22:01:54 +0300 src/xz/file_io.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) -commit eb25743ade39170cffd9566a1aae272098cce216 +commit 310d19816d1652b0c8bb1b82574d46345d924752 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2017-03-30 20:03:05 +0300 + + liblzma: Make lzma_index_decoder_init() visible to other liblzma funcs. + + This is to allow other functions to use it without going + via the public API (lzma_index_decoder()). + + src/liblzma/common/Makefile.inc | 1 + + src/liblzma/common/index_decoder.c | 10 +++++----- + src/liblzma/common/index_decoder.h | 24 ++++++++++++++++++++++++ + 3 files changed, 30 insertions(+), 5 deletions(-) + +commit a27920002dbc469f778a134fc665b7c3ea73701b +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2017-03-30 20:00:09 +0300 + + liblzma: Add generic support for input seeking (LZMA_SEEK). + + Also mention LZMA_SEEK in xz/message.c to silence a warning. + + src/liblzma/api/lzma/base.h | 31 ++++++++++++++++++++++++++++++- + src/liblzma/common/common.c | 12 +++++++++++- + src/xz/message.c | 1 + + 3 files changed, 42 insertions(+), 2 deletions(-) + +commit a0b1dda409bc3e6e2957a2651663fc411d2caf2d Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-03-30 19:47:45 +0300 @@ -3890,7 +6824,7 @@ Date: 2017-03-30 19:47:45 +0300 src/liblzma/common/common.c | 6 ++++-- 3 files changed, 13 insertions(+), 3 deletions(-) -commit ef36c6362f3f3853f21b8a6359bcd06576ebf207 +commit 84462afaada61379f5878e46f8f00e25a1cdcf29 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-03-30 19:16:55 +0300 @@ -3902,7 +6836,7 @@ Date: 2017-03-30 19:16:55 +0300 src/liblzma/common/stream_decoder.c | 5 +---- 4 files changed, 20 insertions(+), 16 deletions(-) -commit 57616032650f03840480b696d7878acdd2065521 +commit cbc74017939690d13441b8926bb743fb03211b83 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2017-03-30 18:58:18 +0300 @@ -3926,17 +6860,7 @@ Date: 2017-03-30 18:58:18 +0300 src/liblzma/common/index_decoder.c | 4 ++-- 2 files changed, 13 insertions(+), 9 deletions(-) -commit 3d566cd519017eee1a400e7961ff14058dfaf33c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2016-12-30 13:26:36 +0200 - - Bump version and soname for 5.2.3. - - src/liblzma/Makefile.am | 2 +- - src/liblzma/api/lzma/version.h | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) - -commit 053e624fe33795e779ff736f16ce44a129c829b5 +commit 78ae13bced912b1b92ae927992c99cbcc463cae7 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-12-30 13:25:10 +0200 @@ -3945,25 +6869,16 @@ Date: 2016-12-30 13:25:10 +0200 NEWS | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) -commit cae412b2b77d7fd88d187ed7659331709311f80d +commit 0297863fdb453aed1a25eb025f3ba7bacbbb1357 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2015-04-01 14:45:25 +0300 +Date: 2016-12-26 20:55:52 +0200 - xz: Fix the Capsicum rights on user_abort_pipe. - - src/xz/file_io.c | 6 +++++- - 1 file changed, 5 insertions(+), 1 deletion(-) - -commit 9ccbae41000572193b9a09e7102f9e84dc6d96de -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2016-12-28 21:05:22 +0200 + Document --enable-sandbox configure option in INSTALL. - Mention potential sandboxing bugs in INSTALL. + INSTALL | 23 +++++++++++++++++++++++ + 1 file changed, 23 insertions(+) - INSTALL | 5 ++++- - 1 file changed, 4 insertions(+), 1 deletion(-) - -commit e013a337d3de77cce24360dffe956ea2339489b6 +commit d4a0462abe5478193521c14625e1c81fead87f9f Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-11-21 20:24:50 +0200 @@ -4014,118 +6929,7 @@ Date: 2016-11-21 20:24:50 +0200 src/liblzma/simple/x86.c | 15 +-- 35 files changed, 532 insertions(+), 423 deletions(-) -commit 8e0f1af3dcaec00a3879cce8ad7441edc6359d1c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2016-12-26 20:50:25 +0200 - - Document --enable-sandbox configure option in INSTALL. - - INSTALL | 25 +++++++++++++++++++++++++ - 1 file changed, 25 insertions(+) - -commit ce2542d220de06acd618fd9f5c0a6683029fb4eb -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2015-03-31 22:19:34 +0300 - - xz: Add support for sandboxing with Capsicum (disabled by default). - - In the v5.2 branch this feature is considered experimental - and thus disabled by default. - - The sandboxing is used conditionally as described in main.c. - This isn't optimal but it was much easier to implement than - a full sandboxing solution and it still covers the most common - use cases where xz is writing to standard output. This should - have practically no effect on performance even with small files - as fork() isn't needed. - - C and locale libraries can open files as needed. This has been - fine in the past, but it's a problem with things like Capsicum. - io_sandbox_enter() tries to ensure that various locale-related - files have been loaded before cap_enter() is called, but it's - possible that there are other similar problems which haven't - been seen yet. - - Currently Capsicum is available on FreeBSD 10 and later - and there is a port to Linux too. - - Thanks to Loganaden Velvindron for help. - - configure.ac | 41 +++++++++++++++++++++++++++ - src/xz/Makefile.am | 2 +- - src/xz/file_io.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ - src/xz/file_io.h | 6 ++++ - src/xz/main.c | 18 ++++++++++++ - src/xz/private.h | 4 +++ - 6 files changed, 151 insertions(+), 1 deletion(-) - -commit 3ca1d5e6320111043e19434da881065fadafa0e4 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2015-03-31 21:12:30 +0300 - - Fix bugs and otherwise improve ax_check_capsicum.m4. - - AU_ALIAS was removed because the new version is incompatible - with the old version. - - It no longer checks for <sys/capability.h> separately. - It's enough to test for it as part of AC_CHECK_DECL. - The defines HAVE_CAPSICUM_SYS_CAPSICUM_H and - HAVE_CAPSICUM_SYS_CAPABILITY_H were removed as unneeded. - HAVE_SYS_CAPSICUM_H from AC_CHECK_HEADERS is enough. - - It no longer does a useless search for the Capsicum library - if the header wasn't found. - - Fixed a bug in ACTION-IF-FOUND (the first argument). Specifying - the argument omitted the default action but the given action - wasn't used instead. - - AC_DEFINE([HAVE_CAPSICUM]) is now always called when Capsicum - support is found. Previously it was part of the default - ACTION-IF-FOUND which a custom action would override. Now - the default action only prepends ${CAPSICUM_LIB} to LIBS. - - The documentation was updated. - - Since there as no serial number, "#serial 2" was added. - - m4/ax_check_capsicum.m4 | 103 ++++++++++++++++++++++++------------------------ - 1 file changed, 51 insertions(+), 52 deletions(-) - -commit 5f3a742b64197fe8bedb6f05fc6ce5d177d11145 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2015-03-31 19:20:24 +0300 - - Add m4/ax_check_capsicum.m4 for detecting Capsicum support. - - The file was loaded from this web page: - https://github.com/google/capsicum-test/blob/dev/autoconf/m4/ax_check_capsicum.m4 - - Thanks to Loganaden Velvindron for pointing it out for me. - - m4/ax_check_capsicum.m4 | 86 +++++++++++++++++++++++++++++++++++++++++++++++++ - 1 file changed, 86 insertions(+) - -commit d74377e62b4c649e40294dd441de72c0f092e67c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2015-10-12 20:29:09 +0300 - - liblzma: Fix a memory leak in error path of lzma_index_dup(). - - lzma_index_dup() calls index_dup_stream() which, in case of - an error, calls index_stream_end() to free memory allocated - by index_stream_init(). However, it illogically didn't - actually free the memory. To make it logical, the tree - handling code was modified a bit in addition to changing - index_stream_end(). - - Thanks to Evan Nemerson for the bug report. - - src/liblzma/common/index.c | 18 +++++++++--------- - 1 file changed, 9 insertions(+), 9 deletions(-) - -commit f580732216dcf971f3f006fe8e01cd4979e1d964 +commit a01794c52add98263b49119842c3e7141d1b9ced Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-10-24 18:53:25 +0300 @@ -4134,7 +6938,7 @@ Date: 2016-10-24 18:53:25 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 88d7a7fd153bf1355cdf798ffdac7443d0169afc +commit df8f446e3ad47e5148b8c8d8b6e519d3ce29cb9d Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-10-24 18:51:36 +0300 @@ -4158,7 +6962,7 @@ Date: 2016-10-24 18:51:36 +0300 src/common/tuklib_cpucores.c | 9 +++++++++ 2 files changed, 38 insertions(+), 1 deletion(-) -commit 51baf684376903dbeddd840582bfdf9fa91b311b +commit 446e4318fa79788e09299d5953b5dd428953d14b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-06-30 20:27:36 +0300 @@ -4174,7 +6978,7 @@ Date: 2016-06-30 20:27:36 +0300 src/xz/file_io.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+), 1 deletion(-) -commit 1ddc479851139d6e8202e5835421bfe6578d9e07 +commit 1b0ac0c53c761263e91e34195cb21dfdcfeac0bd Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-06-16 22:46:02 +0300 @@ -4185,7 +6989,7 @@ Date: 2016-06-16 22:46:02 +0300 src/xz/file_io.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) -commit be647ff5ed5a1c244a65722af6ce250259f3b14a +commit c83b7a03342c3325ff10400b22ee21edfcd1e026 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-04-10 20:55:49 +0300 @@ -4196,7 +7000,7 @@ Date: 2016-04-10 20:55:49 +0300 src/xz/Makefile.am | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit fb6d50c15343831f35305982cefa82053099191d +commit ade31a2bfb95c94d05fbfc0ecbba5d6377f2506e Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-04-10 20:54:17 +0300 @@ -4210,7 +7014,7 @@ Date: 2016-04-10 20:54:17 +0300 configure.ac | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -commit 74f8dad9f912a2993768d93d108ea2b0b2c196e0 +commit ac398c3bafa6e4c80e20571373a96947db863b3d Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-03-13 20:21:49 +0200 @@ -4240,7 +7044,7 @@ Date: 2016-03-13 20:21:49 +0200 src/liblzma/check/check.h | 16 ++++------ 3 files changed, 79 insertions(+), 49 deletions(-) -commit ea7f6ff04cb5bb1498088eb09960a4c3f13dfe39 +commit 6fd5ecb589a9fdd7a576ea48c4850d496bab9ce5 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-03-10 20:27:05 +0200 @@ -4249,7 +7053,7 @@ Date: 2016-03-10 20:27:05 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit d0e018016b311232e82d9a98dc68f1e3dabce794 +commit 473ef0dc69a30e64d5fa0d34aca02f7309faa3e9 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2016-03-10 20:26:49 +0200 @@ -4272,7 +7076,7 @@ Date: 2016-03-10 20:26:49 +0200 configure.ac | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) -commit 5daae123915f32a4ed6dc948b831533c2d1beec3 +commit faf302137e54d605b44ecf0373cb51a6403a2de1 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-08 20:16:10 +0200 @@ -4281,7 +7085,7 @@ Date: 2015-11-08 20:16:10 +0200 src/common/tuklib_physmem.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -commit 491acc406e098167ccb7fce0728b94c2f32cff9f +commit e52e9151cf8613022d1de4712ff39dbcb666e991 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-04 23:17:43 +0200 @@ -4290,7 +7094,7 @@ Date: 2015-11-04 23:17:43 +0200 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 8173ff8790ad3502d04e1c07d014cb84a3b8187b +commit 14115f84a38161d55eaa2d070f08739bde37e966 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-04 23:14:00 +0200 @@ -4307,7 +7111,7 @@ Date: 2015-11-04 23:14:00 +0200 src/liblzma/lz/lz_encoder.c | 4 ++++ 1 file changed, 4 insertions(+) -commit 013de2b5ab8094d2c82a2771f3d143eeb656eda9 +commit f4c95ba94beb71a608eb6eadbf82b44f53a0260e Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-03 20:55:45 +0200 @@ -4320,7 +7124,7 @@ Date: 2015-11-03 20:55:45 +0200 src/liblzma/lzma/{lzma_presets.c => lzma_encoder_presets.c} | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -commit a322f70ad96de88968c2c36e6a36bc08ae30bd20 +commit c7bc20a6f3e71920871d48db31a79ab58b5a0a4b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-03 20:47:07 +0200 @@ -4332,7 +7136,7 @@ Date: 2015-11-03 20:47:07 +0200 configure.ac | 3 +++ 1 file changed, 3 insertions(+) -commit 8ea49606cf6427e32319de7693eca9e43f1c8ad6 +commit 5cbca1205deeb6fb7afe7a864fa68a57466d928a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-03 20:35:19 +0200 @@ -4341,7 +7145,7 @@ Date: 2015-11-03 20:35:19 +0200 configure.ac | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) -commit 42131a25e52bfe400acfa7df93469a96bb78bb78 +commit af13781886c8e7a0aabebb5141ea282dc364f5c6 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-03 20:31:31 +0200 @@ -4350,7 +7154,7 @@ Date: 2015-11-03 20:31:31 +0200 windows/config.h | 6 ++++++ 1 file changed, 6 insertions(+) -commit e9184e87cc989d14c7413e6adb3eca98f6ae0290 +commit 9fa5949330f162c5a2f6653f83025327837e8f39 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-03 20:29:58 +0200 @@ -4359,7 +7163,7 @@ Date: 2015-11-03 20:29:58 +0200 dos/config.h | 6 ++++++ 1 file changed, 6 insertions(+) -commit 2296778f3c9a1e3a8699973b09dd3610b8baa402 +commit cb3111e3ed84152912b5138d690c8d9f00c6ef02 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-03 20:29:33 +0200 @@ -4379,7 +7183,7 @@ Date: 2015-11-03 20:29:33 +0200 src/xz/private.h | 5 ++++- 6 files changed, 62 insertions(+), 13 deletions(-) -commit 97a3109281e475d9cf1b5095237d672fa0ad25e5 +commit 4cc584985c0b7a13901da1b7a64ef9f7cc36e8ab Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-03 18:06:40 +0200 @@ -4396,7 +7200,7 @@ Date: 2015-11-03 18:06:40 +0200 src/liblzma/lzma/{lzma_encoder_presets.c => lzma_presets.c} | 3 ++- 2 files changed, 7 insertions(+), 2 deletions(-) -commit dc6b78d7f0f6fe43e9d4215146e8581feb8090e7 +commit 23ed1d41489f632155bbc9660f323d57e09da180 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-03 17:54:48 +0200 @@ -4409,7 +7213,7 @@ Date: 2015-11-03 17:54:48 +0200 configure.ac | 5 ----- 1 file changed, 5 deletions(-) -commit 46d76c9cd3cb26a31f5ae6c3a8bbcf38e6da1add +commit b0bc3e03852af13419ea2960881824258d451117 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-03 17:41:54 +0200 @@ -4422,7 +7226,7 @@ Date: 2015-11-03 17:41:54 +0200 src/liblzma/common/Makefile.inc | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) -commit 16d68f874d89f1e4a1919786a35bbaef7d71a077 +commit c6bf438ab39e0fb4a47d3c81725c227919502726 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-02 18:16:51 +0200 @@ -4435,7 +7239,7 @@ Date: 2015-11-02 18:16:51 +0200 src/liblzma/check/check.h | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) -commit d9311647fc1ab512a3394596221ab8039c00af6b +commit e18adc56f2262aa9394d46681e9e4b9981ed5e97 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-11-02 15:19:10 +0200 @@ -4447,7 +7251,7 @@ Date: 2015-11-02 15:19:10 +0200 src/xz/file_io.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) -commit f59c4183f3c9066626ce45dc3db4642fa603fa21 +commit 282e768a1484e88c8b7ec35655ee4959954ec87a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-10-12 21:08:42 +0300 @@ -4456,7 +7260,7 @@ Date: 2015-10-12 21:08:42 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit 35f189673e280c12e4c5129f9f97e54eef3bbc04 +commit 372e402713a1d4337ffce5f56d5c5c9ed99a66d0 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-10-12 21:07:41 +0300 @@ -4465,7 +7269,7 @@ Date: 2015-10-12 21:07:41 +0300 tests/test_index.c | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) -commit e10bfdb0fcaff12f3a6dadee51e0a022aadccb51 +commit 21515d79d778b8730a434f151b07202d52a04611 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-10-12 20:45:15 +0300 @@ -4477,7 +7281,7 @@ Date: 2015-10-12 20:45:15 +0300 src/liblzma/common/index.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) -commit 06f434bd8980f25ca23232eb7bb7df7e37dc8448 +commit 09f395b6b360c0b13e8559eece1d179b908ebd3a Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-10-12 20:31:44 +0300 @@ -4486,17 +7290,25 @@ Date: 2015-10-12 20:31:44 +0300 src/liblzma/common/index.c | 3 +++ 1 file changed, 3 insertions(+) -commit 9815cdf6987ef91a85493bfcfd1ce2aaf3b47a0a +commit 3bf857edfef51374f6f3fffae3d817f57d3264a0 Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2015-09-29 13:59:35 +0300 +Date: 2015-10-12 20:29:09 +0300 - Bump version and soname for 5.2.2. + liblzma: Fix a memory leak in error path of lzma_index_dup(). + + lzma_index_dup() calls index_dup_stream() which, in case of + an error, calls index_stream_end() to free memory allocated + by index_stream_init(). However, it illogically didn't + actually free the memory. To make it logical, the tree + handling code was modified a bit in addition to changing + index_stream_end(). + + Thanks to Evan Nemerson for the bug report. - src/liblzma/Makefile.am | 2 +- - src/liblzma/api/lzma/version.h | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) + src/liblzma/common/index.c | 18 +++++++++--------- + 1 file changed, 9 insertions(+), 9 deletions(-) -commit cbe0cec8476bdd0416c7ca9bc83895c9bea1cf78 +commit 7f05803979b4b79642d5be4218a79da7a0b12c47 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-09-29 13:57:28 +0300 @@ -4505,21 +7317,7 @@ Date: 2015-09-29 13:57:28 +0300 NEWS | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) -commit 49427ce7eececdd18bbd35dab23c81910d083e1c -Author: Andre Noll <maan@tuebingen.mpg.de> -Date: 2015-05-28 15:50:00 +0200 - - Fix typo in German translation. - - As pointed out by Robert Pollak, there's a typo in the German - translation of the compression preset option (-0 ... -9) help text. - "The compressor" translates to "der Komprimierer", and the genitive - form is "des Komprimierers". The old word makes no sense at all. - - po/de.po | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 608d6f06c940e7f28c25de005e8b99bdff42d27c +commit 397fcc0946315b55c3c6d80e37e82a2a78bc15c1 Author: Hauke Henningsen <sqrt@entless.org> Date: 2015-08-17 04:59:54 +0200 @@ -4548,7 +7346,7 @@ Date: 2015-08-17 04:59:54 +0200 po/de.po | 383 ++++++++++++++++++++++++++++++++------------------------------- 1 file changed, 196 insertions(+), 187 deletions(-) -commit c8988414e5b67b8ef2fe0ba7b1ccdd0ec73c60d3 +commit cbc9e39bae715accb44168930a71888480aad569 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-08-11 13:23:04 +0300 @@ -4560,7 +7358,7 @@ Date: 2015-08-11 13:23:04 +0300 configure.ac | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) -commit 85a6dfed53477906bfe9a7c0123dd412e391cb48 +commit bcacd8ce7a031566858e5e03c1009064c3f1c89e Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-08-11 13:21:52 +0300 @@ -4569,7 +7367,7 @@ Date: 2015-08-11 13:21:52 +0300 configure.ac | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) -commit 77f270be8432df2e4516a0c48814b6976d6618c5 +commit 0275a5398c01d57b724dec7fea52dec3bd6edc6c Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-08-09 21:06:26 +0300 @@ -4583,7 +7381,7 @@ Date: 2015-08-09 21:06:26 +0300 windows/liblzma_dll.vcxproj | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) -commit 8c975446c5903090a5a8493b5b96b71003056a88 +commit a74525cf9b945fb0b370e64cf406104beb31729b Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-08-09 21:02:20 +0300 @@ -4593,7 +7391,7 @@ Date: 2015-08-09 21:02:20 +0300 windows/liblzma_dll.vcxproj | 5 ----- 2 files changed, 10 deletions(-) -commit 119a00434954726ca58e4a578e6469f530fca30e +commit fbbb295a91caf39faf8838c8c39526e4cb4dc121 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-07-12 20:48:19 +0300 @@ -4602,7 +7400,7 @@ Date: 2015-07-12 20:48:19 +0300 src/liblzma/api/lzma.h | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) -commit d4e7c557fcab353539c9481a8d95cb04bcb15c7c +commit 713dbe5c230fe00865a54f5c32358ea30f9a1156 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-06-19 20:38:55 +0300 @@ -4611,7 +7409,7 @@ Date: 2015-06-19 20:38:55 +0300 THANKS | 2 ++ 1 file changed, 2 insertions(+) -commit 98001740ca56c894a7bd32eb47e9857a8a7d878d +commit 3a5d755d055d51f99c523b4c2952727e1e69cfa1 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-06-19 20:21:30 +0300 @@ -4622,7 +7420,7 @@ Date: 2015-06-19 20:21:30 +0300 windows/{INSTALL-Windows.txt => INSTALL-MinGW.txt} | 2 +- 3 files changed, 67 insertions(+), 11 deletions(-) -commit 28195e4c877007cc760ecea1d17f740693d66873 +commit b0798c6aa6184efcefd0bdcca20f96121a13feda Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-06-19 17:25:31 +0300 @@ -4635,7 +7433,21 @@ Date: 2015-06-19 17:25:31 +0300 windows/xz_win.sln | 48 ++++++ 3 files changed, 795 insertions(+) -commit 960440f3230dc628f6966d9f7614fc1b28baf44e +commit 9b02a4ffdac1b9f066658ec4c95c0834f4cd2fb7 +Author: Andre Noll <maan@tuebingen.mpg.de> +Date: 2015-05-28 15:50:00 +0200 + + Fix typo in German translation. + + As pointed out by Robert Pollak, there's a typo in the German + translation of the compression preset option (-0 ... -9) help text. + "The compressor" translates to "der Komprimierer", and the genitive + form is "des Komprimierers". The old word makes no sense at all. + + po/de.po | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +commit c7f4041f6b8f4729f88d3bc888b2a4080ae51f72 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-05-13 20:57:55 +0300 @@ -4646,7 +7458,7 @@ Date: 2015-05-13 20:57:55 +0300 tests/test_bcj_exact_size.c | 1 + 1 file changed, 1 insertion(+) -commit 68cd35acafbdcdf4e8ea8b5bb843c736939d6f8b +commit 17b29d4f0ae0f780fbd69e15a398dc478d8492f8 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-05-12 18:08:24 +0300 @@ -4657,7 +7469,7 @@ Date: 2015-05-12 18:08:24 +0300 NEWS | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -commit ff96ed6d25786728356017a13baf8c14731b4f1e +commit 49c26920d6e2d85e5c6123e34958aed2e77485ad Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-05-11 21:26:16 +0300 @@ -4666,7 +7478,7 @@ Date: 2015-05-11 21:26:16 +0300 src/xz/xz.1 | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) -commit 00d37b64a64ea8597fd2422d5187afd761ab9531 +commit 5b2458cb244ed237efe4de1ebcf06e1b3a1f4256 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-04-20 20:20:29 +0300 @@ -4675,7 +7487,7 @@ Date: 2015-04-20 20:20:29 +0300 THANKS | 1 + 1 file changed, 1 insertion(+) -commit db190a832c49ca3aed6d69cc992fa5583cae7b11 +commit 6bd0349c58451b13442e8f463e35de83548bf985 Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-04-20 19:59:18 +0300 @@ -4693,6 +7505,118 @@ Date: 2015-04-20 19:59:18 +0300 src/xz/file_io.c | 9 +-------- 2 files changed, 3 insertions(+), 10 deletions(-) +commit fc0df0f8db87dff45543708a711e17d29c37f632 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2015-04-01 14:45:25 +0300 + + xz: Fix the Capsicum rights on user_abort_pipe. + + src/xz/file_io.c | 6 +++++- + 1 file changed, 5 insertions(+), 1 deletion(-) + +commit 57393615b31b3788dd77280452d845bcc12d33af +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2015-03-31 22:20:11 +0300 + + Update THANKS. + + THANKS | 1 + + 1 file changed, 1 insertion(+) + +commit 1238381143a9a7ce84839c2582ccd56ff750a440 +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2015-03-31 22:19:34 +0300 + + xz: Add support for sandboxing with Capsicum. + + The sandboxing is used conditionally as described in main.c. + This isn't optimal but it was much easier to implement than + a full sandboxing solution and it still covers the most common + use cases where xz is writing to standard output. This should + have practically no effect on performance even with small files + as fork() isn't needed. + + C and locale libraries can open files as needed. This has been + fine in the past, but it's a problem with things like Capsicum. + io_sandbox_enter() tries to ensure that various locale-related + files have been loaded before cap_enter() is called, but it's + possible that there are other similar problems which haven't + been seen yet. + + Currently Capsicum is available on FreeBSD 10 and later + and there is a port to Linux too. + + Thanks to Loganaden Velvindron for help. + + configure.ac | 41 +++++++++++++++++++++++++++ + src/xz/Makefile.am | 2 +- + src/xz/file_io.c | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ + src/xz/file_io.h | 6 ++++ + src/xz/main.c | 18 ++++++++++++ + src/xz/private.h | 4 +++ + 6 files changed, 151 insertions(+), 1 deletion(-) + +commit 29a087fb5a0c879f0b1bc4c6b989f7b87bacdf9e +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2015-03-31 21:12:30 +0300 + + Fix bugs and otherwise improve ax_check_capsicum.m4. + + AU_ALIAS was removed because the new version is incompatible + with the old version. + + It no longer checks for <sys/capability.h> separately. + It's enough to test for it as part of AC_CHECK_DECL. + The defines HAVE_CAPSICUM_SYS_CAPSICUM_H and + HAVE_CAPSICUM_SYS_CAPABILITY_H were removed as unneeded. + HAVE_SYS_CAPSICUM_H from AC_CHECK_HEADERS is enough. + + It no longer does a useless search for the Capsicum library + if the header wasn't found. + + Fixed a bug in ACTION-IF-FOUND (the first argument). Specifying + the argument omitted the default action but the given action + wasn't used instead. + + AC_DEFINE([HAVE_CAPSICUM]) is now always called when Capsicum + support is found. Previously it was part of the default + ACTION-IF-FOUND which a custom action would override. Now + the default action only prepends ${CAPSICUM_LIB} to LIBS. + + The documentation was updated. + + Since there as no serial number, "#serial 2" was added. + + m4/ax_check_capsicum.m4 | 103 ++++++++++++++++++++++++------------------------ + 1 file changed, 51 insertions(+), 52 deletions(-) + +commit 6e845c6a3eddf2fde9db5a29950421dff60a43ac +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2015-03-31 19:20:24 +0300 + + Add m4/ax_check_capsicum.m4 for detecting Capsicum support. + + The file was loaded from this web page: + https://github.com/google/capsicum-test/blob/dev/autoconf/m4/ax_check_capsicum.m4 + + Thanks to Loganaden Velvindron for pointing it out for me. + + m4/ax_check_capsicum.m4 | 86 +++++++++++++++++++++++++++++++++++++++++++++++++ + 1 file changed, 86 insertions(+) + +commit 3717885f9ef2c06f1bcbad9f4c2ed2d5695f844e +Author: Lasse Collin <lasse.collin@tukaani.org> +Date: 2015-03-30 22:44:02 +0300 + + Bump version to 5.3.0alpha and soname to 5.3.99. + + The idea of 99 is that it looks a bit weird in this context. + For new features there's no API/ABI stability in devel versions. + + src/liblzma/Makefile.am | 2 +- + src/liblzma/api/lzma/version.h | 6 +++--- + 2 files changed, 4 insertions(+), 4 deletions(-) + commit eccd8155e107c5ada03d13e7730675cdf1a44ddc Author: Lasse Collin <lasse.collin@tukaani.org> Date: 2015-03-29 22:14:47 +0300 @@ -5076,4187 +8000,3 @@ Date: 2014-12-21 18:11:17 +0200 src/liblzma/api/lzma/version.h | 6 +++--- src/liblzma/liblzma.map | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) - -commit 3e8bd1d15e417f2d588e9be50ce027ee3d48b2da -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-21 18:05:03 +0200 - - Avoid variable-length arrays in the debug programs. - - debug/full_flush.c | 3 ++- - debug/sync_flush.c | 3 ++- - 2 files changed, 4 insertions(+), 2 deletions(-) - -commit 72f7307cfdceb941aeb2bf30d424cc0d13621786 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-21 18:01:45 +0200 - - Build: Include 04_compress_easy_mt.c in the tarball. - - Makefile.am | 1 + - 1 file changed, 1 insertion(+) - -commit 2cb82ff21c62def11f3683a8bb0aaf363102aaa0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-21 18:00:38 +0200 - - Fix build when --disable-threads is used. - - src/common/mythread.h | 2 ++ - 1 file changed, 2 insertions(+) - -commit 9b9e3536e458ef958f66b0e8982efc9d36de4d17 -Author: Adrien Nader <adrien@notk.org> -Date: 2014-12-21 15:56:15 +0100 - - po/fr: improve wording for help for --lzma1/--lzma2. - - po/fr.po | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit a8b6b569e7fadbf5b5b9139d53bc764015c15027 -Author: Adrien Nader <adrien@notk.org> -Date: 2014-12-21 15:55:48 +0100 - - po/fr: missing line in translation of --extreme. - - po/fr.po | 1 + - 1 file changed, 1 insertion(+) - -commit f168a6fd1a888cf4f0caaddcafcb21dadc6ab6e9 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-21 14:32:33 +0200 - - Update NEWS for 5.2.0. - - NEWS | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 1 file changed, 65 insertions(+) - -commit cec2ee863b3a88f4bf039cb00f73c4a4fc93a429 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-21 14:32:22 +0200 - - Update NEWS for 5.0.8. - - NEWS | 12 ++++++++++++ - 1 file changed, 12 insertions(+) - -commit 42e97a32649bf53ce43be2258b902a417c6e7fa1 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-21 14:07:54 +0200 - - xz: Fix a comment. - - src/xz/options.c | 4 ++-- - 1 file changed, 2 insertions(+), 2 deletions(-) - -commit 29b95d5d6665cedffa6a9d6d3d914f981e852182 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-20 20:43:14 +0200 - - Update INSTALL about the dependencies of the scripts. - - INSTALL | 15 ++++++++++----- - 1 file changed, 10 insertions(+), 5 deletions(-) - -commit 3af91040bb42c21afbb81f5568c3313125e61192 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-20 20:42:33 +0200 - - Windows: Update build instructions. - - INSTALL | 15 +++++++++------ - windows/INSTALL-Windows.txt | 44 +++++++++++++++++++++----------------------- - 2 files changed, 30 insertions(+), 29 deletions(-) - -commit 0152f72bf6289d744823dc6c849538f3a139ad70 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-20 20:41:48 +0200 - - Windows: Update the build script and README-Windows.txt. - - The 32-bit build is now for i686 or newer because the - prebuilt MinGW-w64 toolchains include i686 code in the - executables even if one uses -march=i486. - - The build script builds 32-bit SSE2 enabled version too. - Run-time detection of SSE2 support would be nice (on any OS) - but it's not implemented in XZ Utils yet. - - windows/README-Windows.txt | 30 ++++++++++++++++-------------- - windows/build.bash | 23 ++++++++++++++--------- - 2 files changed, 30 insertions(+), 23 deletions(-) - -commit 4a1f6133ee5533cee8d91e06fcc22443e5f1881a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-19 15:51:50 +0200 - - Windows: Define TUKLIB_SYMBOL_PREFIX in config.h. - - It is to keep all symbols in the lzma_ namespace. - - windows/config.h | 3 +++ - 1 file changed, 3 insertions(+) - -commit 7f7d093de79eee0c7dbfd7433647e46302f19f82 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-16 21:00:09 +0200 - - xz: Update the man page about --threads. - - src/xz/xz.1 | 5 ----- - 1 file changed, 5 deletions(-) - -commit 009823448b82aa5f465668878a544c5842885407 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-16 20:57:43 +0200 - - xz: Update the man page about --block-size. - - src/xz/xz.1 | 41 +++++++++++++++++++++++++++++++++-------- - 1 file changed, 33 insertions(+), 8 deletions(-) - -commit 7dddfbeb499e528940bc12047355c184644aafe9 -Author: Adrien Nader <adrien@notk.org> -Date: 2014-12-10 22:26:57 +0100 - - po/fr: several more translation updates: reword and handle --ignore-check. - - po/fr.po | 50 ++++++++++++++++++++++++++------------------------ - 1 file changed, 26 insertions(+), 24 deletions(-) - -commit 6eca5be40e04ddc4b738d493e4e56835956d8b69 -Author: Adrien Nader <adrien@notk.org> -Date: 2014-12-10 22:23:01 +0100 - - po/fr: yet another place where my email address had to be updated. - - po/fr.po | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit d1003673e92ba47edd6aeeb3dbea05c18269d0e7 -Author: Adrien Nader <adrien@notk.org> -Date: 2014-12-10 22:22:20 +0100 - - po/fr: fix several typos that have been around since the beginning. - - po/fr.po | 22 +++++++++++----------- - 1 file changed, 11 insertions(+), 11 deletions(-) - -commit 4c5aa911a0df027e46171e368debc543d2fa72b2 -Author: Adrien Nader <adrien@notk.org> -Date: 2014-12-03 20:02:31 +0100 - - po/fr: last batch of new translations for now. - - Four new error messages. - - po/fr.po | 16 ++++++++-------- - 1 file changed, 8 insertions(+), 8 deletions(-) - -commit 3e3099e36d27059499e7996fb38a62e8ab01d356 -Author: Adrien Nader <adrien@notk.org> -Date: 2014-12-03 20:01:32 +0100 - - po/fr: translations for --threads, --block-size and --block-list. - - po/fr.po | 12 ++++++++---- - 1 file changed, 8 insertions(+), 4 deletions(-) - -commit e7d96a5933eec4e9d4a62569ee88df0ebb0f1d53 -Author: Adrien Nader <adrien@notk.org> -Date: 2014-12-03 20:00:53 +0100 - - po/fr: remove fuzzy marker for error messages that will be kept in English. - - The following is a copy of a comment inside fr.po: - - Note from translator on "file status flags". - The following entry is kept un-translated on purpose. It is difficult to - translate and should only happen in exceptional circumstances which means - that translating would: - - lose some of the meaning - - make it more difficult to look up in search engines; it might happen one - in - a million times, if we dilute the error message in 20 languages, it will be - almost impossible to find an explanation and support for the error. - - po/fr.po | 22 ++++++++++++++++------ - 1 file changed, 16 insertions(+), 6 deletions(-) - -commit 46cbb9033af8a21fafe543302d6919746e0d72af -Author: Adrien Nader <adrien@notk.org> -Date: 2014-12-03 19:58:25 +0100 - - po/fr: several minor updates and better wording. - - Meaning doesn't change at all: it's only for better wording and/or - formatting of a few strings. - - po/fr.po | 14 +++++++------- - 1 file changed, 7 insertions(+), 7 deletions(-) - -commit 7ce49d444f04e73145f79c832eb4d510594b074a -Author: Adrien Nader <adrien@notk.org> -Date: 2014-12-03 19:56:12 +0100 - - po/fr: update my email address and copyright years. - - po/fr.po | 4 ++-- - 1 file changed, 2 insertions(+), 2 deletions(-) - -commit 214c553ebc3047cd720da1ce5c80cf7c38118d3c -Author: Adrien Nader <adrien@notk.org> -Date: 2014-11-26 10:08:26 +0100 - - fr.po: commit file after only "update-po" so actual is readable. - - po/fr.po | 311 ++++++++++++++++++++++++++++++++++++++++----------------------- - 1 file changed, 199 insertions(+), 112 deletions(-) - -commit 1190c641af09cde85f8bd0fbe5c4906f4a29431b -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-12-02 20:04:07 +0200 - - liblzma: Document how lzma_mt.block_size affects memory usage. - - src/liblzma/api/lzma/container.h | 4 ++++ - 1 file changed, 4 insertions(+) - -commit e4fc1d2f9571fba79ce383595be2ea2a9257def0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-28 20:07:18 +0200 - - Update INSTALL about a "make check" failure in test_scripts.sh. - - INSTALL | 24 +++++++++++++++++------- - 1 file changed, 17 insertions(+), 7 deletions(-) - -commit 34f9e40a0a0c3bd2c2730cdb9cd550bbb8a3f2fe -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-26 20:12:27 +0200 - - Remove LZMA_UNSTABLE macro. - - src/liblzma/api/lzma/container.h | 4 ---- - src/liblzma/common/common.h | 2 -- - src/xz/private.h | 1 - - 3 files changed, 7 deletions(-) - -commit 6d9c0ce9f2677b159e32b224aba5b535b304a705 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-26 20:10:33 +0200 - - liblzma: Update lzma_stream_encoder_mt() API docs. - - src/liblzma/api/lzma/container.h | 5 +++-- - 1 file changed, 3 insertions(+), 2 deletions(-) - -commit 2301f3f05dd9742f42cda8f0f318864f5dc39ab3 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-25 12:32:05 +0200 - - liblzma: Verify the filter chain in threaded encoder initialization. - - This way an invalid filter chain is detected at the Stream - encoder initialization instead of delaying it to the first - call to lzma_code() which triggers the initialization of - the actual filter encoder(s). - - src/liblzma/common/stream_encoder_mt.c | 9 ++++++--- - 1 file changed, 6 insertions(+), 3 deletions(-) - -commit 107a263d5bb63cd3593fd6a5c938706539f84523 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-17 19:11:49 +0200 - - Build: Update m4/ax_pthread.m4 from Autoconf Archive. - - m4/ax_pthread.m4 | 71 +++++++++++++++++++++++++++++++++++++------------------- - 1 file changed, 47 insertions(+), 24 deletions(-) - -commit b13a781833399ff5726cfc997f3cb2f0acbdbf31 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-17 18:52:21 +0200 - - Build: Replace obsolete AC_HELP_STRING with AS_HELP_STRING. - - configure.ac | 36 ++++++++++++++++++------------------ - m4/tuklib_integer.m4 | 2 +- - 2 files changed, 19 insertions(+), 19 deletions(-) - -commit 542cac122ed3550148a2af0033af22b757491378 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-17 18:43:19 +0200 - - Build: Fix Autoconf warnings about escaped backquotes. - - Thanks to Daniel Richard G. for pointing out that it's - good to sometimes run autoreconf -fi with -Wall. - - configure.ac | 7 +++---- - 1 file changed, 3 insertions(+), 4 deletions(-) - -commit 7b03a15cea8cd4f19ed680b51c4bcbae3ce4142f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-10 18:54:40 +0200 - - xzdiff: Use mkdir if mktemp isn't available. - - src/scripts/xzdiff.in | 17 ++++++++++++++++- - 1 file changed, 16 insertions(+), 1 deletion(-) - -commit f8c13e5e3609581d5dd9f8777985ca07f2390ad7 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-10 18:45:01 +0200 - - xzdiff: Create a temporary directory to hold a temporary file. - - This avoids the possibility of "File name too long" when - creating a temp file when the input file name is very long. - - This also means that other users on the system can no longer - see the input file names in /tmp (or whatever $TMPDIR is) - since the temporary directory will have a generic name. This - usually doesn't matter since on many systems one can see - the arguments given to all processes anyway. - - The number X chars to mktemp where increased from 6 to 10. - - Note that with some shells temp files or dirs won't be used at all. - - src/scripts/xzdiff.in | 10 +++++----- - 1 file changed, 5 insertions(+), 5 deletions(-) - -commit 7716dcf9df7f457500cb657314e7a9aea5fedb06 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-10 15:38:47 +0200 - - liblzma: Fix lzma_mt.preset in lzma_stream_encoder_mt_memusage(). - - It read the filter chain from a wrong variable. This is a similar - bug that was fixed in 9494fb6d0ff41c585326f00aa8f7fe58f8106a5e. - - src/liblzma/common/stream_encoder_mt.c | 3 +-- - 1 file changed, 1 insertion(+), 2 deletions(-) - -commit 230fa4a605542c84b4178a57381695a0af4e779b -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-11-10 14:49:55 +0200 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 4e4ae08bc7c1711e399c9f2d26eb375d39d08101 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-10-29 21:28:25 +0200 - - Update .gitignore files. - - .gitignore | 2 ++ - m4/.gitignore | 3 +++ - 2 files changed, 5 insertions(+) - -commit c923b140b27d1a055db6284e10fd546ad1a7fcdb -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-10-29 21:15:35 +0200 - - Build: Prepare to support Automake's subdir-objects. - - Due to a bug in Automake, subdir-objects won't be enabled - for now. - - http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17354 - - Thanks to Daniel Richard G. for the original patches. - - configure.ac | 7 ++++++- - src/Makefile.am | 22 +++++++++++++++++++++- - src/liblzma/Makefile.am | 4 ++-- - src/lzmainfo/Makefile.am | 4 ++-- - src/xz/Makefile.am | 10 +++++----- - src/xzdec/Makefile.am | 8 ++++---- - 6 files changed, 40 insertions(+), 15 deletions(-) - -commit 08c2aa16bea0df82828f665d51fba2e0a5e8997f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-10-24 20:09:29 +0300 - - Translations: Update the Italian translation. - - Thanks to Milo Casagrande. - - po/it.po | 452 ++++++++++++++++++++++++++++++++++++++------------------------- - 1 file changed, 275 insertions(+), 177 deletions(-) - -commit 2f9f61aa83539c54ff6c118a2693890f0519b3dd -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-10-18 18:51:45 +0300 - - Translations: Update the Polish translation. - - Thanks to Jakub Bogusz. - - po/pl.po | 332 ++++++++++++++++++++++++++++++++++++++++----------------------- - 1 file changed, 214 insertions(+), 118 deletions(-) - -commit 4f9d233f67aea25e532824d11b7642cf7dee7a76 -Author: Andre Noll <maan@tuebingen.mpg.de> -Date: 2014-10-14 17:30:30 +0200 - - l10n: de.po: Change translator email address. - - Although the old address is still working, the new one should - be preferred. So this commit changes all three places in de.po - accordingly. - - Signed-off-by: Andre Noll <maan@tuebingen.mpg.de> - - po/de.po | 6 +++--- - 1 file changed, 3 insertions(+), 3 deletions(-) - -commit 00502b2bedad43f0cc167ac17ae0608837ee196b -Author: Andre Noll <maan@tuebingen.mpg.de> -Date: 2014-10-14 17:30:29 +0200 - - l10n: de.po: Update German translation - - Signed-off-by: Andre Noll <maan@systemlinux.org> - - po/de.po | 531 +++++++++++++++++++++++++++++++++------------------------------ - 1 file changed, 281 insertions(+), 250 deletions(-) - -commit 706b0496753fb609e69f1570ec603f11162189d1 -Author: Andre Noll <maan@tuebingen.mpg.de> -Date: 2014-10-14 17:30:28 +0200 - - l10n: de.po: Fix typo: Schießen -> Schließen. - - That's a funny one since "schießen" means to shoot :) - - Signed-off-by: Andre Noll <maan@systemlinux.org> - - po/de.po | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 7c32e6a935c3d7ee366abad1679bd5f322f0c7d4 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-10-09 19:42:26 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 076258cc458f1e705041ac7a729b15ffe8c5214a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-10-09 19:41:51 +0300 - - Add support for AmigaOS/AROS to tuklib_physmem(). - - Thanks to Fredrik Wikstrom. - - m4/tuklib_physmem.m4 | 3 ++- - src/common/tuklib_physmem.c | 7 +++++++ - 2 files changed, 9 insertions(+), 1 deletion(-) - -commit efa7b0a210e1baa8e128fc98c5443a944c39ad24 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-10-09 18:42:14 +0300 - - xzgrep: Avoid passing both -q and -l to grep. - - The behavior of grep -ql varies: - - GNU grep behaves like grep -q. - - OpenBSD grep behaves like grep -l. - - POSIX doesn't make it 100 % clear what behavior is expected. - Anyway, using both -q and -l at the same time makes no sense - so both options simply should never be used at the same time. - - Thanks to Christian Weisgerber. - - src/scripts/xzgrep.in | 6 ++++-- - 1 file changed, 4 insertions(+), 2 deletions(-) - -commit 9c5f76098c9986b48d2fc574a0b764f4cde0c538 -Author: Trần Ngọc Quân <vnwildman@gmail.com> -Date: 2014-09-25 09:22:45 +0700 - - l10n: vi.po: Update Vietnamese translation - - Signed-off-by: Trần Ngọc Quân <vnwildman@gmail.com> - - po/vi.po | 136 +++++++++++++++++++++++++++++++++++++++------------------------ - 1 file changed, 84 insertions(+), 52 deletions(-) - -commit c4911f2db36d811896c73c008b4218d8fa9a4730 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-09-25 18:38:48 +0300 - - Build: Detect supported compiler warning flags better. - - Clang and nowadays also GCC accept any -Wfoobar option - but then may give a warning that an unknown warning option - was specified. To avoid adding unsupported warning options, - the options are now tested with -Werror. - - Thanks to Charles Diza. - - configure.ac | 5 +++-- - 1 file changed, 3 insertions(+), 2 deletions(-) - -commit 76e75522ed6f5c228d55587dee5a997893f6e474 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-09-20 21:01:21 +0300 - - Update NEWS for 5.0.7. - - NEWS | 11 +++++++++++ - 1 file changed, 11 insertions(+) - -commit d62028b4c1174fc67b6929f126f5eb24c018c700 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-09-20 19:42:56 +0300 - - liblzma: Fix a portability problem in Makefile.am. - - POSIX supports $< only in inference rules (suffix rules). - Using it elsewhere is a GNU make extension and doesn't - work e.g. with OpenBSD make. - - Thanks to Christian Weisgerber for the patch. - - src/liblzma/Makefile.am | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit c35de31d4283edad3e57d37ffe939406542cb7bb -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-09-14 21:54:09 +0300 - - Bump the version number to 5.1.4beta. - - src/liblzma/api/lzma/version.h | 4 ++-- - src/liblzma/liblzma.map | 2 +- - 2 files changed, 3 insertions(+), 3 deletions(-) - -commit e9e097e22cacdaa23e5414fea7913535449cb340 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-09-14 21:50:13 +0300 - - Update NEWS for 5.0.6 and 5.1.4beta. - - NEWS | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ - 1 file changed, 50 insertions(+) - -commit 642f856bb8562ab66704b1e01ac7bc08b6d0a663 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-09-14 21:02:41 +0300 - - Update TODO. - - TODO | 38 ++++++++++++++++++++++++++++++++++---- - 1 file changed, 34 insertions(+), 4 deletions(-) - -commit 6b5e3b9eff5b8cedb2aac5f524d4d60fc8a48124 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-08-05 22:32:36 +0300 - - xz: Add --ignore-check. - - src/xz/args.c | 7 +++++++ - src/xz/args.h | 1 + - src/xz/coder.c | 10 +++++++++- - src/xz/message.c | 2 ++ - src/xz/xz.1 | 19 +++++++++++++++++++ - 5 files changed, 38 insertions(+), 1 deletion(-) - -commit 9adbc2ff373f979c917cdfd3679ce0ebd59f1040 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-08-05 22:15:07 +0300 - - liblzma: Add support for LZMA_IGNORE_CHECK. - - src/liblzma/api/lzma/container.h | 24 ++++++++++++++++++++++++ - src/liblzma/common/common.h | 1 + - src/liblzma/common/stream_decoder.c | 14 ++++++++++++-- - 3 files changed, 37 insertions(+), 2 deletions(-) - -commit 0e0f34b8e4f1c60ecaec15c2105982381cc9c3e6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-08-05 22:03:30 +0300 - - liblzma: Add support for lzma_block.ignore_check. - - Note that this slightly changes how lzma_block_header_decode() - has been documented. Earlier it said that the .version is set - to the lowest required value, but now it says that the .version - field is kept unchanged if possible. In practice this doesn't - affect any old code, because before this commit the only - possible .version was 0. - - src/liblzma/api/lzma/block.h | 50 ++++++++++++++++++++++++------- - src/liblzma/common/block_buffer_encoder.c | 2 +- - src/liblzma/common/block_decoder.c | 18 ++++++++--- - src/liblzma/common/block_encoder.c | 2 +- - src/liblzma/common/block_header_decoder.c | 12 ++++++-- - src/liblzma/common/block_header_encoder.c | 2 +- - src/liblzma/common/block_util.c | 2 +- - 7 files changed, 68 insertions(+), 20 deletions(-) - -commit 71e1437ab585b46f7a25f5a131557d3d1c0cbaa2 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-08-04 19:25:58 +0300 - - liblzma: Use lzma_memcmplen() in the BT3 match finder. - - I had missed this when writing the commit - 5db75054e900fa06ef5ade5f2c21dffdd5d16141. - - Thanks to Jun I Jin. - - src/liblzma/lz/lz_encoder_mf.c | 5 ++--- - 1 file changed, 2 insertions(+), 3 deletions(-) - -commit 41dc9ea06e1414ebe8ef52afc8fc15b6e3282b04 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-08-04 00:25:44 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 5dcffdbcc23a68abc3ac3539b30be71bc9b5af84 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-08-03 21:32:25 +0300 - - liblzma: SHA-256: Optimize the Maj macro slightly. - - The Maj macro is used where multiple things are added - together, so making Maj a sum of two expressions allows - some extra freedom for the compiler to schedule the - instructions. - - I learned this trick from - <http://www.hackersdelight.org/corres.txt>. - - src/liblzma/check/sha256.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit a9477d1e0c6fd0e47e637d051e7b9e2a5d9af517 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-08-03 21:08:12 +0300 - - liblzma: SHA-256: Optimize the way rotations are done. - - This looks weird because the rotations become sequential, - but it helps quite a bit on both 32-bit and 64-bit x86: - - - It requires fewer instructions on two-operand - instruction sets like x86. - - - It requires one register less which matters especially - on 32-bit x86. - - I hope this doesn't hurt other archs. - - I didn't invent this idea myself, but I don't remember where - I saw it first. - - src/liblzma/check/sha256.c | 17 +++++++++++------ - 1 file changed, 11 insertions(+), 6 deletions(-) - -commit 5a76c7c8ee9a0afbeedb1c211db9224260404347 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-08-03 20:38:13 +0300 - - liblzma: SHA-256: Remove the GCC #pragma that became unneeded. - - The unrolling in the previous commit should avoid the - situation where a compiler may think that an uninitialized - variable might be accessed. - - src/liblzma/check/sha256.c | 5 ----- - 1 file changed, 5 deletions(-) - -commit 9a096f8e57509775c331950b8351bbca77bdcfa8 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-08-03 20:33:38 +0300 - - liblzma: SHA-256: Unroll a little more. - - This way a branch isn't needed for each operation - to choose between blk0 and blk2, and still the code - doesn't grow as much as it would with full unrolling. - - src/liblzma/check/sha256.c | 25 ++++++++++++++++--------- - 1 file changed, 16 insertions(+), 9 deletions(-) - -commit bc7650d87bf27f85f1a2a806dc2db1780e09e6a5 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-08-03 19:56:43 +0300 - - liblzma: SHA-256: Do the byteswapping without a temporary buffer. - - src/liblzma/check/sha256.c | 13 +------------ - 1 file changed, 1 insertion(+), 12 deletions(-) - -commit 544aaa3d13554e8640f9caf7db717a96360ec0f6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-07-25 22:38:28 +0300 - - liblzma: Use lzma_memcmplen() in normal mode of LZMA. - - Two locations were not changed yet because the simplest change - assumes that the initial "len" may be greater than "limit". - - src/liblzma/lzma/lzma_encoder_optimum_normal.c | 20 +++++--------------- - 1 file changed, 5 insertions(+), 15 deletions(-) - -commit f48fce093b07aeda95c18850f5e086d9f2383380 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-07-25 22:30:38 +0300 - - liblzma: Simplify LZMA fast mode code by using memcmp(). - - src/liblzma/lzma/lzma_encoder_optimum_fast.c | 11 +---------- - 1 file changed, 1 insertion(+), 10 deletions(-) - -commit 6bf5308e34e23dede5b301b1b9b4f131dacd9218 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-07-25 22:29:49 +0300 - - liblzma: Use lzma_memcmplen() in fast mode of LZMA. - - src/liblzma/lzma/lzma_encoder_optimum_fast.c | 6 +++--- - 1 file changed, 3 insertions(+), 3 deletions(-) - -commit 353212137e51e45b105a3a3fc2e6879f1cf0d492 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-07-25 21:16:23 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 5db75054e900fa06ef5ade5f2c21dffdd5d16141 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-07-25 21:15:07 +0300 - - liblzma: Use lzma_memcmplen() in the match finders. - - This doesn't change the match finder output. - - src/liblzma/lz/lz_encoder.c | 13 ++++++++++++- - src/liblzma/lz/lz_encoder_mf.c | 33 +++++++++++---------------------- - 2 files changed, 23 insertions(+), 23 deletions(-) - -commit e1c8f1d01f4a4e2136173edab2dc63c71ef038f4 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-07-25 20:57:20 +0300 - - liblzma: Add lzma_memcmplen() for fast memory comparison. - - This commit just adds the function. Its uses will be in - separate commits. - - This hasn't been tested much yet and it's perhaps a bit early - to commit it but if there are bugs they should get found quite - quickly. - - Thanks to Jun I Jin from Intel for help and for pointing out - that string comparison needs to be optimized in liblzma. - - configure.ac | 13 +++ - src/liblzma/common/Makefile.inc | 1 + - src/liblzma/common/memcmplen.h | 170 ++++++++++++++++++++++++++++++++++++++++ - 3 files changed, 184 insertions(+) - -commit 765735cf52e5123586e74a51b9c073b5257f631f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-07-12 21:10:09 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 59da01785ef66c7e62f36e70ca808fd2824bb995 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-07-12 20:06:08 +0300 - - Translations: Add Vietnamese translation. - - Thanks to Trần Ngọc Quân. - - po/LINGUAS | 1 + - po/vi.po | 1007 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 1008 insertions(+) - -commit 17215f751c354852700e7f8592ccf319570a0721 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-06-29 20:54:14 +0300 - - xz: Update the help message of a few options. - - Updated: --threads, --block-size, and --block-list - Added: --flush-timeout - - src/xz/message.c | 18 +++++++++++------- - 1 file changed, 11 insertions(+), 7 deletions(-) - -commit 96864a6ddf91ad693d102ea165f3d7918744d582 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-06-18 22:07:06 +0300 - - xz: Use lzma_cputhreads() instead of own copy of tuklib_cpucores(). - - src/xz/Makefile.am | 1 - - src/xz/hardware.c | 12 +++++++++--- - 2 files changed, 9 insertions(+), 4 deletions(-) - -commit a115cc3748482e277f42a968baa3cd266f031dba -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-06-18 22:04:24 +0300 - - liblzma: Add lzma_cputhreads(). - - src/liblzma/Makefile.am | 8 +++++++- - src/liblzma/api/lzma/hardware.h | 14 ++++++++++++++ - src/liblzma/common/Makefile.inc | 1 + - src/liblzma/common/hardware_cputhreads.c | 22 ++++++++++++++++++++++ - src/liblzma/liblzma.map | 1 + - 5 files changed, 45 insertions(+), 1 deletion(-) - -commit 3ce3e7976904fbab4e6482bafa442856f77a51fa -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-06-18 19:11:52 +0300 - - xz: Check for filter chain compatibility for --flush-timeout. - - This avoids LZMA_PROG_ERROR from lzma_code() with filter chains - that don't support LZMA_SYNC_FLUSH. - - src/xz/coder.c | 30 +++++++++++++++++++++--------- - 1 file changed, 21 insertions(+), 9 deletions(-) - -commit 381ac14ed79e5d38809f251705be8b3193bba417 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-06-13 19:21:54 +0300 - - xzgrep: List xzgrep_expected_output in tests/Makefile.am. - - tests/Makefile.am | 3 ++- - 1 file changed, 2 insertions(+), 1 deletion(-) - -commit 4244b65b06d5ecaf6f9dd0387ac7e3166bd2364e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-06-13 18:58:22 +0300 - - xzgrep: Improve the test script. - - Now it should be close to the functionality of the original - version by Pavel Raiskup. - - tests/Makefile.am | 3 ++- - tests/test_scripts.sh | 24 ++++++++++++++---------- - tests/xzgrep_expected_output | 39 +++++++++++++++++++++++++++++++++++++++ - 3 files changed, 55 insertions(+), 11 deletions(-) - -commit 1e60f2c0a0ee6c18b02943ce56214799a70aac26 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-06-11 21:03:25 +0300 - - xzgrep: Add a test for the previous fix. - - This is a simplified version of Pavel Raiskup's - original patch. - - tests/test_scripts.sh | 26 ++++++++++++++++++++++---- - 1 file changed, 22 insertions(+), 4 deletions(-) - -commit ceca37901783988204caaf40dff4623d535cc789 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-06-11 20:43:28 +0300 - - xzgrep: exit 0 when at least one file matches. - - Mimic the original grep behavior and return exit_success when - at least one xz compressed file matches given pattern. - - Original bugreport: - https://bugzilla.redhat.com/show_bug.cgi?id=1108085 - - Thanks to Pavel Raiskup for the patch. - - src/scripts/xzgrep.in | 15 +++++++++++++-- - 1 file changed, 13 insertions(+), 2 deletions(-) - -commit 8c19216baccb92d011694590df8a1262da2e980c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-06-09 21:21:24 +0300 - - xz: Force single-threaded mode when --flush-timeout is used. - - src/xz/coder.c | 11 +++++++++++ - 1 file changed, 11 insertions(+) - -commit 87f1a24810805187d7bbc8ac5512e7eec307ddf5 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-05-25 22:05:39 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit da1718f266fcfc091e7bf08aae1bc986d0e6cc6b -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-05-25 21:45:56 +0300 - - liblzma: Use lzma_alloc_zero() in LZ encoder initialization. - - This avoids a memzero() call for a newly-allocated memory, - which can be expensive when encoding small streams with - an over-sized dictionary. - - To avoid using lzma_alloc_zero() for memory that doesn't - need to be zeroed, lzma_mf.son is now allocated separately, - which requires handling it separately in normalize() too. - - Thanks to Vincenzo Innocente for reporting the problem. - - src/liblzma/lz/lz_encoder.c | 84 ++++++++++++++++++++++-------------------- - src/liblzma/lz/lz_encoder.h | 2 +- - src/liblzma/lz/lz_encoder_mf.c | 31 +++++++++------- - 3 files changed, 62 insertions(+), 55 deletions(-) - -commit 28af24e9cf2eb259997c85dce13d4c97b3daa47a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-05-25 19:25:57 +0300 - - liblzma: Add the internal function lzma_alloc_zero(). - - src/liblzma/common/common.c | 21 +++++++++++++++++++++ - src/liblzma/common/common.h | 6 ++++++ - 2 files changed, 27 insertions(+) - -commit ed9ac85822c490e34b68c259afa0b385d21d1c40 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-05-08 18:03:09 +0300 - - xz: Fix uint64_t vs. size_t which broke 32-bit build. - - Thanks to Christian Hesse. - - src/xz/coder.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit d716acdae3fa7996f9e68a7bac012e6d8d13dd02 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-05-04 11:09:11 +0300 - - Docs: Update comments to refer to lzma/lzma12.h in example programs. - - doc/examples/03_compress_custom.c | 6 +++--- - 1 file changed, 3 insertions(+), 3 deletions(-) - -commit 4d5b7b3fda31241ca86ed35e08e73f776ee916e0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-05-04 11:07:17 +0300 - - liblzma: Rename the private API header lzma/lzma.h to lzma/lzma12.h. - - It can be confusing that two header files have the same name. - The public API file is still lzma.h. - - src/liblzma/api/Makefile.am | 2 +- - src/liblzma/api/lzma.h | 2 +- - src/liblzma/api/lzma/{lzma.h => lzma12.h} | 2 +- - 3 files changed, 3 insertions(+), 3 deletions(-) - -commit 1555a9c5664afc7893a2b75e9970105437f01ef1 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-04-25 17:53:42 +0300 - - Build: Fix the combination of --disable-xzdec --enable-lzmadec. - - In this case "make install" could fail if the man page directory - didn't already exist at the destination. If it did exist, a - dangling symlink was created there. Now the link is omitted - instead. This isn't the best fix but it's better than the old - behavior. - - src/xzdec/Makefile.am | 10 +++++++++- - 1 file changed, 9 insertions(+), 1 deletion(-) - -commit 56056571df3377eaa6ae6233b3ccc5d72e81d43d -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-04-25 17:44:26 +0300 - - Build: Add --disable-doc to configure. - - INSTALL | 6 ++++++ - Makefile.am | 2 ++ - configure.ac | 6 ++++++ - 3 files changed, 14 insertions(+) - -commit 6de61d8721097a6214810841aa85b08e303ac538 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-04-24 18:06:24 +0300 - - Update INSTALL. - - Add a note about failing "make check". The source of - the problem should be fixed in libtool (if it really is - a libtool bug and not mine) but I'm unable to spend time - on that for now. Thanks to Nelson H. F. Beebe for reporting - the issue. - - Add a note about a possible need to run "ldconfig" after - "make install". - - INSTALL | 25 +++++++++++++++++++++++++ - 1 file changed, 25 insertions(+) - -commit 54df428799a8d853639b753d0e6784694d73eb3e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-04-09 17:26:10 +0300 - - xz: Rename a variable to avoid a namespace collision on Solaris. - - I don't know the details but I have an impression that there's - no problem in practice if using GCC since people have built xz - with GCC (without patching xz), but renaming the variable cannot - hurt either. - - Thanks to Mark Ashley. - - src/xz/signals.c | 12 +++++++----- - 1 file changed, 7 insertions(+), 5 deletions(-) - -commit 5876ca27daa1429676b1160007d9688266907f00 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-01-29 20:19:41 +0200 - - Docs: Add example program for threaded encoding. - - I didn't add -DLZMA_UNSTABLE to Makefile so one has to - specify it manually as long as LZMA_UNSTABLE is needed. - - doc/examples/04_compress_easy_mt.c | 184 +++++++++++++++++++++++++++++++++++++ - doc/examples/Makefile | 3 +- - 2 files changed, 186 insertions(+), 1 deletion(-) - -commit 9494fb6d0ff41c585326f00aa8f7fe58f8106a5e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-01-29 20:13:51 +0200 - - liblzma: Fix lzma_mt.preset not working with lzma_stream_encoder_mt(). - - It read the filter chain from a wrong variable. - - src/liblzma/common/stream_encoder_mt.c | 4 ++-- - 1 file changed, 2 insertions(+), 2 deletions(-) - -commit 673a4cb53de3a715685cb1b836da57a3c7dcd43c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-01-20 11:20:40 +0200 - - liblzma: Fix typo in a comment. - - src/liblzma/api/lzma/block.h | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit ad96a871a1470eb76d6233d3890ce9338047b7a3 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-01-12 19:38:43 +0200 - - Windows: Add config.h for building liblzma with MSVC 2013. - - This is for building liblzma. Building xz tool too requires - a little more work. Maybe it will be supported, but for most - MSVC users it's enough to be able to build liblzma. - - C99 support in MSVC 2013 is almost usable which is a big - improvement over earlier versions. It's "almost" because - there's a dumb bug that breaks mixed declarations after - an "if" statements unless the "if" statement uses braces: - - https://connect.microsoft.com/VisualStudio/feedback/details/808650/visual-studio-2013-c99-compiler-bug - https://connect.microsoft.com/VisualStudio/feedback/details/808472/c99-support-of-mixed-declarations-and-statements-fails-with-certain-types-and-constructs - - Hopefully it will get fixed. Then liblzma should be - compilable with MSVC 2013 without patching. - - windows/config.h | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 1 file changed, 139 insertions(+) - -commit 3d5c090872fab4212b57c290e8ed4d02c78c1737 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-01-12 17:41:14 +0200 - - xz: Fix a comment. - - src/xz/coder.c | 4 ++-- - 1 file changed, 2 insertions(+), 2 deletions(-) - -commit 69fd4e1c932c7975476a0143c86e45d81b60d3f9 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-01-12 17:04:33 +0200 - - Windows: Add MSVC defines for inline and restrict keywords. - - src/common/sysdefs.h | 10 ++++++++++ - 1 file changed, 10 insertions(+) - -commit a19d9e8575ee6647cd9154cf1f20203f1330485f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-01-12 16:44:52 +0200 - - liblzma: Avoid C99 compound literal arrays. - - MSVC 2013 doesn't like them. Maybe they aren't so good - for readability either since many aren't used to them. - - src/liblzma/lzma/lzma_encoder_presets.c | 8 +++++--- - 1 file changed, 5 insertions(+), 3 deletions(-) - -commit e28528f1c867b2ed4ac91195ad08efb9bb8a6263 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-01-12 12:50:30 +0200 - - liblzma: Remove a useless C99ism from sha256.c. - - Unsurprisingly it makes no difference in compiled output. - - src/liblzma/check/sha256.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 5ad1effc45adfb7dabc9a98e79736077e6b7e2d5 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-01-12 12:17:08 +0200 - - xz: Fix use of wrong variable. - - Since the only call to suffix_set() uses optarg - as the argument, fixing this bug doesn't change - the behavior of the program. - - src/xz/suffix.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 3e62c68d75b5a3fdd46dbb34bb335d73289860d5 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2014-01-12 12:11:36 +0200 - - Fix typos in comments. - - src/common/mythread.h | 2 +- - src/liblzma/check/crc32_fast.c | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) - -commit e90ea601fb72867ec04adf456cbe4bf9520fd412 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-11-26 18:20:16 +0200 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit b22e94d8d15764416354e04729382a7371ae2c30 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-11-26 18:20:09 +0200 - - liblzma: Document the need for block->check for lzma_block_header_decode(). - - Thanks to Tomer Chachamu. - - src/liblzma/api/lzma/block.h | 3 +++ - 1 file changed, 3 insertions(+) - -commit d1cd8b1cb824b72421d1ee370e628024d2fcbec4 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-11-12 16:38:57 +0200 - - xz: Update the man page about --block-size and --block-list. - - src/xz/xz.1 | 24 +++++++++++++++--------- - 1 file changed, 15 insertions(+), 9 deletions(-) - -commit 76be7c612e6bcc38724488ccc3b8bcb1cfec9f0a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-11-12 16:30:53 +0200 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit dd750acbe2259d75444ef0f8da2d4bacc90d7afc -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-11-12 16:29:48 +0200 - - xz: Make --block-list and --block-size work together in single-threaded. - - Previously, --block-list and --block-size only worked together - in threaded mode. Boundaries are specified by --block-list, but - --block-size specifies the maximum size for a Block. Now this - works in single-threaded mode too. - - Thanks to James M Leddy for the original patch. - - src/xz/coder.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++---------- - 1 file changed, 75 insertions(+), 15 deletions(-) - -commit ae222fe9805d0161d022d75ba8485dab8bf6d7d5 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-10-26 13:26:14 +0300 - - Bump the version number to 5.1.3alpha. - - src/liblzma/api/lzma/version.h | 2 +- - src/liblzma/liblzma.map | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) - -commit 2193837a6a597cd3bf4e9ddf49421a5697d8e155 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-10-26 13:25:02 +0300 - - Update NEWS for 5.1.3alpha. - - NEWS | 35 +++++++++++++++++++++++++++++++++++ - 1 file changed, 35 insertions(+) - -commit ed48e75e2763876173aef8902da407a8eb28854b -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-10-26 12:47:04 +0300 - - Update TODO. - - TODO | 4 ---- - 1 file changed, 4 deletions(-) - -commit 841da0352d79a56a44796a4c39163429c9f039a3 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-10-25 22:41:28 +0300 - - xz: Document behavior of --block-list with threads. - - This needs to be updated before 5.2.0. - - src/xz/xz.1 | 24 +++++++++++++++++++++--- - 1 file changed, 21 insertions(+), 3 deletions(-) - -commit 56feb8665b78c1032aabd53c619c62af51defe64 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-10-22 20:03:12 +0300 - - xz: Document --flush-timeout=TIMEOUT on the man page. - - src/xz/xz.1 | 37 ++++++++++++++++++++++++++++++++++++- - 1 file changed, 36 insertions(+), 1 deletion(-) - -commit ba413da1d5bb3324287cf3174922acd921165971 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-10-22 19:51:55 +0300 - - xz: Take advantage of LZMA_FULL_BARRIER with --block-list. - - Now if --block-list is used in threaded mode, the encoder - won't need to flush at each Block boundary specified via - --block-list. This improves performance a lot, making - threading helpful with --block-list. - - The flush timer was reset after LZMA_FULL_FLUSH but since - LZMA_FULL_BARRIER doesn't flush, resetting the timer is - no longer done. - - src/xz/coder.c | 32 +++++++++++++++----------------- - 1 file changed, 15 insertions(+), 17 deletions(-) - -commit 0cd45fc2bc5537de287a0bc005e2d67467a92148 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-10-02 20:05:23 +0300 - - liblzma: Support LZMA_FULL_FLUSH and _BARRIER in threaded encoder. - - Now --block-list=SIZES works with in the threaded mode too, - although the performance is still bad due to the use of - LZMA_FULL_FLUSH instead of the new LZMA_FULL_BARRIER. - - src/liblzma/common/stream_encoder_mt.c | 55 ++++++++++++++++++++++++---------- - 1 file changed, 39 insertions(+), 16 deletions(-) - -commit 97bb38712f414fabecca908af2e38a12570293fd -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-10-02 12:55:11 +0300 - - liblzma: Add LZMA_FULL_BARRIER support to single-threaded encoder. - - In the single-threaded encoder LZMA_FULL_BARRIER is simply - an alias for LZMA_FULL_FLUSH. - - src/liblzma/api/lzma/base.h | 37 ++++++++++++++++++++++++++++++------- - src/liblzma/common/common.c | 17 +++++++++++++++-- - src/liblzma/common/common.h | 7 ++++++- - src/liblzma/common/stream_encoder.c | 4 +++- - 4 files changed, 54 insertions(+), 11 deletions(-) - -commit fef0c6b410c08e581c9178700a4e7599f0895ff9 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-09-17 11:57:51 +0300 - - liblzma: Add block_buffer_encoder.h into Makefile.inc. - - This should have been in b465da5988dd59ad98fda10c2e4ea13d0b9c73bc. - - src/liblzma/common/Makefile.inc | 1 + - 1 file changed, 1 insertion(+) - -commit 8083e03291b6d21c0f538163e187b4e8cd5594e4 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-09-17 11:55:38 +0300 - - xz: Add a missing test for TUKLIB_DOSLIKE. - - src/xz/file_io.c | 2 ++ - 1 file changed, 2 insertions(+) - -commit 6b44b4a775fe29ecc7bcb7996e086e3bc09e5fd0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-09-17 11:52:28 +0300 - - Add native threading support on Windows. - - Now liblzma only uses "mythread" functions and types - which are defined in mythread.h matching the desired - threading method. - - Before Windows Vista, there is no direct equivalent to - pthread condition variables. Since this package doesn't - use pthread_cond_broadcast(), pre-Vista threading can - still be kept quite simple. The pre-Vista code doesn't - use anything that wasn't already available in Windows 95, - so the binaries should run even on Windows 95 if someone - happens to care. - - INSTALL | 41 ++- - configure.ac | 118 ++++++-- - src/common/mythread.h | 513 ++++++++++++++++++++++++++------- - src/liblzma/common/stream_encoder_mt.c | 83 +++--- - src/xz/coder.c | 8 +- - windows/README-Windows.txt | 2 +- - windows/build.bash | 23 +- - 7 files changed, 573 insertions(+), 215 deletions(-) - -commit ae0ab74a88d5b9b15845f1d9a24ade4349a54f9f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-09-11 14:40:35 +0300 - - Build: Remove a comment about Automake 1.10 from configure.ac. - - The previous commit supports silent rules and that requires - Automake 1.11. - - configure.ac | 2 -- - 1 file changed, 2 deletions(-) - -commit 72975df6c8c59aaf849138ab3606e8fb6970596a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-09-09 20:37:03 +0300 - - Build: Create liblzma.pc in a src/liblzma/Makefile.am. - - Previously it was done in configure, but doing that goes - against the Autoconf manual. Autoconf requires that it is - possible to override e.g. prefix after running configure - and that doesn't work correctly if liblzma.pc is created - by configure. - - A potential downside of this change is that now e.g. - libdir in liblzma.pc is a standalone string instead of - being defined via ${prefix}, so if one overrides prefix - when running pkg-config the libdir won't get the new value. - I don't know if this matters in practice. - - Thanks to Vincent Torri. - - configure.ac | 1 - - src/liblzma/Makefile.am | 20 ++++++++++++++++++++ - 2 files changed, 20 insertions(+), 1 deletion(-) - -commit 1c2b6e7e8382ed390f53e140f160488bb2205ecc -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-08-04 15:24:09 +0300 - - Fix the previous commit which broke the build. - - Apparently I didn't even compile-test the previous commit. - - Thanks to Christian Hesse. - - src/common/tuklib_cpucores.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 124eb69c7857f618b4807588c51bc9ba21bf8691 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-08-03 13:52:58 +0300 - - Windows: Add Windows support to tuklib_cpucores(). - - It is used for Cygwin too. I'm not sure if that is - a good or bad idea. - - Thanks to Vincent Torri. - - m4/tuklib_cpucores.m4 | 19 +++++++++++++++++-- - src/common/tuklib_cpucores.c | 13 ++++++++++++- - 2 files changed, 29 insertions(+), 3 deletions(-) - -commit eada8a875ce3fd521cb42e4ace2624d3d49c5f35 -Author: Anders F Bjorklund <afb@users.sourceforge.net> -Date: 2013-08-02 15:59:46 +0200 - - macosx: separate liblzma package - - macosx/build.sh | 23 +++++++++++++++-------- - 1 file changed, 15 insertions(+), 8 deletions(-) - -commit be0100d01ca6a75899d051bee00acf17e6dc0c15 -Author: Anders F Bjorklund <afb@users.sourceforge.net> -Date: 2013-08-02 15:58:44 +0200 - - macosx: set minimum to leopard - - macosx/build.sh | 13 ++++++++----- - 1 file changed, 8 insertions(+), 5 deletions(-) - -commit 416729e2d743f4b2fe9fd438eedeb98adce033c3 -Author: Anders F Bjorklund <afb@users.sourceforge.net> -Date: 2011-08-07 13:13:30 +0200 - - move configurables into variables - - macosx/build.sh | 25 ++++++++++++++++++------- - 1 file changed, 18 insertions(+), 7 deletions(-) - -commit 16581080e5f29f9a4e49efece21c5bf572323acc -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-07-15 14:08:41 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 3e2b198ba37b624efd9c7caee2a435dc986b46c6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-07-15 14:08:02 +0300 - - Build: Fix the detection of missing CRC32. - - Thanks to Vincent Torri. - - configure.ac | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit dee6ad3d5915422bc30a6821efeacaeb8ca8ef00 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-07-04 14:18:46 +0300 - - xz: Add preliminary support for --flush-timeout=TIMEOUT. - - When --flush-timeout=TIMEOUT is used, xz will use - LZMA_SYNC_FLUSH if read() would block and at least - TIMEOUT milliseconds has elapsed since the previous flush. - - This can be useful in realtime-like use cases where the - data is simultanously decompressed by another process - (possibly on a different computer). If new uncompressed - input data is produced slowly, without this option xz could - buffer the data for a long time until it would become - decompressible from the output. - - If TIMEOUT is 0, the feature is disabled. This is the default. - - This commit affects the compression side. Using xz for - the decompression side for the above purpose doesn't work - yet so well because there is quite a bit of input and - output buffering when decompressing. - - The --long-help or man page were not updated yet. - The details of this feature may change. - - src/xz/args.c | 7 +++++++ - src/xz/coder.c | 46 +++++++++++++++++++++++++++++++++++----------- - src/xz/file_io.c | 46 ++++++++++++++++++++++++++++++++++++---------- - 3 files changed, 78 insertions(+), 21 deletions(-) - -commit fa381acaf9a29a8114e1c0a97de99bab9adb014e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-07-04 13:41:03 +0300 - - xz: Don't set src_eof=true after an I/O error because it's useless. - - src/xz/file_io.c | 3 --- - 1 file changed, 3 deletions(-) - -commit ea00545beace5b950f709ec21e46878e0f448678 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-07-04 13:25:11 +0300 - - xz: Fix the test when to read more input. - - Testing for end of file was no longer correct after full flushing - became possible with --block-size=SIZE and --block-list=SIZES. - There was no bug in practice though because xz just made a few - unneeded zero-byte reads. - - src/xz/coder.c | 6 +++--- - 1 file changed, 3 insertions(+), 3 deletions(-) - -commit 736903c64bef394c06685d79908e397bcb08b88f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-07-04 12:51:57 +0300 - - xz: Move some of the timing code into mytime.[hc]. - - This switches units from microseconds to milliseconds. - - New clock_gettime(CLOCK_MONOTONIC) will be used if available. - There is still a fallback to gettimeofday(). - - src/xz/Makefile.am | 2 ++ - src/xz/coder.c | 5 +++ - src/xz/message.c | 54 +++++++++------------------------ - src/xz/mytime.c | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ - src/xz/mytime.h | 47 ++++++++++++++++++++++++++++ - src/xz/private.h | 1 + - 6 files changed, 158 insertions(+), 40 deletions(-) - -commit 24edf8d807e24ffaa1e793114d94cca3b970027d -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-07-01 14:35:03 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit c0627b3fceacfa1ed162f5f55235360ea26f569a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-07-01 14:34:11 +0300 - - xz: Silence a warning seen with _FORTIFY_SOURCE=2. - - Thanks to Christian Hesse. - - src/xz/file_io.c | 8 +++++++- - 1 file changed, 7 insertions(+), 1 deletion(-) - -commit 1936718bb38ee394bd89836fdd4eabc0beb02443 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-30 19:40:11 +0300 - - Update NEWS for 5.0.5. - - NEWS | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++ - 1 file changed, 52 insertions(+) - -commit a37ae8b5eb6093a530198f109c6f7a538c80ecf0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-30 18:02:27 +0300 - - Man pages: Use similar syntax for synopsis as in xz. - - The man pages of lzmainfo, xzmore, and xzdec had similar - constructs as the man page of xz had before the commit - eb6ca9854b8eb9fbf72497c1cf608d6b19d2d494. Eric S. Raymond - didn't mention these man pages in his bug report, but - it's nice to be consistent. - - src/lzmainfo/lzmainfo.1 | 4 ++-- - src/scripts/xzmore.1 | 6 +++--- - src/xzdec/xzdec.1 | 10 +++++----- - 3 files changed, 10 insertions(+), 10 deletions(-) - -commit cdba9ddd870ae72fd6219a125662c20ec997f86c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-29 15:59:13 +0300 - - xz: Use non-blocking I/O for the output file. - - Now both reading and writing should be without - race conditions with signals. - - They might still be signal handling issues left. - Signals are blocked during many operations to avoid - EINTR but it may cause problems e.g. if writing to - stderr blocks when trying to display an error message. - - src/xz/file_io.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++-------- - 1 file changed, 49 insertions(+), 8 deletions(-) - -commit e61a5c95da3fe31281d959e5e842885a8ba2b5bd -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-28 23:56:17 +0300 - - xz: Fix return value type in io_write_buf(). - - It didn't affect the behavior of the code since -1 - becomes true anyway. - - src/xz/file_io.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 9dc319eabb34a826f4945f91c71620f14a60e9e2 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-28 23:48:05 +0300 - - xz: Use the self-pipe trick to avoid a race condition with signals. - - It is possible that a signal to set user_abort arrives right - before a blocking system call is made. In this case the call - may block until another signal arrives, while the wanted - behavior is to make xz clean up and exit as soon as possible. - - After this commit, the race condition is avoided with the - input side which already uses non-blocking I/O. The output - side still uses blocking I/O and thus has the race condition. - - src/xz/file_io.c | 56 ++++++++++++++++++++++++++++++++++++++++++++------------ - src/xz/file_io.h | 8 ++++++++ - src/xz/signals.c | 5 +++++ - 3 files changed, 57 insertions(+), 12 deletions(-) - -commit 3541bc79d0cfabc0ad155c99bfdad1289f17fec3 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-28 22:51:02 +0300 - - xz: Use non-blocking I/O for the input file. - - src/xz/file_io.c | 156 +++++++++++++++++++++++++++++++++++++++---------------- - 1 file changed, 111 insertions(+), 45 deletions(-) - -commit 78673a08bed5066c81e8a8e90d20e670c28ecfd5 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-28 18:46:13 +0300 - - xz: Remove an outdated NetBSD-specific comment. - - Nowadays errno == EFTYPE is documented in open(2). - - src/xz/file_io.c | 4 ---- - 1 file changed, 4 deletions(-) - -commit a616fdad34b48b2932ef03fb87309dcc8b829527 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-28 18:09:47 +0300 - - xz: Fix error detection of fcntl(fd, F_SETFL, flags) calls. - - POSIX says that fcntl(fd, F_SETFL, flags) returns -1 on - error and "other than -1" on success. This is how it is - documented e.g. on OpenBSD too. On Linux, success with - F_SETFL is always 0 (at least accorinding to fcntl(2) - from man-pages 3.51). - - src/xz/file_io.c | 8 ++++---- - 1 file changed, 4 insertions(+), 4 deletions(-) - -commit 4a08a6e4c61c65ab763ab314100a6d7a3bb89298 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-28 17:36:47 +0300 - - xz: Fix use of wrong variable in a fcntl() call. - - Due to a wrong variable name, when writing a sparse file - to standard output, *all* file status flags were cleared - (to the extent the operating system allowed it) instead of - only clearing the O_APPEND flag. In practice this worked - fine in the common situations on GNU/Linux, but I didn't - check how it behaved elsewhere. - - The original flags were still restored correctly. I still - changed the code to use a separate boolean variable to - indicate when the flags should be restored instead of - relying on a special value in stdout_flags. - - src/xz/file_io.c | 24 +++++++++++++----------- - 1 file changed, 13 insertions(+), 11 deletions(-) - -commit b790b435daa3351067f80a5973b647f8d55367a2 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-28 14:55:37 +0300 - - xz: Fix assertion related to posix_fadvise(). - - Input file can be a FIFO or something else that doesn't - support posix_fadvise() so don't check the return value - even with an assertion. Nothing bad happens if the call - to posix_fadvise() fails. - - src/xz/file_io.c | 10 ++-------- - 1 file changed, 2 insertions(+), 8 deletions(-) - -commit 84d2da6c9dc252f441deb7626c2522202b005d4d -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-26 13:30:57 +0300 - - xz: Check the value of lzma_stream_flags.version in --list. - - It is a no-op for now, but if an old xz version is used - together with a newer liblzma that supports something new, - then this check becomes important and will stop the old xz - from trying to parse files that it won't understand. - - src/xz/list.c | 14 ++++++++++++++ - 1 file changed, 14 insertions(+) - -commit 9376f5f8f762296f2173d61af9101112c36f38c0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-26 12:17:00 +0300 - - Build: Require Automake 1.12 and use serial-tests option. - - It should actually still work with Automake 1.10 if - the serial-tests option is removed. Automake 1.13 started - using parallel tests by default and the option to get - the old behavior isn't supported before 1.12. - - At least for now, parallel tests don't improve anything - in XZ Utils but they hide the progress output from - test_compress.sh. - - configure.ac | 4 +++- - 1 file changed, 3 insertions(+), 1 deletion(-) - -commit b7e200d7bd0a3c7c171c13ad37d68296d6f73374 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-23 18:59:13 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 46540e4c10923e363741ff5aab99e79fc0ce6ee8 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-23 18:57:23 +0300 - - liblzma: Avoid a warning about a shadowed variable. - - On Mac OS X wait() is declared in <sys/wait.h> that - we include one way or other so don't use "wait" as - a variable name. - - Thanks to Christian Kujau. - - src/liblzma/common/stream_encoder_mt.c | 4 ++-- - 1 file changed, 2 insertions(+), 2 deletions(-) - -commit ebb501ec73cecc546c67117dd01b5e33c90bfb4a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-23 17:36:47 +0300 - - xz: Validate Uncompressed Size from Block Header in list.c. - - This affects only "xz -lvv". Normal decompression with xz - already detected if Block Header and Index had mismatched - Uncompressed Size fields. So this just makes "xz -lvv" - show such files as corrupt instead of showing the - Uncompressed Size from Index. - - src/xz/list.c | 14 +++++++++++++- - 1 file changed, 13 insertions(+), 1 deletion(-) - -commit c09e91dd236d3cabee0fc48312b3dc8cceae41ab -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-21 22:08:11 +0300 - - Update THANKS. - - THANKS | 2 ++ - 1 file changed, 2 insertions(+) - -commit eb6ca9854b8eb9fbf72497c1cf608d6b19d2d494 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-21 22:04:45 +0300 - - xz: Make the man page more friendly to doclifter. - - Thanks to Eric S. Raymond. - - src/xz/xz.1 | 7 ++++--- - 1 file changed, 4 insertions(+), 3 deletions(-) - -commit 0c0a1947e6ad90a0a10b7a5c39f6ab99a0aa5c93 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-21 21:54:59 +0300 - - xz: A couple of man page fixes. - - Now the interaction of presets and custom filter chains - is described correctly. Earlier it contradicted itself. - - Thanks to DevHC who reported these issues on IRC to me - on 2012-12-14. - - src/xz/xz.1 | 35 +++++++++++++++++++++++------------ - 1 file changed, 23 insertions(+), 12 deletions(-) - -commit 2fcda89939c903106c429e109083d43d894049e0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-06-21 21:50:26 +0300 - - xz: Fix interaction between preset and custom filter chains. - - There was somewhat illogical behavior when --extreme was - specified and mixed with custom filter chains. - - Before this commit, "xz -9 --lzma2 -e" was equivalent - to "xz --lzma2". After it is equivalent to "xz -6e" - (all earlier preset options get forgotten when a custom - filter chain is specified and the default preset is 6 - to which -e is applied). I find this less illogical. - - This also affects the meaning of "xz -9e --lzma2 -7". - Earlier it was equivalent to "xz -7e" (the -e specified - before a custom filter chain wasn't forgotten). Now it - is "xz -7". Note that "xz -7e" still is the same as "xz -e7". - - Hopefully very few cared about this in the first place, - so pretty much no one should even notice this change. - - Thanks to Conley Moorhous. - - src/xz/coder.c | 35 +++++++++++++++++++++-------------- - 1 file changed, 21 insertions(+), 14 deletions(-) - -commit 97379c5ea758da3f8b0bc444d5f7fa43753ce610 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-04-27 22:07:46 +0300 - - Build: Use -Wvla with GCC if supported. - - Variable-length arrays are mandatory in C99 but optional in C11. - The code doesn't currently use any VLAs and it shouldn't in the - future either to stay compatible with C11 without requiring any - optional C11 features. - - configure.ac | 1 + - 1 file changed, 1 insertion(+) - -commit 8957c58609d3987c58aa72b96c436cf565cc4917 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-04-15 19:29:09 +0300 - - xzdec: Improve the --help message. - - The options are now ordered in the same order as in xz's help - message. - - Descriptions were added to the options that are ignored. - I left them in parenthesis even if it looks a bit weird - because I find it easier to spot the ignored vs. non-ignored - options from the list that way. - - src/xzdec/xzdec.c | 10 +++++----- - 1 file changed, 5 insertions(+), 5 deletions(-) - -commit ed886e1a92534a24401d0e99c11f1dcff3b5220a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-04-05 19:25:40 +0300 - - Update THANKS. - - THANKS | 2 ++ - 1 file changed, 2 insertions(+) - -commit 5019413a055ce29e660dbbf15e02443cb5a26c59 -Author: Jeff Bastian <jbastian@redhat.com> -Date: 2013-04-03 13:59:17 +0200 - - xzgrep: make the '-h' option to be --no-filename equivalent - - * src/scripts/xzgrep.in: Accept the '-h' option in argument parsing. - - src/scripts/xzgrep.in | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 5ea900cb5ad862bca81316729f92357c1fc040ce -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-03-23 22:25:15 +0200 - - liblzma: Be less picky in lzma_alone_decoder(). - - To avoid false positives when detecting .lzma files, - rare values in dictionary size and uncompressed size fields - were rejected. They will still be rejected if .lzma files - are decoded with lzma_auto_decoder(), but when using - lzma_alone_decoder() directly, such files will now be accepted. - Hopefully this is an OK compromise. - - This doesn't affect xz because xz still has its own file - format detection code. This does affect lzmadec though. - So after this commit lzmadec will accept files that xz or - xz-emulating-lzma doesn't. - - NOTE: lzma_alone_decoder() still won't decode all .lzma files - because liblzma's LZMA decoder doesn't support lc + lp > 4. - - Reported here: - http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/7068827 - - src/liblzma/common/alone_decoder.c | 22 ++++++++++++++-------- - src/liblzma/common/alone_decoder.h | 5 +++-- - src/liblzma/common/auto_decoder.c | 2 +- - 3 files changed, 18 insertions(+), 11 deletions(-) - -commit bb117fffa84604b6e3811b068c80db82bf7f7b05 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-03-23 21:55:13 +0200 - - liblzma: Use lzma_block_buffer_bound64() in threaded encoder. - - Now it uses lzma_block_uncomp_encode() if the data doesn't - fit into the space calculated by lzma_block_buffer_bound64(). - - src/liblzma/common/stream_encoder_mt.c | 66 +++++++++++++++++++++++++--------- - 1 file changed, 50 insertions(+), 16 deletions(-) - -commit e572e123b55b29527e54ce5f0807f115481d78b9 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-03-23 21:51:38 +0200 - - liblzma: Fix another deadlock in the threaded encoder. - - This race condition could cause a deadlock if lzma_end() was - called before finishing the encoding. This can happen with - xz with debugging enabled (non-debugging version doesn't - call lzma_end() before exiting). - - src/liblzma/common/stream_encoder_mt.c | 9 ++++++--- - 1 file changed, 6 insertions(+), 3 deletions(-) - -commit b465da5988dd59ad98fda10c2e4ea13d0b9c73bc -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-03-23 19:17:33 +0200 - - liblzma: Add lzma_block_uncomp_encode(). - - This also adds a new internal function - lzma_block_buffer_bound64() which is similar to - lzma_block_buffer_bound() but uses uint64_t instead - of size_t. - - src/liblzma/api/lzma/block.h | 18 ++++++ - src/liblzma/common/block_buffer_encoder.c | 94 +++++++++++++++++++++---------- - src/liblzma/common/block_buffer_encoder.h | 24 ++++++++ - src/liblzma/liblzma.map | 1 + - 4 files changed, 106 insertions(+), 31 deletions(-) - -commit 9e6dabcf22ef4679f4faaae15ebd5b137ae2fad1 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2013-03-05 19:14:50 +0200 - - Avoid unneeded use of awk in xzless. - - Use "read" instead of "awk" in xzless to get the version - number of "less". The need for awk was introduced in - the commit db5c1817fabf7cbb9e4087b1576eb26f0747338e. - - Thanks to Ariel P for the patch. - - src/scripts/xzless.in | 3 +-- - 1 file changed, 1 insertion(+), 2 deletions(-) - -commit e7b424d267a34803db8d92a3515528be2ed45abd -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-12-14 20:13:32 +0200 - - Make the progress indicator smooth in threaded mode. - - This adds lzma_get_progress() to liblzma and takes advantage - of it in xz. - - lzma_get_progress() collects progress information from - the thread-specific structures so that fairly accurate - progress information is available to applications. Adding - a new function seemed to be a better way than making the - information directly available in lzma_stream (like total_in - and total_out are) because collecting the information requires - locking mutexes. It's waste of time to do it more often than - the up to date information is actually needed by an application. - - src/liblzma/api/lzma/base.h | 22 +++++++++- - src/liblzma/common/common.c | 16 +++++++ - src/liblzma/common/common.h | 6 +++ - src/liblzma/common/stream_encoder_mt.c | 77 +++++++++++++++++++++++++++++++--- - src/liblzma/liblzma.map | 1 + - src/xz/message.c | 20 +++++---- - 6 files changed, 129 insertions(+), 13 deletions(-) - -commit 2ebbb994e367f55f2561aa7c9e7451703c171f2f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-12-14 11:01:41 +0200 - - liblzma: Fix mythread_sync for nested locking. - - src/common/mythread.h | 5 +++-- - 1 file changed, 3 insertions(+), 2 deletions(-) - -commit 4c7e28705f6de418d19cc77324ef301f996e01ff -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-12-13 21:05:36 +0200 - - xz: Mention --threads in --help. - - Thanks to Olivier Delhomme for pointing out that this - was still missing. - - src/xz/message.c | 4 ++++ - 1 file changed, 4 insertions(+) - -commit db5c1817fabf7cbb9e4087b1576eb26f0747338e -Author: Jonathan Nieder <jrnieder@gmail.com> -Date: 2012-11-19 00:10:10 -0800 - - xzless: Make "less -V" parsing more robust - - In v4.999.9beta~30 (xzless: Support compressed standard input, - 2009-08-09), xzless learned to parse ‘less -V’ output to figure out - whether less is new enough to handle $LESSOPEN settings starting - with “|-”. That worked well for a while, but the version string from - ‘less’ versions 448 (June, 2012) is misparsed, producing a warning: - - $ xzless /tmp/test.xz; echo $? - /usr/bin/xzless: line 49: test: 456 (GNU regular expressions): \ - integer expression expected - 0 - - More precisely, modern ‘less’ lists the regexp implementation along - with its version number, and xzless passes the entire version number - with attached parenthetical phrase as a number to "test $a -gt $b", - producing the above confusing message. - - $ less-444 -V | head -1 - less 444 - $ less -V | head -1 - less 456 (no regular expressions) - - So relax the pattern matched --- instead of expecting "less <number>", - look for a line of the form "less <number>[ (extra parenthetical)]". - While at it, improve the behavior when no matching line is found --- - instead of producing a cryptic message, we can fall back on a LESSPIPE - setting that is supported by all versions of ‘less’. - - The implementation uses "awk" for simplicity. Hopefully that’s - portable enough. - - Reported-by: Jörg-Volker Peetz <jvpeetz@web.de> - Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> - - src/scripts/xzless.in | 3 ++- - 1 file changed, 2 insertions(+), 1 deletion(-) - -commit 65536214a31ecd33b6b03b68a351fb597d3703d6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-10-03 15:54:24 +0300 - - xz: Fix the note about --rsyncable on the man page. - - src/xz/xz.1 | 17 +++++++++-------- - 1 file changed, 9 insertions(+), 8 deletions(-) - -commit 3d93b6354927247a1569caf22ad27b07e97ee904 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-09-28 20:11:09 +0300 - - xz: Improve handling of failed realloc in xrealloc. - - Thanks to Jim Meyering. - - src/xz/util.c | 14 ++++++++++++-- - 1 file changed, 12 insertions(+), 2 deletions(-) - -commit ab225620664e235637833be2329935f9d290ba80 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-08-24 16:27:31 +0300 - - A few typo fixes to comments and the xz man page. - - Thanks to Jim Meyering. - - configure.ac | 2 +- - src/liblzma/check/sha256.c | 1 - - src/xz/xz.1 | 4 ++-- - 3 files changed, 3 insertions(+), 4 deletions(-) - -commit f3c1ec69d910175ffd431fd82968dd35cec806ed -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-08-13 21:40:09 +0300 - - xz: Add a warning to --help about alpha and beta versions. - - src/xz/message.c | 5 +++++ - 1 file changed, 5 insertions(+) - -commit d8eaf9d8278c23c2cf2b7ca5562d4de570d3b5db -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-08-02 17:13:30 +0300 - - Build: Bump gettext version requirement to 0.18. - - Otherwise too old version of m4/lib-link.m4 gets included - when autoreconf -fi is run. - - configure.ac | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 96e08902b09f0f304d4ff80c6e83ef7fff883f34 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-17 18:29:08 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 3778db1be53e61ff285c573af5ee468803008456 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-17 18:19:59 +0300 - - liblzma: Make the use of lzma_allocator const-correct. - - There is a tiny risk of causing breakage: If an application - assigns lzma_stream.allocator to a non-const pointer, such - code won't compile anymore. I don't know why anyone would do - such a thing though, so in practice this shouldn't cause trouble. - - Thanks to Jan Kratochvil for the patch. - - src/liblzma/api/lzma/base.h | 4 +++- - src/liblzma/api/lzma/block.h | 6 ++--- - src/liblzma/api/lzma/container.h | 9 +++++--- - src/liblzma/api/lzma/filter.h | 13 ++++++----- - src/liblzma/api/lzma/index.h | 16 ++++++------- - src/liblzma/api/lzma/index_hash.h | 4 ++-- - src/liblzma/common/alone_decoder.c | 6 ++--- - src/liblzma/common/alone_decoder.h | 2 +- - src/liblzma/common/alone_encoder.c | 8 +++---- - src/liblzma/common/auto_decoder.c | 6 ++--- - src/liblzma/common/block_buffer_decoder.c | 2 +- - src/liblzma/common/block_buffer_encoder.c | 4 ++-- - src/liblzma/common/block_decoder.c | 6 ++--- - src/liblzma/common/block_decoder.h | 2 +- - src/liblzma/common/block_encoder.c | 8 +++---- - src/liblzma/common/block_encoder.h | 2 +- - src/liblzma/common/block_header_decoder.c | 4 ++-- - src/liblzma/common/common.c | 10 ++++----- - src/liblzma/common/common.h | 20 +++++++++-------- - src/liblzma/common/easy_buffer_encoder.c | 4 ++-- - src/liblzma/common/filter_buffer_decoder.c | 3 ++- - src/liblzma/common/filter_buffer_encoder.c | 7 +++--- - src/liblzma/common/filter_common.c | 4 ++-- - src/liblzma/common/filter_common.h | 2 +- - src/liblzma/common/filter_decoder.c | 7 +++--- - src/liblzma/common/filter_decoder.h | 2 +- - src/liblzma/common/filter_encoder.c | 2 +- - src/liblzma/common/filter_encoder.h | 2 +- - src/liblzma/common/filter_flags_decoder.c | 2 +- - src/liblzma/common/index.c | 26 ++++++++++----------- - src/liblzma/common/index_decoder.c | 12 +++++----- - src/liblzma/common/index_encoder.c | 6 ++--- - src/liblzma/common/index_encoder.h | 2 +- - src/liblzma/common/index_hash.c | 6 +++-- - src/liblzma/common/outqueue.c | 4 ++-- - src/liblzma/common/outqueue.h | 5 +++-- - src/liblzma/common/stream_buffer_decoder.c | 2 +- - src/liblzma/common/stream_buffer_encoder.c | 3 ++- - src/liblzma/common/stream_decoder.c | 9 ++++---- - src/liblzma/common/stream_decoder.h | 5 +++-- - src/liblzma/common/stream_encoder.c | 10 ++++----- - src/liblzma/common/stream_encoder_mt.c | 16 ++++++------- - src/liblzma/delta/delta_common.c | 4 ++-- - src/liblzma/delta/delta_decoder.c | 6 ++--- - src/liblzma/delta/delta_decoder.h | 5 +++-- - src/liblzma/delta/delta_encoder.c | 6 ++--- - src/liblzma/delta/delta_encoder.h | 3 ++- - src/liblzma/delta/delta_private.h | 2 +- - src/liblzma/lz/lz_decoder.c | 8 +++---- - src/liblzma/lz/lz_decoder.h | 7 +++--- - src/liblzma/lz/lz_encoder.c | 19 ++++++++-------- - src/liblzma/lz/lz_encoder.h | 6 ++--- - src/liblzma/lzma/lzma2_decoder.c | 8 +++---- - src/liblzma/lzma/lzma2_decoder.h | 5 +++-- - src/liblzma/lzma/lzma2_encoder.c | 6 ++--- - src/liblzma/lzma/lzma2_encoder.h | 2 +- - src/liblzma/lzma/lzma_decoder.c | 8 +++---- - src/liblzma/lzma/lzma_decoder.h | 7 +++--- - src/liblzma/lzma/lzma_encoder.c | 7 +++--- - src/liblzma/lzma/lzma_encoder.h | 5 +++-- - src/liblzma/simple/arm.c | 8 ++++--- - src/liblzma/simple/armthumb.c | 8 ++++--- - src/liblzma/simple/ia64.c | 8 ++++--- - src/liblzma/simple/powerpc.c | 8 ++++--- - src/liblzma/simple/simple_coder.c | 10 ++++----- - src/liblzma/simple/simple_coder.h | 36 ++++++++++++++++++++---------- - src/liblzma/simple/simple_decoder.c | 2 +- - src/liblzma/simple/simple_decoder.h | 2 +- - src/liblzma/simple/simple_private.h | 3 ++- - src/liblzma/simple/sparc.c | 8 ++++--- - src/liblzma/simple/x86.c | 8 ++++--- - 71 files changed, 269 insertions(+), 219 deletions(-) - -commit d625c7cf824fd3b61c6da84f56179e94917ff603 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-05 07:36:28 +0300 - - Tests: Remove tests/test_block.c that had gotten committed accidentally. - - tests/test_block.c | 52 ---------------------------------------------------- - 1 file changed, 52 deletions(-) - -commit 0b09d266cce72bc4841933b171e79551e488927c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-05 07:33:35 +0300 - - Build: Include macosx/build.sh in the distribution. - - It has been in the Git repository since 2010 but probably - few people have seen it since it hasn't been included in - the release tarballs. :-( - - Makefile.am | 1 + - 1 file changed, 1 insertion(+) - -commit d6e0b23d4613b9f417893dd96cc168c8005ece3d -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-05 07:28:53 +0300 - - Build: Include validate_map.sh in the distribution. - - It's required by "make mydist". - - Fix also the location of EXTRA_DIST+= so that those files - get distributed also if symbol versioning isn't enabled. - - src/liblzma/Makefile.am | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 19de545d86097c3954d69ab5d12820387f6a09bc -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-05 07:24:45 +0300 - - Docs: Fix the name LZMA Utils -> XZ Utils in debug/README. - - debug/README | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 672eccf57c31a40dfb956b7662db06d43e18618e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-05 07:23:17 +0300 - - Include debug/translation.bash in the distribution. - - Also fix the script name mentioned in README. - - README | 4 ++-- - debug/Makefile.am | 3 +++ - 2 files changed, 5 insertions(+), 2 deletions(-) - -commit cafb523adac1caf305e70a04bc37f25602bf990c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-04 22:31:58 +0300 - - xz: Document --block-list better. - - Thanks to Jonathan Nieder. - - src/xz/xz.1 | 8 +++++++- - 1 file changed, 7 insertions(+), 1 deletion(-) - -commit c7ff218528bc8f7c65e7ef73c6515777346c6794 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-04 20:01:49 +0300 - - Bump the version number to 5.1.2alpha. - - src/liblzma/api/lzma/version.h | 2 +- - src/liblzma/liblzma.map | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) - -commit 8f3c1d886f93e6478ad509ff52102b2ce7faa999 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-04 20:01:19 +0300 - - Update NEWS for 5.1.2alpha. - - NEWS | 41 +++++++++++++++++++++++++++++++++++++++++ - 1 file changed, 41 insertions(+) - -commit 0d5fa05466e580fbc458820f87013ae7644e20e5 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-04 19:58:23 +0300 - - xz: Fix the version number printed by xz -lvv. - - The decoder bug was fixed in 5.0.2 instead of 5.0.3. - - src/xz/list.c | 6 +++--- - 1 file changed, 3 insertions(+), 3 deletions(-) - -commit df11317985a4165731dde12bb0f0028da0e7b77f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-04 17:11:31 +0300 - - Build: Add a comment to configure.ac about symbol versioning. - - configure.ac | 4 ++++ - 1 file changed, 4 insertions(+) - -commit bd9cc179e8be3ef515201d3ed9c7dd79ae88869d -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-04 17:06:49 +0300 - - Update TODO. - - TODO | 12 ++++++++++-- - 1 file changed, 10 insertions(+), 2 deletions(-) - -commit 4a238dd9b22f462cac5e199828bf1beb0df05884 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-04 17:05:46 +0300 - - Document --enable-symbol-versions in INSTALL. - - INSTALL | 5 +++++ - 1 file changed, 5 insertions(+) - -commit 88ccf47205d7f3aa314d358c72ef214f10f68b43 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-03 21:16:39 +0300 - - xz: Add incomplete support for --block-list. - - It's broken with threads and when also --block-size is used. - - src/xz/args.c | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - src/xz/args.h | 1 + - src/xz/coder.c | 48 ++++++++++++++++++++++++++++------ - src/xz/coder.h | 4 +++ - src/xz/main.c | 1 + - src/xz/message.c | 6 +++++ - src/xz/xz.1 | 23 +++++++++++++++-- - 7 files changed, 151 insertions(+), 10 deletions(-) - -commit 972179cdcdf5d8949c48ee31737d87d3050b44af -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-07-01 18:44:33 +0300 - - xz: Update the man page about the new field in --robot -lvv. - - src/xz/xz.1 | 18 +++++++++++++++++- - 1 file changed, 17 insertions(+), 1 deletion(-) - -commit 1403707fc64a70976aebe66f8d9a9bd12f73a2c5 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-06-28 10:47:49 +0300 - - liblzma: Check that the first byte of range encoded data is 0x00. - - It is just to be more pedantic and thus perhaps catch broken - files slightly earlier. - - src/liblzma/lzma/lzma_decoder.c | 8 ++++++-- - src/liblzma/rangecoder/range_decoder.h | 12 +++++++++--- - 2 files changed, 15 insertions(+), 5 deletions(-) - -commit eccd8017ffe2c5de473222c4963ec53c62f7fda2 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-06-22 19:00:23 +0300 - - Update NEWS from 5.0.4. - - NEWS | 37 +++++++++++++++++++++++++++++++++++++ - 1 file changed, 37 insertions(+) - -commit 2e6754eac26a431e8d340c28906f63bcd1e177e8 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-06-22 14:34:03 +0300 - - xz: Update man page date to match the latest update. - - src/xz/xz.1 | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit b3235a0b1af45d5e1244cbe3191516966c076fa0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-06-18 21:27:47 +0300 - - Docs: Language fix to 01_compress_easy.c. - - Thanks to Jonathan Nieder. - - doc/examples/01_compress_easy.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit f1675f765fe228cb5a5f904f853445a03e33cfe9 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-06-14 20:15:30 +0300 - - Fix the top-level Makefile.am for the new example programs. - - Makefile.am | 12 ++++++++++-- - 1 file changed, 10 insertions(+), 2 deletions(-) - -commit 3a0c5378abefaf86aa39a62a7c9682bdb21568a1 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-06-14 10:52:33 +0300 - - Docs: Add new example programs. - - These have more comments than the old examples and - human-readable error messages. More tutorial-like examples - are needed but these are a start. - - doc/examples/00_README.txt | 27 ++++ - doc/examples/01_compress_easy.c | 297 ++++++++++++++++++++++++++++++++++++++ - doc/examples/02_decompress.c | 287 ++++++++++++++++++++++++++++++++++++ - doc/examples/03_compress_custom.c | 193 +++++++++++++++++++++++++ - doc/examples/Makefile | 23 +++ - 5 files changed, 827 insertions(+) - -commit 1bd2c2c553e30c4a73cfb82abc6908efd6be6b8d -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-06-14 10:33:27 +0300 - - Docs: Move xz_pipe_comp.c and xz_pipe_decomp.c to doc/examples_old. - - It is good to keep these around to so that if someone has - copied the decompressor bug from xz_pipe_decomp.c he has - an example how to easily fix it. - - doc/{examples => examples_old}/xz_pipe_comp.c | 0 - doc/{examples => examples_old}/xz_pipe_decomp.c | 0 - 2 files changed, 0 insertions(+), 0 deletions(-) - -commit 905f0ab5b5ce544d4b68a2ed6077df0f3d021292 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-06-14 10:33:01 +0300 - - Docs: Fix a bug in xz_pipe_decomp.c example program. - - doc/examples/xz_pipe_decomp.c | 10 +++++++++- - 1 file changed, 9 insertions(+), 1 deletion(-) - -commit 4bd1a3bd5fdf4870b2f96dd0b8a21657c8a58ad8 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-30 23:14:33 +0300 - - Translations: Update the French translation. - - Thanks to Adrien Nader. - - po/fr.po | 148 ++++++++++++++++++++++++++++++++++----------------------------- - 1 file changed, 79 insertions(+), 69 deletions(-) - -commit d2e836f2f3a87df6fe6bb0589b037db51205d910 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-29 23:42:37 +0300 - - Translations: Update the German translation. - - The previous only included the new strings in v5.0. - - po/de.po | 229 +++++++++++++++++++++++++++++++++++++-------------------------- - 1 file changed, 133 insertions(+), 96 deletions(-) - -commit c9a16151577ba459afd6e3528df23bc0ddb95171 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-29 22:26:27 +0300 - - Translations: Update the German translation. - - po/de.po | 169 ++++++++++++++++++++++++++++++++++----------------------------- - 1 file changed, 91 insertions(+), 78 deletions(-) - -commit 1530a74fd48f8493372edad137a24541efe24713 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-29 22:14:21 +0300 - - Translations: Update Polish translation. - - po/pl.po | 283 +++++++++++++++++++++++++++++++++++++-------------------------- - 1 file changed, 165 insertions(+), 118 deletions(-) - -commit d8db706acb8316f9861abd432cfbe001dd6d0c5c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-28 20:42:11 +0300 - - liblzma: Fix possibility of incorrect LZMA_BUF_ERROR. - - lzma_code() could incorrectly return LZMA_BUF_ERROR if - all of the following was true: - - - The caller knows how many bytes of output to expect - and only provides that much output space. - - - When the last output bytes are decoded, the - caller-provided input buffer ends right before - the LZMA2 end of payload marker. So LZMA2 won't - provide more output anymore, but it won't know it - yet and thus won't return LZMA_STREAM_END yet. - - - A BCJ filter is in use and it hasn't left any - unfiltered bytes in the temp buffer. This can happen - with any BCJ filter, but in practice it's more likely - with filters other than the x86 BCJ. - - Another situation where the bug can be triggered happens - if the uncompressed size is zero bytes and no output space - is provided. In this case the decompression can fail even - if the whole input file is given to lzma_code(). - - A similar bug was fixed in XZ Embedded on 2011-09-19. - - src/liblzma/simple/simple_coder.c | 2 +- - tests/Makefile.am | 4 +- - tests/test_bcj_exact_size.c | 112 ++++++++++++++++++++++++++++++++++++++ - 3 files changed, 116 insertions(+), 2 deletions(-) - -commit 3f94b6d87f1b8f1c421ba548f8ebb83dca9c8cda -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-28 15:38:32 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 7769ea051d739a38a1640fd448cf5eb83cb119c6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-28 15:37:43 +0300 - - xz: Don't show a huge number in -vv when memory limit is disabled. - - src/xz/message.c | 12 +++++++++++- - 1 file changed, 11 insertions(+), 1 deletion(-) - -commit ec921105725e4d3ef0a683dd83eee6f24ab60ccd -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-27 22:30:17 +0300 - - xz: Document the "summary" lines of --robot -lvv. - - This documents only the columns that are in v5.0. - The new columns added in the master branch aren't - necessarily stable yet. - - src/xz/xz.1 | 19 +++++++++++++++++++ - 1 file changed, 19 insertions(+) - -commit 27d24eb0a9f6eed96d6a4594c2b0bf7a91d29f9a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-27 21:53:20 +0300 - - xz: Fix output of verbose --robot --list modes. - - It printed the filename in "filename (x/y)" format - which it obviously shouldn't do in robot mode. - - src/xz/message.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit ab25b82a91754d9388c89abddf806424671d9431 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-24 18:33:54 +0300 - - Build: Upgrade m4/acx_pthread.m4 to the latest version. - - m4/ax_pthread.m4 | 98 +++++++++++++++++++++++++++++++++++--------------------- - 1 file changed, 62 insertions(+), 36 deletions(-) - -commit d05d6d65c41a4bc83f162fa3d67c5d84e8751634 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-10 21:15:17 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit e077391982f9f28dbfe542bba8800e7c5b916666 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-05-10 21:14:16 +0300 - - Docs: Cleanup line wrapping a bit. - - README | 12 ++++++------ - doc/history.txt | 49 +++++++++++++++++++++++++------------------------ - 2 files changed, 31 insertions(+), 30 deletions(-) - -commit fc39849c350225c6a1cd7f6e6adff1020521eabc -Author: Benno Schulenberg <bensberg@justemail.net> -Date: 2012-03-13 22:04:04 +0100 - - Fix a few typos and add some missing articles in some documents. - - Also hyphenate several compound adjectives. - - Signed-off-by: Benno Schulenberg <bensberg@justemail.net> - - AUTHORS | 6 +++--- - README | 42 ++++++++++++++++++++--------------------- - doc/faq.txt | 24 ++++++++++++------------ - doc/history.txt | 58 ++++++++++++++++++++++++++++----------------------------- - 4 files changed, 65 insertions(+), 65 deletions(-) - -commit 29fa0566d5df199cb9acb2d17bf7eea61acc7fa1 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-04-29 11:51:25 +0300 - - Windows: Update notes about static linking with MSVC. - - windows/README-Windows.txt | 13 +++++++++---- - 1 file changed, 9 insertions(+), 4 deletions(-) - -commit aac1b31ea4e66cf5a7a8c116bdaa15aa45e6c56e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-04-19 15:25:26 +0300 - - liblzma: Remove outdated comments. - - src/liblzma/simple/simple_coder.c | 3 --- - src/liblzma/simple/simple_private.h | 3 +-- - 2 files changed, 1 insertion(+), 5 deletions(-) - -commit df14a46013bea70c0bd35be7821b0b9108f97de7 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-04-19 14:17:52 +0300 - - DOS: Link against DJGPP's libemu to support FPU emulation. - - This way xz should work on 386SX and 486SX. Floating point - only is needed for verbose output in xz. - - dos/Makefile | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 03ed742a3a4931bb5c821357832083b26f577b13 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-04-19 14:02:25 +0300 - - liblzma: Fix Libs.private in liblzma.pc to include -lrt when needed. - - src/liblzma/liblzma.pc.in | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 8c5b13ad59df70f49429bfdfd6ac120b8f892fda -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-04-19 13:58:55 +0300 - - Docs: Update MINIX 3 information in INSTALL. - - INSTALL | 8 +++++--- - 1 file changed, 5 insertions(+), 3 deletions(-) - -commit c7376fc415a1566f38b2de4b516a17013d516a8b -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-02-22 14:23:13 +0200 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit cff070aba6281ba743d29a62b8c0c66e5da4b2a6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-02-22 14:02:34 +0200 - - Fix exit status of xzgrep when grepping binary files. - - When grepping binary files, grep may exit before it has - read all the input. In this case, gzip -q returns 2 (eating - SIGPIPE), but xz and bzip2 show SIGPIPE as the exit status - (e.g. 141). This causes wrong exit status when grepping - xz- or bzip2-compressed binary files. - - The fix checks for the special exit status that indicates SIGPIPE. - It uses kill -l which should be supported everywhere since it - is in both SUSv2 (1997) and POSIX.1-2008. - - Thanks to James Buren for the bug report. - - src/scripts/xzgrep.in | 3 ++- - 1 file changed, 2 insertions(+), 1 deletion(-) - -commit 41cafb2bf9beea915710ee68f05fe929cd17759c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-02-22 12:08:43 +0200 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 2dcea03712fa881930d69ec9eff70855c3d126d9 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-02-22 12:00:16 +0200 - - Fix compiling with IBM XL C on AIX. - - INSTALL | 36 ++++++++++++++++++++++-------------- - configure.ac | 6 +++++- - 2 files changed, 27 insertions(+), 15 deletions(-) - -commit 7db6bdf4abcf524115be2cf5659ed540cef074c5 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2012-01-10 17:13:03 +0200 - - Tests: Fix a compiler warning with _FORTIFY_SOURCE. - - Reported here: - http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/4927385 - - tests/create_compress_files.c | 3 ++- - 1 file changed, 2 insertions(+), 1 deletion(-) - -commit 694952d545b6cf056547893ced69486eff9ece55 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-12-19 21:21:29 +0200 - - Docs: Explain the stable releases better in README. - - README | 6 +++++- - 1 file changed, 5 insertions(+), 1 deletion(-) - -commit 418fe668b3c53a9a20020b6cc652aaf25c734b29 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-11-07 13:07:52 +0200 - - xz: Show minimum required XZ Utils version in xz -lvv. - - Man page wasn't updated yet. - - src/xz/list.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++------ - 1 file changed, 57 insertions(+), 6 deletions(-) - -commit 7081d82c37326bac97184e338345fa1c327e3580 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-11-04 17:57:16 +0200 - - xz: Fix a typo in a comment. - - Thanks to Bela Lubkin. - - src/xz/args.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 232fe7cd70ad258d6a37f17e860e0f1b1891eeb5 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-11-03 17:08:02 +0200 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 74d2bae4d3449c68453b0473dd3430ce91fd90c1 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-11-03 17:07:22 +0200 - - xz: Fix xz on EBCDIC systems. - - Thanks to Chris Donawa. - - src/xz/coder.c | 5 ++++- - 1 file changed, 4 insertions(+), 1 deletion(-) - -commit 4ac4923f47cc0ef97dd9ca5cfcc44fc53eeab34a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-10-23 17:09:10 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit ab50ae3ef40c81e5bf613905ca3fd636548b75e7 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-10-23 17:08:14 +0300 - - liblzma: Fix invalid free() in the threaded encoder. - - It was triggered if initialization failed e.g. due to - running out of memory. - - Thanks to Arkadiusz Miskiewicz. - - src/liblzma/common/outqueue.c | 4 ++++ - 1 file changed, 4 insertions(+) - -commit 6b620a0f0813d28c3c544b4ff8cb595b38a6e908 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-10-23 17:05:55 +0300 - - liblzma: Fix a deadlock in the threaded encoder. - - It was triggered when reinitializing the encoder, - e.g. when encoding two files. - - src/liblzma/common/stream_encoder_mt.c | 4 +++- - 1 file changed, 3 insertions(+), 1 deletion(-) - -commit bd52cf150ecd51e3ab63a9cc1a3cff6a77500178 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-09-06 12:03:41 +0300 - - Build: Fix "make check" on Windows. - - tests/Makefile.am | 7 +++++-- - windows/build.bash | 2 ++ - 2 files changed, 7 insertions(+), 2 deletions(-) - -commit 5c5b2256969ac473001b7d67615ed3bd0a54cc82 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-08-09 21:19:13 +0300 - - Update THANKS. - - THANKS | 2 ++ - 1 file changed, 2 insertions(+) - -commit 5b1e1f10741af9e4bbe4cfc3261fb7c7b04f7809 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-08-09 21:16:44 +0300 - - Workaround unusual SIZE_MAX on SCO OpenServer. - - src/common/sysdefs.h | 9 ++++++--- - 1 file changed, 6 insertions(+), 3 deletions(-) - -commit e9ed88126eee86e2511fa42681a5c7104820cf0a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-08-06 20:37:28 +0300 - - Run the scripts with the correct shell in test_scripts.sh. - - The scripts are now made executable in the build tree. - This way the scripts can be run like programs in - test_scripts.sh. Previously test_scripts.sh always - used sh but it's not correct if @POSIX_SHELL@ is set - to something else by configure. - - Thanks to Jonathan Nieder for the patch. - - configure.ac | 8 ++++---- - tests/test_scripts.sh | 8 ++++---- - 2 files changed, 8 insertions(+), 8 deletions(-) - -commit 1c673e5681720491a74fc4b2992e075f47302c22 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-07-31 11:01:47 +0300 - - Fix exit status of "xzdiff foo.xz bar.xz". - - xzdiff was clobbering the exit status from diff in a case - statement used to analyze the exit statuses from "xz" when - its operands were two compressed files. Save and restore - diff's exit status to fix this. - - The bug is inherited from zdiff in GNU gzip and was fixed - there on 2009-10-09. - - Thanks to Jonathan Nieder for the patch and - to Peter Pallinger for reporting the bug. - - src/scripts/xzdiff.in | 2 ++ - tests/Makefile.am | 4 +++- - tests/test_scripts.sh | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++ - 3 files changed, 59 insertions(+), 1 deletion(-) - -commit 324cde7a864f4506c32ae7846d688c359a83fe65 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-06-16 12:15:29 +0300 - - liblzma: Remove unneeded semicolon. - - src/liblzma/lz/lz_encoder_hash.h | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 492c86345551a51a29bf18e55fe55a5e86f169ce -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-28 19:24:56 +0300 - - Build: Make configure print if symbol versioning is enabled or not. - - configure.ac | 2 ++ - 1 file changed, 2 insertions(+) - -commit fc4d4436969bd4d71b704d400a165875e596034a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-28 16:43:26 +0300 - - Don't call close(-1) in tuklib_open_stdxxx() on error. - - Thanks to Jim Meyering. - - src/common/tuklib_open_stdxxx.c | 4 +++- - 1 file changed, 3 insertions(+), 1 deletion(-) - -commit bd35d903a04c4d388adb4065b0fa271302380895 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-28 15:55:39 +0300 - - liblzma: Use symbol versioning. - - Symbol versioning is enabled by default on GNU/Linux, - other GNU-based systems, and FreeBSD. - - I'm not sure how stable this is, so it may need - backward-incompatible changes before the next release. - - The idea is that alpha and beta symbols are considered - unstable and require recompiling the applications that - use those symbols. Once a symbol is stable, it may get - extended with new features in ways that don't break - compatibility with older ABI & API. - - The mydist target runs validate_map.sh which should - catch some probable problems in liblzma.map. Otherwise - I would forget to update the map file for new releases. - - Makefile.am | 1 + - configure.ac | 21 +++++++++ - src/liblzma/Makefile.am | 6 +++ - src/liblzma/liblzma.map | 105 ++++++++++++++++++++++++++++++++++++++++++++ - src/liblzma/validate_map.sh | 68 ++++++++++++++++++++++++++++ - 5 files changed, 201 insertions(+) - -commit afbb244362c9426a37ce4eb9d54aab768da3adad -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-28 09:46:46 +0300 - - Translations: Update the Italian translation. - - Thanks to Milo Casagrande. - - po/it.po | 365 +++++++++++++++++++++++++++++++++++++-------------------------- - 1 file changed, 216 insertions(+), 149 deletions(-) - -commit 79bef85e0543c0c3723281c3c817616c6cec343b -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-28 08:46:04 +0300 - - Tests: Add a test file for the bug in the previous commit. - - tests/files/README | 4 ++++ - tests/files/bad-1-block_header-6.xz | Bin 0 -> 72 bytes - 2 files changed, 4 insertions(+) - -commit c0297445064951807803457dca1611b3c47e7f0f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-27 22:25:44 +0300 - - xz: Fix error handling in xz -lvv. - - It could do an invalid free() and read past the end - of the uninitialized filters array. - - src/xz/list.c | 21 ++++++--------------- - 1 file changed, 6 insertions(+), 15 deletions(-) - -commit 8bd91918ac50731f00b1a2a48072980572eb2ff9 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-27 22:09:49 +0300 - - liblzma: Handle allocation failures correctly in lzma_index_init(). - - Thanks to Jim Meyering. - - src/liblzma/common/index.c | 7 +++++-- - 1 file changed, 5 insertions(+), 2 deletions(-) - -commit fe00f95828ef5627721b57e054f7eb2d42a2c961 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-24 00:23:46 +0300 - - Build: Fix checking for system-provided SHA-256. - - configure.ac | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 21b45b9bab541f419712cbfd473ccc31802e0397 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-23 18:30:30 +0300 - - Build: Set GZIP_ENV=-9n in top-level Makefile.am. - - Makefile.am | 3 +++ - 1 file changed, 3 insertions(+) - -commit 48053e8a4550233af46359024538bff90c870ab1 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-22 16:42:11 +0300 - - Update NEWS for 5.0.3. - - NEWS | 32 ++++++++++++++++++++++++++++++++ - 1 file changed, 32 insertions(+) - -commit bba37df2c9e54ad773e15ff00a09d2d6989fb3b2 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-21 16:28:44 +0300 - - Add French translation. - - It is known that the BCJ filter --help text is only - partially translated. - - po/LINGUAS | 1 + - po/fr.po | 864 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 865 insertions(+) - -commit 4161d7634965a7a287bf208dcd79f6185f448fe8 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-21 15:12:10 +0300 - - xz: Translate also the string used to print the program name. - - French needs a space before a colon, e.g. "xz : foo error". - - src/xz/message.c | 6 +++++- - 1 file changed, 5 insertions(+), 1 deletion(-) - -commit b94aa0c8380cdb18cddb33440d625474c16643cf -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-21 15:08:44 +0300 - - liblzma: Try to use SHA-256 from the operating system. - - If the operating system libc or other base libraries - provide SHA-256, use that instead of our own copy. - Note that this doesn't use OpenSSL or libgcrypt or - such libraries to avoid creating dependencies to - other packages. - - This supports at least FreeBSD, NetBSD, OpenBSD, Solaris, - MINIX, and Darwin. They all provide similar but not - identical SHA-256 APIs; everyone is a little different. - - Thanks to Wim Lewis for the original patch, improvements, - and testing. - - configure.ac | 54 +++++++++++++++++++++++++++ - src/liblzma/check/Makefile.inc | 2 + - src/liblzma/check/check.h | 83 ++++++++++++++++++++++++++++++++++++++---- - 3 files changed, 131 insertions(+), 8 deletions(-) - -commit f004128678d43ea10b4a6401aa184cf83252d6ec -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-17 12:52:18 +0300 - - Don't use clockid_t in mythread.h when clock_gettime() isn't available. - - Thanks to Wim Lewis for the patch. - - src/common/mythread.h | 2 ++ - 1 file changed, 2 insertions(+) - -commit f779516f42ebd2db47a5b7d6143459bf7737cf2f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-17 12:26:28 +0300 - - Update THANKS. - - THANKS | 3 +++ - 1 file changed, 3 insertions(+) - -commit 830ba587775bb562f6eaf05cad61bf669d1f8892 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-17 12:21:33 +0300 - - Update INSTALL with a note about linker problem on OpenSolaris x86. - - INSTALL | 23 +++++++++++++++++------ - 1 file changed, 17 insertions(+), 6 deletions(-) - -commit ec7106309c8060e9c646dba20c4f15689a0bbb04 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-17 12:01:37 +0300 - - Build: Fix initialization of enable_check_* variables in configure.ac. - - This doesn't matter much in practice since it is unlikely - that anyone would have such environment variable names. - - Thanks to Wim Lewis. - - configure.ac | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 4c6e146df99696920f12410fb17754412797ef36 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-17 11:54:38 +0300 - - Add underscores to attributes (__attribute((__foo__))). - - src/liblzma/common/alone_decoder.c | 2 +- - src/liblzma/common/alone_encoder.c | 2 +- - src/liblzma/common/block_encoder.c | 2 +- - src/liblzma/common/common.c | 2 +- - src/liblzma/common/common.h | 2 +- - src/liblzma/common/index_decoder.c | 9 +++++---- - src/liblzma/common/index_encoder.c | 11 ++++++----- - src/liblzma/delta/delta_encoder.c | 2 +- - src/liblzma/lz/lz_decoder.c | 2 +- - src/liblzma/lz/lz_encoder.c | 2 +- - src/liblzma/simple/arm.c | 2 +- - src/liblzma/simple/armthumb.c | 2 +- - src/liblzma/simple/ia64.c | 2 +- - src/liblzma/simple/powerpc.c | 2 +- - src/liblzma/simple/simple_coder.c | 2 +- - src/liblzma/simple/sparc.c | 2 +- - src/lzmainfo/lzmainfo.c | 4 ++-- - src/xz/coder.c | 2 +- - src/xz/hardware.h | 2 +- - src/xz/message.c | 2 +- - src/xz/message.h | 18 +++++++++--------- - src/xz/options.c | 6 +++--- - src/xz/signals.c | 2 +- - src/xz/util.h | 6 +++--- - src/xzdec/xzdec.c | 6 +++--- - 25 files changed, 49 insertions(+), 47 deletions(-) - -commit 7a480e485938884ef3021b48c3b0b9f9699dc9b6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-01 12:24:23 +0300 - - xz: Fix input file position when --single-stream is used. - - Now the following works as you would expect: - - echo foo | xz > foo.xz - echo bar | xz >> foo.xz - ( xz -dc --single-stream ; xz -dc --single-stream ) < foo.xz - - Note that it doesn't work if the input is not seekable - or if there is Stream Padding between the concatenated - .xz Streams. - - src/xz/coder.c | 1 + - src/xz/file_io.c | 15 +++++++++++++++ - src/xz/file_io.h | 13 +++++++++++++ - 3 files changed, 29 insertions(+) - -commit c29e6630c1450c630c4e7b783bdd76515db9004c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-05-01 12:15:51 +0300 - - xz: Print the maximum number of worker threads in xz -vv. - - src/xz/coder.c | 4 ++++ - 1 file changed, 4 insertions(+) - -commit 0b77c4a75158ccc416b07d6e81df8ee0abaea720 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-19 10:44:48 +0300 - - Build: Warn if no supported method to detect the number of CPU cores. - - configure.ac | 11 +++++------ - 1 file changed, 5 insertions(+), 6 deletions(-) - -commit e4622df9ab4982f8faa53d85b17be66216175a58 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-19 09:55:06 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 9c1b05828a88eff54409760b92162c7cc2c7cff6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-19 09:20:44 +0300 - - Fix portability problems in mythread.h. - - Use gettimeofday() if clock_gettime() isn't available - (e.g. Darwin). - - The test for availability of pthread_condattr_setclock() - and CLOCK_MONOTONIC was incorrect. Instead of fixing the - #ifdefs, use an Autoconf test. That way if there exists a - system that supports them but doesn't specify the matching - POSIX #defines, the features will still get detected. - - Don't try to use pthread_sigmask() on OpenVMS. It doesn't - have that function. - - Guard mythread.h against being #included multiple times. - - configure.ac | 7 +++++++ - src/common/mythread.h | 31 +++++++++++++++++++++++++++---- - 2 files changed, 34 insertions(+), 4 deletions(-) - -commit 3de00cc75da7b0e7b65e84c62b5351e231f501e9 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-18 19:35:49 +0300 - - Update THANKS. - - THANKS | 2 ++ - 1 file changed, 2 insertions(+) - -commit bd5002f5821e3d1b04f2f56989e4a19318e73633 -Author: Martin Väth <vaeth@mathematik.uni-wuerzburg.de> -Date: 2011-04-15 04:54:49 -0400 - - xzgrep: fix typo in $0 parsing - - Reported-by: Diego Elio Pettenò <flameeyes@gentoo.org> - Signed-off-by: Martin Väth <vaeth@mathematik.uni-wuerzburg.de> - Signed-off-by: Mike Frysinger <vapier@gentoo.org> - - src/scripts/xzgrep.in | 4 ++-- - 1 file changed, 2 insertions(+), 2 deletions(-) - -commit 6ef4eabc0acc49e1bb9dc68064706e19fa9fcf48 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-12 12:48:31 +0300 - - Bump the version number to 5.1.1alpha and liblzma soname to 5.0.99. - - src/liblzma/Makefile.am | 2 +- - src/liblzma/api/lzma/version.h | 2 +- - 2 files changed, 2 insertions(+), 2 deletions(-) - -commit 9a4377be0d21e597c66bad6c7452873aebfb3c1c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-12 12:42:37 +0300 - - Put the unstable APIs behind #ifdef LZMA_UNSTABLE. - - This way people hopefully won't complain if these APIs - change and break code that used an older API. - - src/liblzma/api/lzma/container.h | 4 ++++ - src/liblzma/common/common.h | 2 ++ - src/xz/private.h | 2 ++ - 3 files changed, 8 insertions(+) - -commit 3e321a3acd50002cf6fdfd259e910f56d3389bc3 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-12 11:59:49 +0300 - - Remove doubled words from documentation and comments. - - Spot candidates by running these commands: - git ls-files |xargs perl -0777 -n \ - -e 'while (/\b(then?|[iao]n|i[fst]|but|f?or|at|and|[dt]o)\s+\1\b/gims)' \ - -e '{$n=($` =~ tr/\n/\n/ + 1); ($v=$&)=~s/\n/\\n/g; print "$ARGV:$n:$v\n"}' - - Thanks to Jim Meyering for the original patch. - - doc/lzma-file-format.txt | 4 ++-- - src/liblzma/common/alone_encoder.c | 2 +- - src/liblzma/lzma/lzma2_encoder.c | 2 +- - src/xz/file_io.c | 2 +- - src/xz/xz.1 | 2 +- - windows/INSTALL-Windows.txt | 2 +- - 6 files changed, 7 insertions(+), 7 deletions(-) - -commit d91a84b534b012d19474f2fda1fbcaef873e1ba4 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-12 11:46:01 +0300 - - Update NEWS. - - NEWS | 47 +++++++++++++++++++++++++++++++++++++++++++++-- - 1 file changed, 45 insertions(+), 2 deletions(-) - -commit 14e6ad8cfe0165c1a8beeb5b2a1536558b29b0a1 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-12 11:45:40 +0300 - - Update TODO. - - TODO | 12 +++++++++++- - 1 file changed, 11 insertions(+), 1 deletion(-) - -commit 70e750f59793f9b5cd306a5adce9b8e427739e04 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-12 11:08:55 +0300 - - xz: Update the man page about threading. - - src/xz/xz.1 | 34 ++++++++++++++++++++-------------- - 1 file changed, 20 insertions(+), 14 deletions(-) - -commit 24e0406c0fb7494d2037dec033686faf1bf67068 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 22:06:03 +0300 - - xz: Add support for threaded compression. - - src/xz/args.c | 3 +- - src/xz/coder.c | 202 +++++++++++++++++++++++++++++++++++---------------------- - 2 files changed, 125 insertions(+), 80 deletions(-) - -commit de678e0c924aa79a19293a8a6ed82e8cb6572a42 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 22:03:30 +0300 - - liblzma: Add lzma_stream_encoder_mt() for threaded compression. - - This is the simplest method to do threading, which splits - the uncompressed data into blocks and compresses them - independently from each other. There's room for improvement - especially to reduce the memory usage, but nevertheless, - this is a good start. - - configure.ac | 1 + - src/liblzma/api/lzma/container.h | 163 +++++ - src/liblzma/common/Makefile.inc | 7 + - src/liblzma/common/common.c | 9 +- - src/liblzma/common/common.h | 14 + - src/liblzma/common/outqueue.c | 180 ++++++ - src/liblzma/common/outqueue.h | 155 +++++ - src/liblzma/common/stream_encoder_mt.c | 1011 ++++++++++++++++++++++++++++++++ - 8 files changed, 1539 insertions(+), 1 deletion(-) - -commit 25fe729532cdf4b8fed56a4519b73cf31efaec50 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 21:15:07 +0300 - - liblzma: Add the forgotten lzma_lzma2_block_size(). - - This should have been in 5eefc0086d24a65e136352f8c1d19cefb0cbac7a. - - src/liblzma/lzma/lzma2_encoder.c | 10 ++++++++++ - src/liblzma/lzma/lzma2_encoder.h | 2 ++ - 2 files changed, 12 insertions(+) - -commit 91afb785a1dee34862078d9bf844ef12b8cc3e35 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 21:04:13 +0300 - - liblzma: Document lzma_easy_(enc|dec)oder_memusage() better too. - - src/liblzma/api/lzma/container.h | 9 +++++++++ - 1 file changed, 9 insertions(+) - -commit 4a9905302a9e4a1601ae09d650d3f08ce98ae9ee -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 20:59:07 +0300 - - liblzma: Document lzma_raw_(enc|dec)oder_memusage() better. - - It didn't mention the return value that is used if - an error occurs. - - src/liblzma/api/lzma/filter.h | 8 ++++++-- - 1 file changed, 6 insertions(+), 2 deletions(-) - -commit 0badb0b1bd649163322783b0bd9e590b4bc7a93d -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 19:28:18 +0300 - - liblzma: Use memzero() to initialize supported_actions[]. - - This is cleaner and makes it simpler to add new members - to lzma_action enumeration. - - src/liblzma/common/common.c | 6 ++---- - 1 file changed, 2 insertions(+), 4 deletions(-) - -commit a7934c446a58e20268689899d2a39f50e571f251 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 19:26:27 +0300 - - liblzma: API comment about lzma_allocator with threaded coding. - - src/liblzma/api/lzma/base.h | 18 +++++++++++++----- - 1 file changed, 13 insertions(+), 5 deletions(-) - -commit 5eefc0086d24a65e136352f8c1d19cefb0cbac7a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 19:16:30 +0300 - - liblzma: Add an internal function lzma_mt_block_size(). - - This is based lzma_chunk_size() that was included in some - development version of liblzma. - - src/liblzma/common/filter_encoder.c | 46 ++++++++++++++++++------------------- - src/liblzma/common/filter_encoder.h | 4 ++-- - 2 files changed, 24 insertions(+), 26 deletions(-) - -commit d1199274758049fc523d98c5b860ff814a799eec -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 13:59:50 +0300 - - liblzma: Don't create an empty Block in lzma_stream_buffer_encode(). - - Empty Block was created if the input buffer was empty. - Empty Block wastes a few bytes of space, but more importantly - it triggers a bug in XZ Utils 5.0.1 and older when trying - to decompress such a file. 5.0.1 and older consider such - files to be corrupt. I thought that no encoder creates empty - Blocks when releasing 5.0.2 but I was wrong. - - src/liblzma/common/stream_buffer_encoder.c | 20 +++++++++++++------- - 1 file changed, 13 insertions(+), 7 deletions(-) - -commit 3b22fc2c87ec85fcdd385c163b68fc49c97aa848 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 13:28:40 +0300 - - liblzma: Fix API docs to mention LZMA_UNSUPPORTED_CHECK. - - This return value was missing from the API comments of - four functions. - - src/liblzma/api/lzma/block.h | 1 + - src/liblzma/api/lzma/container.h | 3 +++ - 2 files changed, 4 insertions(+) - -commit 71b9380145dccf001f22e66a06b9d508905c25ce -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 13:21:28 +0300 - - liblzma: Validate encoder arguments better. - - The biggest problem was that the integrity check type - wasn't validated, and e.g. lzma_easy_buffer_encode() - would create a corrupt .xz Stream if given an unsupported - Check ID. Luckily applications don't usually try to use - an unsupport Check ID, so this bug is unlikely to cause - many real-world problems. - - src/liblzma/common/block_buffer_encoder.c | 18 ++++++++++++------ - src/liblzma/common/block_encoder.c | 5 +++++ - src/liblzma/common/stream_buffer_encoder.c | 3 +++ - 3 files changed, 20 insertions(+), 6 deletions(-) - -commit ec7e3dbad704268825fc48f0bdd4577bc46b4f13 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 09:57:30 +0300 - - xz: Move the description of --block-size in --long-help. - - src/xz/message.c | 8 ++++---- - 1 file changed, 4 insertions(+), 4 deletions(-) - -commit cd3086ff443bb282bdf556919c28b3e3cbed8169 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 09:55:35 +0300 - - Docs: Document --single-stream and --block-size. - - src/xz/xz.1 | 38 ++++++++++++++++++++++++++++++++++++-- - 1 file changed, 36 insertions(+), 2 deletions(-) - -commit fb64a4924334e3c440865710990fe08090f2fed0 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 09:27:57 +0300 - - liblzma: Make lzma_stream_encoder_init() static (second try). - - It's an internal function and it's not needed by - anything outside stream_encoder.c. - - src/liblzma/common/Makefile.inc | 1 - - src/liblzma/common/easy_encoder.c | 1 - - src/liblzma/common/stream_encoder.c | 13 ++++++------- - src/liblzma/common/stream_encoder.h | 23 ----------------------- - 4 files changed, 6 insertions(+), 32 deletions(-) - -commit a34730cf6af4d33a4057914e57227b6dfde6567e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-11 08:31:42 +0300 - - Revert "liblzma: Make lzma_stream_encoder_init() static." - - This reverts commit 352ac82db5d3f64585c07b39e4759388dec0e4d7. - I don't know what I was thinking. - - src/liblzma/common/Makefile.inc | 1 + - src/liblzma/common/stream_encoder.c | 9 +++++---- - src/liblzma/common/stream_encoder.h | 23 +++++++++++++++++++++++ - 3 files changed, 29 insertions(+), 4 deletions(-) - -commit 9f0a806aef7ea79718e3f1f2baf3564295229a27 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-10 21:23:21 +0300 - - Revise mythread.h. - - This adds: - - - mythread_sync() macro to create synchronized blocks - - - mythread_cond structure and related functions - and macros for condition variables with timed - waiting using a relative timeout - - - mythread_create() to create a thread with all - signals blocked - - Some of these wouldn't need to be inline functions, - but I'll keep them this way for now for simplicity. - - For timed waiting on a condition variable, librt is - now required on some systems to use clock_gettime(). - configure.ac was updated to handle this. - - configure.ac | 1 + - src/common/mythread.h | 200 +++++++++++++++++++++++++++++++++++++++++++++----- - 2 files changed, 181 insertions(+), 20 deletions(-) - -commit 352ac82db5d3f64585c07b39e4759388dec0e4d7 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-10 20:37:36 +0300 - - liblzma: Make lzma_stream_encoder_init() static. - - It's an internal function and it's not needed by - anything outside stream_encoder.c. - - src/liblzma/common/Makefile.inc | 1 - - src/liblzma/common/stream_encoder.c | 9 ++++----- - src/liblzma/common/stream_encoder.h | 23 ----------------------- - 3 files changed, 4 insertions(+), 29 deletions(-) - -commit 9e807fe3fe79618ac48f58207cf7082ea20a6928 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-10 14:58:10 +0300 - - DOS: Update the docs and include notes about 8.3 filenames. - - dos/{README => INSTALL.txt} | 13 +---- - dos/README.txt | 123 ++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 125 insertions(+), 11 deletions(-) - -commit ebd54dbd6e481d31e80757f900ac8109ad1423c6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-10 13:09:42 +0300 - - xz/DOS: Add experimental 8.3 filename support. - - This is incompatible with the 8.3 support patch made by - Juan Manuel Guerrero. I think this one is nicer, but - I need to get feedback from DOS users before saying - that this is the final version of 8.3 filename support. - - src/xz/suffix.c | 176 +++++++++++++++++++++++++++++++++++++++++++++++++++++--- - 1 file changed, 167 insertions(+), 9 deletions(-) - -commit cd4fe97852bcaeffe674ee51b4613709292a0972 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-10 12:47:47 +0300 - - xz/DOS: Be more careful with the destination file. - - Try to avoid overwriting the source file if --force is - used and the generated destination filename refers to - the source file. This can happen with 8.3 filenames where - extra characters are ignored. - - If the generated output file refers to a special file - like "con" or "prn", refuse to write to it even if --force - is used. - - src/xz/file_io.c | 35 +++++++++++++++++++++++++++++++++-- - 1 file changed, 33 insertions(+), 2 deletions(-) - -commit 607f9f98ae5ef6d49f4c21c806d462bf6b3d6796 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-09 18:29:30 +0300 - - Update THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit fca396b37410d272b754843a5dc13847be443a3a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-09 18:28:58 +0300 - - liblzma: Add missing #ifdefs to filter_common.c. - - Passing --disable-decoders to configure broke a few - encoders due to missing #ifdefs in filter_common.c. - - Thanks to Jason Gorski for the patch. - - src/liblzma/common/filter_common.c | 6 +++--- - 1 file changed, 3 insertions(+), 3 deletions(-) - -commit b03f6cd3ebadd675f2cc9d518cb26fa860269447 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-09 15:24:59 +0300 - - xz: Avoid unneeded fstat() on DOS-like systems. - - src/xz/file_io.c | 14 ++++++++------ - 1 file changed, 8 insertions(+), 6 deletions(-) - -commit 335fe260a81f61ec99ff5940df733b4c50aedb7c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-09 15:11:13 +0300 - - xz: Minor internal changes to handling of --threads. - - Now it always defaults to one thread. Maybe this - will change again if a threading method is added - that doesn't affect memory usage. - - src/xz/args.c | 4 ++-- - src/xz/hardware.c | 24 ++++++++++++------------ - src/xz/hardware.h | 9 ++++----- - 3 files changed, 18 insertions(+), 19 deletions(-) - -commit 9edd6ee895fbe71d245a173f48e511f154a99875 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-08 17:53:05 +0300 - - xz: Change size_t to uint32_t in a few places. - - src/xz/coder.c | 6 +++--- - src/xz/coder.h | 2 +- - 2 files changed, 4 insertions(+), 4 deletions(-) - -commit 411013ea4506a6df24d35a060fcbd73a57b73eb3 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-08 17:48:41 +0300 - - xz: Fix a typo in a comment. - - src/xz/coder.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit b34c5ce4b22e8d7b81f9895d15054af41d17f805 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-05 22:41:33 +0300 - - liblzma: Use TUKLIB_GNUC_REQ to check GCC version in sha256.c. - - src/liblzma/check/sha256.c | 6 +++--- - 1 file changed, 3 insertions(+), 3 deletions(-) - -commit db33117cc85c17e0b897b5312bd5eb43aac41c03 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-05 17:12:20 +0300 - - Build: Upgrade m4/acx_pthread.m4 to the latest version. - - It was renamed to ax_pthread.m4 in Autoconf Archive. - - configure.ac | 2 +- - m4/{acx_pthread.m4 => ax_pthread.m4} | 170 ++++++++++++++++++----------------- - 2 files changed, 88 insertions(+), 84 deletions(-) - -commit 1039bfcfc098b69d56ecb39d198a092552eacf6d -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-05 15:27:26 +0300 - - xz: Use posix_fadvise() if it is available. - - configure.ac | 3 +++ - src/xz/file_io.c | 15 +++++++++++++++ - 2 files changed, 18 insertions(+) - -commit 1ef3cf44a8eb9512480af4482a5232ea08363b14 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-05 15:13:29 +0300 - - xz: Call lzma_end(&strm) before exiting if debugging is enabled. - - src/xz/coder.c | 10 ++++++++++ - src/xz/coder.h | 5 +++++ - src/xz/main.c | 4 ++++ - 3 files changed, 19 insertions(+) - -commit bd432015d33dcade611d297bc01eb0700088ef6c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-02 14:49:56 +0300 - - liblzma: Fix a memory leak in stream_encoder.c. - - It leaks old filter options structures (hundred bytes or so) - every time the lzma_stream is reinitialized. With the xz tool, - this happens when compressing multiple files. - - src/liblzma/common/stream_encoder.c | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 16889013214e7620d204b6e6c1bf9f3103a13655 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-04-01 08:47:20 +0300 - - Updated NEWS for 5.0.2. - - NEWS | 18 ++++++++++++++++++ - 1 file changed, 18 insertions(+) - -commit 85cdf7dd4e97b078e7b929e47f55a7f1da36010f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-03-31 15:06:58 +0300 - - Update INSTALL with another note about IRIX. - - INSTALL | 4 ++++ - 1 file changed, 4 insertions(+) - -commit c3f4995586873d6a4fb7e451010a128571a9a370 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-03-31 12:22:55 +0300 - - Tests: Add a new file to test empty LZMA2 streams. - - tests/files/README | 4 ++++ - tests/files/good-1-lzma2-5.xz | Bin 0 -> 52 bytes - 2 files changed, 4 insertions(+) - -commit 0d21f49a809dc2088da6cc0da7f948404df7ecfa -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-03-31 11:54:48 +0300 - - liblzma: Fix decoding of LZMA2 streams having no uncompressed data. - - The decoder considered empty LZMA2 streams to be corrupt. - This shouldn't matter much with .xz files, because no encoder - creates empty LZMA2 streams in .xz. This bug is more likely - to cause problems in applications that use raw LZMA2 streams. - - src/liblzma/lzma/lzma2_decoder.c | 8 ++++---- - 1 file changed, 4 insertions(+), 4 deletions(-) - -commit 40277998cb9bad564ce4827aff152e6e1c904dfa -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-03-24 01:42:49 +0200 - - Scripts: Better fix for xzgrep. - - Now it uses "grep -q". - - Thanks to Gregory Margo. - - src/scripts/xzgrep.in | 8 ++++++-- - 1 file changed, 6 insertions(+), 2 deletions(-) - -commit 2118733045ad0ca183a3f181a0399baf876983a6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-03-24 01:22:18 +0200 - - Updated THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit c7210d9a3fca6f31a57208bfddfc9ab20a2e097a -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-03-24 01:21:32 +0200 - - Scripts: Fix xzgrep -l. - - It didn't work at all. It tried to use the -q option - for grep, but it appended it after "--". This works - around it by redirecting to /dev/null. The downside - is that this can be slower with big files compared - to proper use of "grep -q". - - Thanks to Gregory Margo. - - src/scripts/xzgrep.in | 4 ++-- - 1 file changed, 2 insertions(+), 2 deletions(-) - -commit 4eb83e32046a6d670862bc91c3d82530963b455e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-03-19 13:08:22 +0200 - - Scripts: Add lzop (.lzo) support to xzdiff and xzgrep. - - src/scripts/xzdiff.1 | 6 ++++-- - src/scripts/xzdiff.in | 22 ++++++++++++++-------- - src/scripts/xzgrep.1 | 11 +++++++---- - src/scripts/xzgrep.in | 5 +++-- - 4 files changed, 28 insertions(+), 16 deletions(-) - -commit 923b22483bd9356f3219b2b784d96f455f4dc499 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-03-18 19:10:30 +0200 - - xz: Add --block-size=SIZE. - - This uses LZMA_FULL_FLUSH every SIZE bytes of input. - - Man page wasn't updated yet. - - src/xz/args.c | 7 +++++++ - src/xz/coder.c | 50 ++++++++++++++++++++++++++++++++++++++++---------- - src/xz/coder.h | 3 +++ - src/xz/message.c | 4 ++++ - 4 files changed, 54 insertions(+), 10 deletions(-) - -commit 57597d42ca1740ad506437be168d800a50f1a0ad -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-03-18 18:19:19 +0200 - - xz: Add --single-stream. - - This can be useful when there is garbage after the - compressed stream (.xz, .lzma, or raw stream). - - Man page wasn't updated yet. - - src/xz/args.c | 6 ++++++ - src/xz/coder.c | 11 +++++++++-- - src/xz/coder.h | 3 +++ - src/xz/message.c | 6 +++++- - 4 files changed, 23 insertions(+), 3 deletions(-) - -commit 96f94bc925d579a700147fa5d7793b64d69cfc18 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-02-04 22:49:31 +0200 - - xz: Clean up suffix.c. - - struct suffix_pair isn't needed in compresed_name() - so get rid of it there. - - src/xz/suffix.c | 44 ++++++++++++++++++++------------------------ - 1 file changed, 20 insertions(+), 24 deletions(-) - -commit 8930c7ae3f82bdae15aa129f01de08be23d7e8d7 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-02-04 11:29:47 +0200 - - xz: Check if the file already has custom suffix when compressing. - - Now "xz -S .test foo.test" refuses to compress the - file because it already has the suffix .test. The man - page had it documented this way already. - - src/xz/suffix.c | 9 +++++++++ - 1 file changed, 9 insertions(+) - -commit 940d5852c6cf08abccc6befd9d1b5411c9076a58 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-02-02 23:01:51 +0200 - - Updated THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 4ebe65f839613f27f127bab7b8c347d982330ee3 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-02-02 23:00:33 +0200 - - Translations: Add Polish translation. - - Thanks to Jakub Bogusz. - - po/LINGUAS | 1 + - po/pl.po | 825 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 2 files changed, 826 insertions(+) - -commit fc1d292dca1925dfd17174f443f91a696ecd5bf8 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-02-02 22:24:00 +0200 - - Updated THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit 6dd061adfd2775428b079eb03d6fd47d7c0f1ffe -Merge: 9d542ce 5fbce0b -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-02-06 20:13:01 +0200 - - Merge commit '5fbce0b8d96dc96775aa0215e3581addc830e23d' - -commit 5fbce0b8d96dc96775aa0215e3581addc830e23d -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-01-28 20:16:57 +0200 - - Update NEWS for 5.0.1. - - NEWS | 14 ++++++++++++++ - 1 file changed, 14 insertions(+) - -commit 03ebd1bbb314f9f204940219a835c883bf442475 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-01-26 12:19:08 +0200 - - xz: Fix --force on setuid/setgid/sticky and multi-hardlink files. - - xz didn't compress setuid/setgid/sticky files and files - with multiple hard links even with --force. This bug was - introduced in 23ac2c44c3ac76994825adb7f9a8f719f78b5ee4. - - Thanks to Charles Wilson. - - src/xz/file_io.c | 15 +++++++-------- - 1 file changed, 7 insertions(+), 8 deletions(-) - -commit 9d542ceebcbe40b174169c132ccfcdc720ca7089 -Merge: 4f2c69a 7bd0a5e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-01-19 11:45:35 +0200 - - Merge branch 'v5.0' - -commit 7bd0a5e7ccc354f7c2e95c8bc27569c820f6a136 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-01-18 21:25:24 +0200 - - Updated THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit f71c4e16e913f660977526f0ef8d2acdf458d7c9 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2011-01-18 21:23:50 +0200 - - Add alloc_size and malloc attributes to a few functions. - - Thanks to Cristian Rodríguez for the original patch. - - src/common/sysdefs.h | 6 ++++++ - src/liblzma/common/common.h | 2 +- - src/xz/util.h | 5 +++-- - 3 files changed, 10 insertions(+), 3 deletions(-) - -commit 316cbe24465143edde8f6ffb7532834b7b2ea93f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-12-13 16:36:33 +0200 - - Scripts: Fix gzip and bzip2 support in xzdiff. - - src/scripts/xzdiff.in | 12 ++++++------ - 1 file changed, 6 insertions(+), 6 deletions(-) - -commit 4f2c69a4e3e0aee2e37b0b1671d34086e20c8ac6 -Merge: adb89e6 9311774 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-12-12 23:13:22 +0200 - - Merge branch 'v5.0' - -commit 9311774c493c19deab51ded919dcd2e9c4aa2829 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-12-12 21:23:55 +0200 - - Build: Enable ASM on DJGPP by default. - - configure.ac | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit 4a42aaee282fc73b482581684d65110506d5efdd -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-12-12 16:09:42 +0200 - - Updated THANKS. - - THANKS | 1 + - 1 file changed, 1 insertion(+) - -commit ce56f63c41ee210e6308090eb6d49221fdf67d6c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-12-12 16:07:11 +0200 - - Add missing PRIx32 and PRIx64 compatibility definitions. - - This fixes portability to systems that lack C99 inttypes.h. - - Thanks to Juan Manuel Guerrero. - - src/common/sysdefs.h | 9 +++++++++ - 1 file changed, 9 insertions(+) - -commit e6baedddcf54e7da049ebc49183565b99facd4c7 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-12-12 14:50:04 +0200 - - DOS-like: Treat \ and : as directory separators in addition to /. - - Juan Manuel Guerrero had fixed this in his XZ Utils port - to DOS/DJGPP. The bug affects also Windows and OS/2. - - src/xz/suffix.c | 33 +++++++++++++++++++++++++++++---- - 1 file changed, 29 insertions(+), 4 deletions(-) - -commit adb89e68d43a4cadb0c215b45ef7a75737c9c3ec -Merge: 7c24e0d b7afd3e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-12-07 18:53:04 +0200 - - Merge branch 'v5.0' - -commit b7afd3e22a8fac115b75c738d40d3eb1de7e286f -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-12-07 18:52:04 +0200 - - Translations: Fix Czech translation of "sparse file". - - Thanks to Petr Hubený and Marek Černocký. - - po/cs.po | 88 ++++++++++++++++++++++++++++++++-------------------------------- - 1 file changed, 44 insertions(+), 44 deletions(-) - -commit 7c24e0d1b8a2e86e9263b0d56d39621e01aed7af -Merge: b4d42f1 3e56470 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-11-15 14:33:01 +0200 - - Merge branch 'v5.0' - -commit 3e564704bc6f463cb2db11e3f3f0dbd71d85992e -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-11-15 14:28:26 +0200 - - liblzma: Document the return value of lzma_lzma_preset(). - - src/liblzma/api/lzma/lzma.h | 3 +++ - 1 file changed, 3 insertions(+) - -commit 2964d8d691ed92abdcf214888d79ad6d79774735 -Author: Jonathan Nieder <jrnieder@gmail.com> -Date: 2010-11-12 15:22:13 -0600 - - Simplify paths in generated API docs - - Currently the file list generated by Doxygen has src/ at the - beginning of each path. Paths like common/sysdefs.h and - liblzma/api/lzma.h are easier to read without such a prefix. - - Builds from a separate build directory with - - mkdir build - cd build - ../configure - doxygen Doxyfile - - include an even longer prefix /home/someone/src/xz/src; this - patch has the nice side-effect of eliminating that prefix, too. - - Fixes: http://bugs.debian.org/572273 - - Doxyfile.in | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -commit b4d42f1a7120e2cefeb2f14425efe2ca6db85416 -Author: Anders F Bjorklund <afb@users.sourceforge.net> -Date: 2010-11-05 12:56:11 +0100 - - add build script for macosx universal - - macosx/build.sh | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 1 file changed, 92 insertions(+) - -commit 15ee6935abe4a2fc76639ee342ca2e69af3e0ad6 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-11-04 18:31:40 +0200 - - Update the copies of GPLv2 and LGPLv2.1 from gnu.org. - - There are only a few white space changes. - - COPYING.GPLv2 | 14 +++++++------- - COPYING.LGPLv2.1 | 16 +++++++--------- - 2 files changed, 14 insertions(+), 16 deletions(-) - -commit 8e355f7fdbeee6fe394eb02a28f267ce99a882a2 -Merge: 974ebe6 37c2565 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-10-26 15:53:06 +0300 - - Merge branch 'v5.0' - -commit 37c25658efd25b034266daf87cd381d20d1df776 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-10-26 15:48:48 +0300 - - Build: Copy the example programs to $docdir/examples. - - The example programs by Daniel Mealha Cabrita were included - in the git repository, but I had forgot to add them to - Makefile.am. Thus, they didn't get included in the source - package at all by "make dist". - - Makefile.am | 5 +++++ - windows/build.bash | 3 ++- - 2 files changed, 7 insertions(+), 1 deletion(-) - -commit 974ebe63497bdf0d262e06474f0dd5a70b1dd000 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-10-26 10:36:41 +0300 - - liblzma: Rename a few variables and constants. - - This has no semantic changes. I find the new names slightly - more logical and they match the names that are already used - in XZ Embedded. - - The name fastpos wasn't changed (not worth the hassle). - - src/liblzma/lzma/fastpos.h | 55 +++++------ - src/liblzma/lzma/lzma2_encoder.c | 2 +- - src/liblzma/lzma/lzma_common.h | 45 ++++----- - src/liblzma/lzma/lzma_decoder.c | 58 +++++------ - src/liblzma/lzma/lzma_encoder.c | 56 +++++------ - src/liblzma/lzma/lzma_encoder_optimum_fast.c | 9 +- - src/liblzma/lzma/lzma_encoder_optimum_normal.c | 128 ++++++++++++------------- - src/liblzma/lzma/lzma_encoder_private.h | 16 ++-- - 8 files changed, 183 insertions(+), 186 deletions(-) - -commit 7c427ec38d016c0070a42315d752857e33792fc4 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-10-25 12:59:25 +0300 - - Bump version 5.1.0alpha. - - src/liblzma/api/lzma/version.h | 4 ++-- - 1 file changed, 2 insertions(+), 2 deletions(-) - -commit e45929260cd902036efd40c5610a8d0a50d5712b -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-10-23 17:25:52 +0300 - - Build: Fix mydist rule when .git doesn't exist. - - Makefile.am | 1 + - 1 file changed, 1 insertion(+) - -commit 6e1326fcdf6b6209949be57cfe3ad4b781b65168 -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-10-23 14:15:35 +0300 - - Add NEWS for 5.0.0. - - NEWS | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ - 1 file changed, 62 insertions(+) - -commit b667a3ef6338a2c1db7b7706b1f6c99ea392221c -Author: Lasse Collin <lasse.collin@tukaani.org> -Date: 2010-10-23 14:02:53 +0300 - - Bump version to 5.0.0 and liblzma version-info to 5:0:0. - - src/liblzma/Makefile.am | 2 +- - src/liblzma/api/lzma/version.h | 8 ++++---- - 2 files changed, 5 insertions(+), 5 deletions(-) @@ -202,9 +202,77 @@ XZ Utils https://translationproject.org/html/translators.html - Several strings will change in a future version of xz so if you - wish to start a new translation, look at the code in the xz git - repository instead of a 5.2.x release. + Below are notes and testing instructions specific to xz + translations. + + Testing can be done by installing xz into a temporary directory: + + ./configure --disable-shared --prefix=/tmp/xz-test + # <Edit the .po file in the po directory.> + make -C po update-po + make install + bash debug/translation.bash | less + bash debug/translation.bash | less -S # For --list outputs + + Repeat the above as needed (no need to re-run configure though). + + Note especially the following: + + - The output of --help and --long-help must look nice on + an 80-column terminal. It's OK to add extra lines if needed. + + - In contrast, don't add extra lines to error messages and such. + They are often preceded with e.g. a filename on the same line, + so you have no way to predict where to put a \n. Let the terminal + do the wrapping even if it looks ugly. Adding new lines will be + even uglier in the generic case even if it looks nice in a few + limited examples. + + - Be careful with column alignment in tables and table-like output + (--list, --list --verbose --verbose, --info-memory, --help, and + --long-help): + + * All descriptions of options in --help should start in the + same column (but it doesn't need to be the same column as + in the English messages; just be consistent if you change it). + Check that both --help and --long-help look OK, since they + share several strings. + + * --list --verbose and --info-memory print lines that have + the format "Description: %s". If you need a longer + description, you can put extra space between the colon + and %s. Then you may need to add extra space to other + strings too so that the result as a whole looks good (all + values start at the same column). + + * The columns of the actual tables in --list --verbose --verbose + should be aligned properly. Abbreviate if necessary. It might + be good to keep at least 2 or 3 spaces between column headings + and avoid spaces in the headings so that the columns stand out + better, but this is a matter of opinion. Do what you think + looks best. + + - Be careful to put a period at the end of a sentence when the + original version has it, and don't put it when the original + doesn't have it. Similarly, be careful with \n characters + at the beginning and end of the strings. + + - Read the TRANSLATORS comments that have been extracted from the + source code and included in xz.pot. Some comments suggest + testing with a specific command which needs an .xz file. You + may use e.g. any tests/files/good-*.xz. However, these test + commands are included in translations.bash output, so reading + translations.bash output carefully can be enough. + + - If you find language problems in the original English strings, + feel free to suggest improvements. Ask if something is unclear. + + - The translated messages should be understandable (sometimes this + may be a problem with the original English messages too). Don't + make a direct word-by-word translation from English especially if + the result doesn't sound good in your language. + + Thanks for your help! 5. Other implementations of the .xz format @@ -42,8 +42,11 @@ has been important. :-) In alphabetical order: - Michael Fox - Mike Frysinger - Daniel Richard G. + - Tomasz Gajc - Bjarni Ingi Gislason + - John Paul Adrian Glaubitz - Bill Glessner + - Michał Górny - Jason Gorski - Juan Manuel Guerrero - Diederik de Haas @@ -51,6 +54,8 @@ has been important. :-) In alphabetical order: - Christian Hesse - Vincenzo Innocente - Peter Ivanov + - Nicholas Jackson + - Sam James - Jouk Jansen - Jun I Jin - Kiyoshi Kanazawa @@ -61,8 +66,10 @@ has been important. :-) In alphabetical order: - Jan Kratochvil - Christian Kujau - Stephan Kulow + - Ilya Kurdyukov - Peter Lawler - James M Leddy + - Vincent Lefevre - Hin-Tak Leung - Andraž 'ruskie' Levstik - Cary Lewis @@ -79,6 +86,8 @@ has been important. :-) In alphabetical order: - Ivan A. Melnikov - Jim Meyering - Arkadiusz Miskiewicz + - Nathan Moinvaziri + - Étienne Mollier - Conley Moorhous - Rafał Mużyło - Adrien Nader @@ -59,8 +59,6 @@ Missing features - Implement threaded match finders. - Implement pigz-style threading in LZMA2. - Multithreaded decompression - Buffer-to-buffer coding could use less RAM (especially when decompressing LZMA1 or LZMA2). diff --git a/src/common/tuklib_common.h b/src/common/tuklib_common.h index 31fbab58b005..b1f531ea4a68 100644 --- a/src/common/tuklib_common.h +++ b/src/common/tuklib_common.h @@ -14,7 +14,7 @@ #define TUKLIB_COMMON_H // The config file may be replaced by a package-specific file. -// It should include at least stddef.h, inttypes.h, and limits.h. +// It should include at least stddef.h, stdbool.h, inttypes.h, and limits.h. #include "tuklib_config.h" // TUKLIB_SYMBOL_PREFIX is prefixed to all symbols exported by diff --git a/src/common/tuklib_config.h b/src/common/tuklib_config.h index 549cb24d7738..9d470ba732f1 100644 --- a/src/common/tuklib_config.h +++ b/src/common/tuklib_config.h @@ -1,7 +1,10 @@ +// If config.h isn't available, assume that the headers required by +// tuklib_common.h are available. This is required by crc32_tablegen.c. #ifdef HAVE_CONFIG_H # include "sysdefs.h" #else # include <stddef.h> +# include <stdbool.h> # include <inttypes.h> # include <limits.h> #endif diff --git a/src/common/tuklib_integer.h b/src/common/tuklib_integer.h index 6f44a7a0ac22..b58ef68d0905 100644 --- a/src/common/tuklib_integer.h +++ b/src/common/tuklib_integer.h @@ -17,8 +17,8 @@ /// - Byte swapping: bswapXX(num) /// - Byte order conversions to/from native (byteswaps if Y isn't /// the native endianness): convXXYe(num) -/// - Unaligned reads (16/32-bit only): readXXYe(ptr) -/// - Unaligned writes (16/32-bit only): writeXXYe(ptr, num) +/// - Unaligned reads: readXXYe(ptr) +/// - Unaligned writes: writeXXYe(ptr, num) /// - Aligned reads: aligned_readXXYe(ptr) /// - Aligned writes: aligned_writeXXYe(ptr, num) /// @@ -343,6 +343,46 @@ read32le(const uint8_t *buf) } +static inline uint64_t +read64be(const uint8_t *buf) +{ +#if defined(WORDS_BIGENDIAN) || defined(TUKLIB_FAST_UNALIGNED_ACCESS) + uint64_t num = read64ne(buf); + return conv64be(num); +#else + uint64_t num = (uint64_t)buf[0] << 56; + num |= (uint64_t)buf[1] << 48; + num |= (uint64_t)buf[2] << 40; + num |= (uint64_t)buf[3] << 32; + num |= (uint64_t)buf[4] << 24; + num |= (uint64_t)buf[5] << 16; + num |= (uint64_t)buf[6] << 8; + num |= (uint64_t)buf[7]; + return num; +#endif +} + + +static inline uint64_t +read64le(const uint8_t *buf) +{ +#if !defined(WORDS_BIGENDIAN) || defined(TUKLIB_FAST_UNALIGNED_ACCESS) + uint64_t num = read64ne(buf); + return conv64le(num); +#else + uint64_t num = (uint64_t)buf[0]; + num |= (uint64_t)buf[1] << 8; + num |= (uint64_t)buf[2] << 16; + num |= (uint64_t)buf[3] << 24; + num |= (uint64_t)buf[4] << 32; + num |= (uint64_t)buf[5] << 40; + num |= (uint64_t)buf[6] << 48; + num |= (uint64_t)buf[7] << 56; + return num; +#endif +} + + // NOTE: Possible byte swapping must be done in a macro to allow the compiler // to optimize byte swapping of constants when using glibc's or *BSD's // byte swapping macros. The actual write is done in an inline function @@ -350,11 +390,13 @@ read32le(const uint8_t *buf) #if defined(WORDS_BIGENDIAN) || defined(TUKLIB_FAST_UNALIGNED_ACCESS) # define write16be(buf, num) write16ne(buf, conv16be(num)) # define write32be(buf, num) write32ne(buf, conv32be(num)) +# define write64be(buf, num) write64ne(buf, conv64be(num)) #endif #if !defined(WORDS_BIGENDIAN) || defined(TUKLIB_FAST_UNALIGNED_ACCESS) # define write16le(buf, num) write16ne(buf, conv16le(num)) # define write32le(buf, num) write32ne(buf, conv32le(num)) +# define write64le(buf, num) write64ne(buf, conv64le(num)) #endif diff --git a/src/liblzma/api/lzma/base.h b/src/liblzma/api/lzma/base.h index 51421353ca1d..ff2d86968ee8 100644 --- a/src/liblzma/api/lzma/base.h +++ b/src/liblzma/api/lzma/base.h @@ -240,6 +240,36 @@ typedef enum { * can be a sign of a bug in liblzma. See the documentation * how to report bugs. */ + + LZMA_SEEK_NEEDED = 12, + /**< + * \brief Request to change the input file position + * + * Some coders can do random access in the input file. The + * initialization functions of these coders take the file size + * as an argument. No other coders can return LZMA_SEEK_NEEDED. + * + * When this value is returned, the application must seek to + * the file position given in lzma_stream.seek_pos. This value + * is guaranteed to never exceed the file size that was + * specified at the coder initialization. + * + * After seeking the application should read new input and + * pass it normally via lzma_stream.next_in and .avail_in. + */ + + /* + * These eumerations may be used internally by liblzma + * but they will never be returned to applications. + */ + LZMA_RET_INTERNAL1 = 101, + LZMA_RET_INTERNAL2 = 102, + LZMA_RET_INTERNAL3 = 103, + LZMA_RET_INTERNAL4 = 104, + LZMA_RET_INTERNAL5 = 105, + LZMA_RET_INTERNAL6 = 106, + LZMA_RET_INTERNAL7 = 107, + LZMA_RET_INTERNAL8 = 108 } lzma_ret; @@ -520,7 +550,19 @@ typedef struct { void *reserved_ptr2; void *reserved_ptr3; void *reserved_ptr4; - uint64_t reserved_int1; + + /** + * \brief New seek input position for LZMA_SEEK_NEEDED + * + * When lzma_code() returns LZMA_SEEK_NEEDED, the new input position + * needed by liblzma will be available seek_pos. The value is + * guaranteed to not exceed the file size that was specified when + * this lzma_stream was initialized. + * + * In all other situations the value of this variable is undefined. + */ + uint64_t seek_pos; + uint64_t reserved_int2; size_t reserved_int3; size_t reserved_int4; diff --git a/src/liblzma/api/lzma/bcj.h b/src/liblzma/api/lzma/bcj.h index 8e37538ad4af..a1738d93b790 100644 --- a/src/liblzma/api/lzma/bcj.h +++ b/src/liblzma/api/lzma/bcj.h @@ -49,9 +49,13 @@ * Filter for SPARC binaries. */ +#define LZMA_FILTER_ARM64 LZMA_VLI_C(0x0A) + /**< + * Filter for ARM64 binaries. + */ /** - * \brief Options for BCJ filters + * \brief Options for BCJ filters (except ARM64) * * The BCJ filters never change the size of the data. Specifying options * for them is optional: if pointer to options is NULL, default value is diff --git a/src/liblzma/api/lzma/container.h b/src/liblzma/api/lzma/container.h index 9a9ff1c9a486..6e432a1f3b14 100644 --- a/src/liblzma/api/lzma/container.h +++ b/src/liblzma/api/lzma/container.h @@ -69,7 +69,12 @@ typedef struct { * * Set this to zero if no flags are wanted. * - * No flags are currently supported. + * Encoder: No flags are currently supported. + * + * Decoder: Bitwise-or of zero or more of the decoder flags: + * LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, + * LZMA_TELL_ANY_CHECK, LZMA_IGNORE_CHECK, + * LZMA_CONCATENATED, LZMA_FAIL_FAST */ uint32_t flags; @@ -79,7 +84,7 @@ typedef struct { uint32_t threads; /** - * \brief Maximum uncompressed size of a Block + * \brief Encoder only: Maximum uncompressed size of a Block * * The encoder will start a new .xz Block every block_size bytes. * Using LZMA_FULL_FLUSH or LZMA_FULL_BARRIER with lzma_code() @@ -135,7 +140,7 @@ typedef struct { uint32_t timeout; /** - * \brief Compression preset (level and possible flags) + * \brief Encoder only: Compression preset * * The preset is set just like with lzma_easy_encoder(). * The preset is ignored if filters below is non-NULL. @@ -143,7 +148,7 @@ typedef struct { uint32_t preset; /** - * \brief Filter chain (alternative to a preset) + * \brief Encoder only: Filter chain (alternative to a preset) * * If this is NULL, the preset above is used. Otherwise the preset * is ignored and the filter chain specified here is used. @@ -151,7 +156,7 @@ typedef struct { const lzma_filter *filters; /** - * \brief Integrity check type + * \brief Encoder only: Integrity check type * * See check.h for available checks. The xz command line tool * defaults to LZMA_CHECK_CRC64, which is a good choice if you @@ -173,8 +178,50 @@ typedef struct { uint32_t reserved_int2; uint32_t reserved_int3; uint32_t reserved_int4; - uint64_t reserved_int5; - uint64_t reserved_int6; + + /** + * \brief Memory usage limit to reduce the number of threads + * + * Encoder: Ignored. + * + * Decoder: + * + * If the number of threads has been set so high that more than + * memlimit_threading bytes of memory would be needed, the number + * of threads will be reduced so that the memory usage will not exceed + * memlimit_threading bytes. However, if memlimit_threading cannot + * be met even in single-threaded mode, then decoding will continue + * in single-threaded mode and memlimit_threading may be exceeded + * even by a large amount. That is, memlimit_threading will never make + * lzma_code() return LZMA_MEMLIMIT_ERROR. To truly cap the memory + * usage, see memlimit_stop below. + * + * Setting memlimit_threading to UINT64_MAX or a similar huge value + * means that liblzma is allowed to keep the whole compressed file + * and the whole uncompressed file in memory in addition to the memory + * needed by the decompressor data structures used by each thread! + * In other words, a reasonable value limit must be set here or it + * will cause problems sooner or later. If you have no idea what + * a reasonable value could be, try lzma_physmem() / 4 as a starting + * point. Setting this limit will never prevent decompression of + * a file; this will only reduce the number of threads. + * + * If memlimit_threading is greater than memlimit_stop, then the value + * of memlimit_stop will be used for both. + */ + uint64_t memlimit_threading; + + /** + * \brief Memory usage limit that should never be exceeded + * + * Encoder: Ignored. + * + * Decoder: If decompressing will need more than this amount of + * memory even in the single-threaded mode, then lzma_code() will + * return LZMA_MEMLIMIT_ERROR. + */ + uint64_t memlimit_stop; + uint64_t reserved_int7; uint64_t reserved_int8; void *reserved_ptr1; @@ -444,6 +491,60 @@ extern LZMA_API(lzma_ret) lzma_stream_buffer_encode( lzma_nothrow lzma_attr_warn_unused_result; +/** + * \brief MicroLZMA encoder + * + * The MicroLZMA format is a raw LZMA stream whose first byte (always 0x00) + * has been replaced with bitwise-negation of the LZMA properties (lc/lp/pb). + * This encoding ensures that the first byte of MicroLZMA stream is never + * 0x00. There is no end of payload marker and thus the uncompressed size + * must be stored separately. For the best error detection the dictionary + * size should be stored separately as well but alternatively one may use + * the uncompressed size as the dictionary size when decoding. + * + * With the MicroLZMA encoder, lzma_code() behaves slightly unusually. + * The action argument must be LZMA_FINISH and the return value will never be + * LZMA_OK. Thus the encoding is always done with a single lzma_code() after + * the initialization. The benefit of the combination of initialization + * function and lzma_code() is that memory allocations can be re-used for + * better performance. + * + * lzma_code() will try to encode as much input as is possible to fit into + * the given output buffer. If not all input can be encoded, the stream will + * be finished without encoding all the input. The caller must check both + * input and output buffer usage after lzma_code() (total_in and total_out + * in lzma_stream can be convenient). Often lzma_code() can fill the output + * buffer completely if there is a lot of input, but sometimes a few bytes + * may remain unused because the next LZMA symbol would require more space. + * + * lzma_stream.avail_out must be at least 6. Otherwise LZMA_PROG_ERROR + * will be returned. + * + * The LZMA dictionary should be reasonably low to speed up the encoder + * re-initialization. A good value is bigger than the resulting + * uncompressed size of most of the output chunks. For example, if output + * size is 4 KiB, dictionary size of 32 KiB or 64 KiB is good. If the + * data compresses extremely well, even 128 KiB may be useful. + * + * The MicroLZMA format and this encoder variant were made with the EROFS + * file system in mind. This format may be convenient in other embedded + * uses too where many small streams are needed. XZ Embedded includes a + * decoder for this format. + * + * \return - LZMA_STREAM_END: All good. Check the amounts of input used + * and output produced. Store the amount of input used + * (uncompressed size) as it needs to be known to decompress + * the data. + * - LZMA_OPTIONS_ERROR + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR: In addition to the generic reasons for this + * error code, this may also be returned if there isn't enough + * output space (6 bytes) to create a valid MicroLZMA stream. + */ +extern LZMA_API(lzma_ret) lzma_microlzma_encoder( + lzma_stream *strm, const lzma_options_lzma *options); + + /************ * Decoding * ************/ @@ -501,8 +602,8 @@ extern LZMA_API(lzma_ret) lzma_stream_buffer_encode( /** * This flag enables decoding of concatenated files with file formats that * allow concatenating compressed files as is. From the formats currently - * supported by liblzma, only the .xz format allows concatenated files. - * Concatenated files are not allowed with the legacy .lzma format. + * supported by liblzma, only the .xz and .lz formats allow concatenated + * files. Concatenated files are not allowed with the legacy .lzma format. * * This flag also affects the usage of the `action' argument for lzma_code(). * When LZMA_CONCATENATED is used, lzma_code() won't return LZMA_STREAM_END @@ -516,6 +617,35 @@ extern LZMA_API(lzma_ret) lzma_stream_buffer_encode( /** + * This flag makes the threaded decoder report errors (like LZMA_DATA_ERROR) + * as soon as they are detected. This saves time when the application has no + * interest in a partially decompressed truncated or corrupt file. Note that + * due to timing randomness, if the same truncated or corrupt input is + * decompressed multiple times with this flag, a different amount of output + * may be produced by different runs, and even the error code might vary. + * + * When using LZMA_FAIL_FAST, it is recommended to use LZMA_FINISH to tell + * the decoder when no more input will be coming because it can help fast + * detection and reporting of truncated files. Note that in this situation + * truncated files might be diagnosed with LZMA_DATA_ERROR instead of + * LZMA_OK or LZMA_BUF_ERROR! + * + * Without this flag the threaded decoder will provide as much output as + * possible at first and then report the pending error. This default behavior + * matches the single-threaded decoder and provides repeatable behavior + * with truncated or corrupt input. There are a few special cases where the + * behavior can still differ like memory allocation failures (LZMA_MEM_ERROR). + * + * Single-threaded decoders currently ignore this flag. + * + * Support for this flag was added in liblzma 5.3.3alpha. Note that in older + * versions this flag isn't supported (LZMA_OPTIONS_ERROR) even by functions + * that ignore this flag in newer liblzma versions. + */ +#define LZMA_FAIL_FAST UINT32_C(0x20) + + +/** * \brief Initialize .xz Stream decoder * * \param strm Pointer to properly prepared lzma_stream @@ -527,7 +657,7 @@ extern LZMA_API(lzma_ret) lzma_stream_buffer_encode( * \param flags Bitwise-or of zero or more of the decoder flags: * LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, * LZMA_TELL_ANY_CHECK, LZMA_IGNORE_CHECK, - * LZMA_CONCATENATED + * LZMA_CONCATENATED, LZMA_FAIL_FAST * * \return - LZMA_OK: Initialization was successful. * - LZMA_MEM_ERROR: Cannot allocate memory. @@ -540,11 +670,43 @@ extern LZMA_API(lzma_ret) lzma_stream_decoder( /** - * \brief Decode .xz Streams and .lzma files with autodetection + * \brief Initialize multithreaded .xz Stream decoder * - * This decoder autodetects between the .xz and .lzma file formats, and - * calls lzma_stream_decoder() or lzma_alone_decoder() once the type - * of the input file has been detected. + * \param strm Pointer to properly prepared lzma_stream + * \param options Pointer to multithreaded compression options + * + * The decoder can decode multiple Blocks in parallel. This requires that each + * Block Header contains the Compressed Size and Uncompressed size fields + * which are added by the multi-threaded encoder, see lzma_stream_encoder_mt(). + * + * A Stream with one Block will only utilize one thread. A Stream with multiple + * Blocks but without size information in Block Headers will be processed in + * single-threaded mode in the same way as done by lzma_stream_decoder(). + * Concatenated Streams are processed one Stream at a time; no inter-Stream + * parallelization is done. + * + * This function behaves like lzma_stream_decoder() when options->threads == 1 + * and options->memlimit_threading <= 1. + * + * \return - LZMA_OK: Initialization was successful. + * - LZMA_MEM_ERROR: Cannot allocate memory. + * - LZMA_MEMLIMIT_ERROR: Memory usage limit was reached. + * - LZMA_OPTIONS_ERROR: Unsupported flags. + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_stream_decoder_mt( + lzma_stream *strm, const lzma_mt *options) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Decode .xz, .lzma, and .lz (lzip) files with autodetection + * + * This decoder autodetects between the .xz, .lzma, and .lz file formats, + * and calls lzma_stream_decoder(), lzma_alone_decoder(), or + * lzma_lzip_decoder() once the type of the input file has been detected. + * + * Support for .lz was added in 5.4.0. * * If the flag LZMA_CONCATENATED is used and the input is a .lzma file: * For historical reasons concatenated .lzma files aren't supported. @@ -562,7 +724,7 @@ extern LZMA_API(lzma_ret) lzma_stream_decoder( * \param flags Bitwise-or of zero or more of the decoder flags: * LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, * LZMA_TELL_ANY_CHECK, LZMA_IGNORE_CHECK, - * LZMA_CONCATENATED + * LZMA_CONCATENATED, LZMA_FAIL_FAST * * \return - LZMA_OK: Initialization was successful. * - LZMA_MEM_ERROR: Cannot allocate memory. @@ -598,6 +760,64 @@ extern LZMA_API(lzma_ret) lzma_alone_decoder( /** + * \brief Initialize .lz (lzip) decoder (a foreign file format) + * + * \param strm Pointer to properly prepared lzma_stream + * \param memlimit Memory usage limit as bytes. Use UINT64_MAX + * to effectively disable the limiter. + * \param flags Bitwise-or of flags, or zero for no flags. + * All decoder flags listed above are supported + * although only LZMA_CONCATENATED and (in very rare + * cases) LZMA_IGNORE_CHECK are actually useful. + * LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, + * and LZMA_FAIL_FAST do nothing. LZMA_TELL_ANY_CHECK + * is supported for consistency only as CRC32 is + * always used in the .lz format. + * + * This decoder supports the .lz format version 0 and the unextended .lz + * format version 1: + * + * - Files in the format version 0 were produced by lzip 1.3 and older. + * Such files aren't common but may be found from file archives + * as a few source packages were released in this format. People + * might have old personal files in this format too. Decompression + * support for the format version 0 was removed in lzip 1.18. + * + * - lzip 1.3 added decompression support for .lz format version 1 files. + * Compression support was added in lzip 1.4. In lzip 1.6 the .lz format + * version 1 was extended to support the Sync Flush marker. This extension + * is not supported by liblzma. lzma_code() will return LZMA_DATA_ERROR + * at the location of the Sync Flush marker. In practice files with + * the Sync Flush marker are very rare and thus liblzma can decompress + * almost all .lz files. + * + * Just like with lzma_stream_decoder() for .xz files, LZMA_CONCATENATED + * should be used when decompressing normal standalone .lz files. + * + * The .lz format allows putting non-.lz data at the end of a file after at + * least one valid .lz member. That is, one can append custom data at the end + * of a .lz file and the decoder is required to ignore it. In liblzma this + * is relevant only when LZMA_CONCATENATED is used. In that case lzma_code() + * will return LZMA_STREAM_END and leave lzma_stream.next_in pointing to + * the first byte of the non-.lz data. An exception to this is if the first + * 1-3 bytes of the non-.lz data are identical to the .lz magic bytes + * (0x4C, 0x5A, 0x49, 0x50; "LZIP" in US-ASCII). In such a case the 1-3 bytes + * will have been ignored by lzma_code(). If one wishes to locate the non-.lz + * data reliably, one must ensure that the first byte isn't 0x4C. Actually + * one should ensure that none of the first four bytes of trailing data are + * equal to the magic bytes because lzip >= 1.20 requires it by default. + * + * \return - LZMA_OK: Initialization was successful. + * - LZMA_MEM_ERROR: Cannot allocate memory. + * - LZMA_OPTIONS_ERROR: Unsupported flags + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_lzip_decoder( + lzma_stream *strm, uint64_t memlimit, uint32_t flags) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** * \brief Single-call .xz Stream decoder * * \param memlimit Pointer to how much memory the decoder is allowed @@ -606,9 +826,9 @@ extern LZMA_API(lzma_ret) lzma_alone_decoder( * returned. * \param flags Bitwise-or of zero or more of the decoder flags: * LZMA_TELL_NO_CHECK, LZMA_TELL_UNSUPPORTED_CHECK, - * LZMA_IGNORE_CHECK, LZMA_CONCATENATED. Note that - * LZMA_TELL_ANY_CHECK is not allowed and will - * return LZMA_PROG_ERROR. + * LZMA_IGNORE_CHECK, LZMA_CONCATENATED, + * LZMA_FAIL_FAST. Note that LZMA_TELL_ANY_CHECK + * is not allowed and will return LZMA_PROG_ERROR. * \param allocator lzma_allocator for custom allocator functions. * Set to NULL to use malloc() and free(). * \param in Beginning of the input buffer @@ -642,3 +862,43 @@ extern LZMA_API(lzma_ret) lzma_stream_buffer_decode( const uint8_t *in, size_t *in_pos, size_t in_size, uint8_t *out, size_t *out_pos, size_t out_size) lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief MicroLZMA decoder + * + * See lzma_microlzma_decoder() for more information. + * + * The lzma_code() usage with this decoder is completely normal. The + * special behavior of lzma_code() applies to lzma_microlzma_encoder() only. + * + * \param strm Pointer to properly prepared lzma_stream + * \param comp_size Compressed size of the MicroLZMA stream. + * The caller must somehow know this exactly. + * \param uncomp_size Uncompressed size of the MicroLZMA stream. + * If the exact uncompressed size isn't known, this + * can be set to a value that is at most as big as + * the exact uncompressed size would be, but then the + * next argument uncomp_size_is_exact must be false. + * \param uncomp_size_is_exact + * If true, uncomp_size must be exactly correct. + * This will improve error detection at the end of + * the stream. If the exact uncompressed size isn't + * known, this must be false. uncomp_size must still + * be at most as big as the exact uncompressed size + * is. Setting this to false when the exact size is + * known will work but error detection at the end of + * the stream will be weaker. + * \param dict_size LZMA dictionary size that was used when + * compressing the data. It is OK to use a bigger + * value too but liblzma will then allocate more + * memory than would actually be required and error + * detection will be slightly worse. (Note that with + * the implementation in XZ Embedded it doesn't + * affect the memory usage if one specifies bigger + * dictionary than actually required.) + */ +extern LZMA_API(lzma_ret) lzma_microlzma_decoder( + lzma_stream *strm, uint64_t comp_size, + uint64_t uncomp_size, lzma_bool uncomp_size_is_exact, + uint32_t dict_size); diff --git a/src/liblzma/api/lzma/filter.h b/src/liblzma/api/lzma/filter.h index 5ec9976d4a68..58c83bf7edf4 100644 --- a/src/liblzma/api/lzma/filter.h +++ b/src/liblzma/api/lzma/filter.h @@ -125,6 +125,27 @@ extern LZMA_API(lzma_ret) lzma_filters_copy( /** + * \brief Free the options in the array of lzma_filter structures + * + * This frees the filter chain options. The filters array itself is not freed. + * + * The filters array must have at most LZMA_FILTERS_MAX + 1 elements + * including the terminating element which must have .id = LZMA_VLI_UNKNOWN. + * For all elements before the terminating element: + * - options will be freed using the given lzma_allocator or, + * if allocator is NULL, using free(). + * - options will be set to NULL. + * - id will be set to LZMA_VLI_UNKNOWN. + * + * If filters is NULL, this does nothing but remember that this never frees + * the filters array itself. + */ +extern LZMA_API(void) lzma_filters_free( + lzma_filter *filters, const lzma_allocator *allocator) + lzma_nothrow; + + +/** * \brief Calculate approximate memory requirements for raw encoder * * This function can be used to calculate the memory requirements for @@ -205,21 +226,27 @@ extern LZMA_API(lzma_ret) lzma_raw_decoder( /** * \brief Update the filter chain in the encoder * - * This function is for advanced users only. This function has two slightly - * different purposes: + * This function may be called after lzma_code() has returned LZMA_STREAM_END + * when LZMA_FULL_BARRIER, LZMA_FULL_FLUSH, or LZMA_SYNC_FLUSH was used: + * + * - After LZMA_FULL_BARRIER or LZMA_FULL_FLUSH: Single-threaded .xz Stream + * encoder (lzma_stream_encoder()) and (since liblzma 5.4.0) multi-threaded + * Stream encoder (lzma_stream_encoder_mt()) allow setting a new filter + * chain to be used for the next Block(s). * - * - After LZMA_FULL_FLUSH when using Stream encoder: Set a new filter - * chain, which will be used starting from the next Block. + * - After LZMA_SYNC_FLUSH: Raw encoder (lzma_raw_encoder()), + * Block encocder (lzma_block_encoder()), and single-threaded .xz Stream + * encoder (lzma_stream_encoder()) allow changing certain filter-specific + * options in the middle of encoding. The actual filters in the chain + * (Filter IDs) must not be changed! Currently only the lc, lp, and pb + * options of LZMA2 (not LZMA1) can be changed this way. * - * - After LZMA_SYNC_FLUSH using Raw, Block, or Stream encoder: Change - * the filter-specific options in the middle of encoding. The actual - * filters in the chain (Filter IDs) cannot be changed. In the future, - * it might become possible to change the filter options without - * using LZMA_SYNC_FLUSH. + * - In the future some filters might allow changing some of their options + * without any barrier or flushing but currently such filters don't exist. * - * While rarely useful, this function may be called also when no data has - * been compressed yet. In that case, this function will behave as if - * LZMA_FULL_FLUSH (Stream encoder) or LZMA_SYNC_FLUSH (Raw or Block + * This function may also be called when no data has been compressed yet + * although this is rarely useful. In that case, this function will behave + * as if LZMA_FULL_FLUSH (Stream encoders) or LZMA_SYNC_FLUSH (Raw or Block * encoder) had been used right before calling this function. * * \return - LZMA_OK @@ -427,3 +454,261 @@ extern LZMA_API(lzma_ret) lzma_filter_flags_decode( lzma_filter *filter, const lzma_allocator *allocator, const uint8_t *in, size_t *in_pos, size_t in_size) lzma_nothrow lzma_attr_warn_unused_result; + + +/*********** + * Strings * + ***********/ + +/** + * \brief Allow or show all filters + * + * By default only the filters supported in the .xz format are accept by + * lzma_str_to_filters() or shown by lzma_str_list_filters(). + */ +#define LZMA_STR_ALL_FILTERS UINT32_C(0x01) + + +/** + * \brief Do not validate the filter chain in lzma_str_to_filters() + * + * By default lzma_str_to_filters() can return an error if the filter chain + * as a whole isn't usable in the .xz format or in the raw encoder or decoder. + * With this flag the validation is skipped (this doesn't affect the handling + * of the individual filter options). + */ +#define LZMA_STR_NO_VALIDATION UINT32_C(0x02) + + +/** + * \brief Stringify encoder options + * + * Show the filter-specific options that the encoder will use. + * This may be useful for verbose diagnostic messages. + * + * Note that if options were decoded from .xz headers then the encoder options + * may be undefined. This flag shouldn't be used in such a situation. + */ +#define LZMA_STR_ENCODER UINT32_C(0x10) + + +/** + * \brief Stringify decoder options + * + * Show the filter-specific options that the decoder will use. + * This may be useful for showing what filter options were decoded + * from file headers. + */ +#define LZMA_STR_DECODER UINT32_C(0x20) + + +/** + * \brief Produce xz-compatible getopt_long() syntax + * + * That is, "delta:dist=2 lzma2:dict=4MiB,pb=1,lp=1" becomes + * "--delta=dist=2 --lzma2=dict=4MiB,pb=1,lp=1". + * + * This syntax is compatible with xz 5.0.0 as long as the filters and + * their options are supported too. + */ +#define LZMA_STR_GETOPT_LONG UINT32_C(0x40) + + +/** + * \brief Use two dashes "--" instead of a space to separate filters + * + * That is, "delta:dist=2 lzma2:pb=1,lp=1" becomes + * "delta:dist=2--lzma2:pb=1,lp=1". This looks slightly odd but this + * kind of strings should be usable on the command line without quoting. + * However, it is possible that future versions with new filter options + * might produce strings that require shell quoting anyway as the exact + * set of possible characters isn't frozen for now. + * + * It is guaranteed that the single quote (') will never be used in + * filter chain strings (even if LZMA_STR_NO_SPACES isn't used). + */ +#define LZMA_STR_NO_SPACES UINT32_C(0x80) + + +/** + * \brief Convert a string to a filter chain + * + * This tries to make it easier to write applications that allow users + * to set custom compression options. This only handles the filter + * configuration (including presets) but not the number of threads, + * block size, check type, or memory limits. + * + * The input string can be either a preset or a filter chain. Presets + * begin with a digit 0-9 and may be followed by zero or more flags + * which are lower-case letters. Currently only "e" is supported, matching + * LZMA_PRESET_EXTREME. For partial xz command line syntax compatibility, + * a preset string may start with a single dash "-". + * + * A filter chain consists of one or more "filtername:opt1=value1,opt2=value2" + * strings separated by one or more spaces. Leading and trailing spaces are + * ignored. All names and values must be lower-case. Extra commas in the + * option list are ignored. The order of filters is significant: when + * encoding, the uncompressed input data goes to the leftmost filter first. + * Normally "lzma2" is the last filter in the chain. + * + * If one wishes to avoid spaces, for example, to avoid shell quoting, + * it is possible to use two dashes "--" instead of spaces to separate + * the filters. + * + * For xz command line compatibility, each filter may be prefixed with + * two dashes "--" and the colon ":" separating the filter name from + * the options may be replaced with an equals sign "=". + * + * By default, only filters that can be used in the .xz format are accepted. + * To allow all filters (LZMA1) use the flag LZMA_STR_ALL_FILTERS. + * + * By default, very basic validation is done for the filter chain as a whole, + * for example, that LZMA2 is only used as the last filter in the chain. + * The validation isn't perfect though and it's possible that this function + * succeeds but using the filter chain for encoding or decoding will still + * result in LZMA_OPTIONS_ERROR. To disable this validation, use the flag + * LZMA_STR_NO_VALIDATION. + * + * The available filter names and their options are available via + * lzma_str_list_filters(). See the xz man page for the description + * of filter names and options. + * + * \param str User-supplied string describing a preset or + * a filter chain. If a default value is needed and + * you don't know what would be good, use "6" since + * that is the default preset in xz too. + * \param error_pos If this isn't NULL, this value will be set on + * both success and on all errors. This tells the + * location of the error in the string. This is + * an int to make it straightforward to use this + * as printf() field width. The value is guaranteed + * to be in the range [0, INT_MAX] even if strlen(str) + * somehow was greater than INT_MAX. + * \param filters An array of lzma_filter structures. There must + * be LZMA_FILTERS_MAX + 1 (that is, five) elements + * in the array. The old contents are ignored so it + * doesn't need to be initialized. This array is + * modified only if this function returns LZMA_OK. + * Once the allocated filter options are no longer + * needed, lzma_filters_free() can be used to free the + * options (it doesn't free the filters array itself). + * \param flags Bitwise-or of zero or more of the flags + * LZMA_STR_ALL_FILTERS and LZMA_STR_NO_VALIDATION. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return On success, NULL is returned. On error, a statically-allocated + * error message is returned which together with the error_pos + * should give some idea what is wrong. + * + * For command line applications, below is an example how an error message + * can be displayed. Note the use of an empty string for the field width. + * If "^" was used there it would create an off-by-one error except at + * the very beginning of the line. + * + * \code{.c} + * const char *str = ...; // From user + * lzma_filter filters[LZMA_FILTERS_MAX + 1]; + * int pos; + * const char *msg = lzma_str_to_filters(str, &pos, filters, 0, NULL); + * if (msg != NULL) { + * printf("%s: Error in XZ compression options:\n", argv[0]); + * printf("%s: %s\n", argv[0], str); + * printf("%s: %*s^\n", argv[0], errpos, ""); + * printf("%s: %s\n", argv[0], msg); + * } + * \endcode + */ +extern LZMA_API(const char *) lzma_str_to_filters( + const char *str, int *error_pos, lzma_filter *filters, + uint32_t flags, const lzma_allocator *allocator) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief Convert a filter chain to a string + * + * Use cases: + * + * - Verbose output showing the full encoder options to the user + * (use LZMA_STR_ENCODER in flags) + * + * - Showing the filters and options that are required to decode a file + * (use LZMA_STR_DECODER in flags) + * + * - Showing the filter names without any options in informational messages + * where the technical details aren't important (no flags). In this case + * the .options in the filters array are ignored and may be NULL even if + * a filter has a mandatory options structure. + * + * Note that even if the filter chain was specified using a preset, + * the resulting filter chain isn't reversed to a preset. So if you + * specify "6" to lzma_str_to_filters() then lzma_str_from_filters() + * will produce a string containing "lzma2". + * + * \param str On success *str will be set to point to an + * allocated string describing the given filter + * chain. Old value is ignored. On error *str is + * always set to NULL. + * \param filters Array of 1-4 filters and a terminating element + * with .id = LZMA_VLI_UNKNOWN. + * \param flags Bitwise-or of zero or more of the flags + * LZMA_STR_ENCODER, LZMA_STR_DECODER, + * LZMA_STR_GETOPT_LONG, and LZMA_STR_NO_SPACES. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return - LZMA_OK + * - LZMA_OPTIONS_ERROR: Empty filter chain + * (filters[0].id == LZMA_VLI_UNKNOWN) or the filter chain + * includes a Filter ID that is not supported by this function. + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_str_from_filters( + char **str, const lzma_filter *filters, uint32_t flags, + const lzma_allocator *allocator) + lzma_nothrow lzma_attr_warn_unused_result; + + +/** + * \brief List available filters and/or their options (for help message) + * + * If a filter_id is given then only one line is created which contains the + * filter name. If LZMA_STR_ENCODER or LZMA_STR_DECODER is used then the + * options required for encoding or decoding are listed on the same line too. + * + * If filter_id is LZMA_VLI_UNKNOWN then all supported .xz-compatible filters + * are listed: + * + * - If neither LZMA_STR_ENCODER nor LZMA_STR_DECODER is used then + * the supported filter names are listed on a single line separated + * by spaces. + * + * - If LZMA_STR_ENCODER or LZMA_STR_DECODER is used then filters and + * the supported options are listed one filter per line. There won't + * be a '\n' after the last filter. + * + * - If LZMA_STR_ALL_FILTERS is used then the list will include also + * those filters that cannot be used in the .xz format (LZMA1). + * + * \param str On success *str will be set to point to an + * allocated string listing the filters and options. + * Old value is ignored. On error *str is always set + * to NULL. + * \param filter_id Filter ID or LZMA_VLI_UNKNOWN. + * \param flags Bitwise-or of zero or more of the flags + * LZMA_STR_ALL_FILTERS, LZMA_STR_ENCODER, + * LZMA_STR_DECODER, and LZMA_STR_GETOPT_LONG. + * \param allocator lzma_allocator for custom allocator functions. + * Set to NULL to use malloc() and free(). + * + * \return - LZMA_OK + * - LZMA_OPTIONS_ERROR: Unsupported filter_id or flags + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_str_list_filters( + char **str, lzma_vli filter_id, uint32_t flags, + const lzma_allocator *allocator) + lzma_nothrow lzma_attr_warn_unused_result; diff --git a/src/liblzma/api/lzma/index.h b/src/liblzma/api/lzma/index.h index 3dac6fb85ccb..144d416615a4 100644 --- a/src/liblzma/api/lzma/index.h +++ b/src/liblzma/api/lzma/index.h @@ -684,3 +684,69 @@ extern LZMA_API(lzma_ret) lzma_index_buffer_decode(lzma_index **i, uint64_t *memlimit, const lzma_allocator *allocator, const uint8_t *in, size_t *in_pos, size_t in_size) lzma_nothrow; + + +/** + * \brief Initialize a .xz file information decoder + * + * \param strm Pointer to a properly prepared lzma_stream + * \param dest_index Pointer to a pointer where the decoder will put + * the decoded lzma_index. The old value + * of *dest_index is ignored (not freed). + * \param memlimit How much memory the resulting lzma_index is + * allowed to require. Use UINT64_MAX to + * effectively disable the limiter. + * \param file_size Size of the input .xz file + * + * This decoder decodes the Stream Header, Stream Footer, Index, and + * Stream Padding field(s) from the input .xz file and stores the resulting + * combined index in *dest_index. This information can be used to get the + * uncompressed file size with lzma_index_uncompressed_size(*dest_index) or, + * for example, to implement random access reading by locating the Blocks + * in the Streams. + * + * To get the required information from the .xz file, lzma_code() may ask + * the application to seek in the input file by returning LZMA_SEEK_NEEDED + * and having the target file position specified in lzma_stream.seek_pos. + * The number of seeks required depends on the input file and how big buffers + * the application provides. When possible, the decoder will seek backward + * and forward in the given buffer to avoid useless seek requests. Thus, if + * the application provides the whole file at once, no external seeking will + * be required (that is, lzma_code() won't return LZMA_SEEK_NEEDED). + * + * The value in lzma_stream.total_in can be used to estimate how much data + * liblzma had to read to get the file information. However, due to seeking + * and the way total_in is updated, the value of total_in will be somewhat + * inaccurate (a little too big). Thus, total_in is a good estimate but don't + * expect to see the same exact value for the same file if you change the + * input buffer size or switch to a different liblzma version. + * + * Valid `action' arguments to lzma_code() are LZMA_RUN and LZMA_FINISH. + * You only need to use LZMA_RUN; LZMA_FINISH is only supported because it + * might be convenient for some applications. If you use LZMA_FINISH and if + * lzma_code() asks the application to seek, remember to reset `action' back + * to LZMA_RUN unless you hit the end of the file again. + * + * Possible return values from lzma_code(): + * - LZMA_OK: All OK so far, more input needed + * - LZMA_SEEK_NEEDED: Provide more input starting from the absolute + * file position strm->seek_pos + * - LZMA_STREAM_END: Decoding was successful, *dest_index has been set + * - LZMA_FORMAT_ERROR: The input file is not in the .xz format (the + * expected magic bytes were not found from the beginning of the file) + * - LZMA_OPTIONS_ERROR: File looks valid but contains headers that aren't + * supported by this version of liblzma + * - LZMA_DATA_ERROR: File is corrupt + * - LZMA_BUF_ERROR + * - LZMA_MEM_ERROR + * - LZMA_MEMLIMIT_ERROR + * - LZMA_PROG_ERROR + * + * \return - LZMA_OK + * - LZMA_MEM_ERROR + * - LZMA_PROG_ERROR + */ +extern LZMA_API(lzma_ret) lzma_file_info_decoder( + lzma_stream *strm, lzma_index **dest_index, + uint64_t memlimit, uint64_t file_size) + lzma_nothrow; diff --git a/src/liblzma/api/lzma/lzma12.h b/src/liblzma/api/lzma/lzma12.h index df5f23b61a43..d34e78392b51 100644 --- a/src/liblzma/api/lzma/lzma12.h +++ b/src/liblzma/api/lzma/lzma12.h @@ -18,18 +18,41 @@ /** - * \brief LZMA1 Filter ID + * \brief LZMA1 Filter ID (for raw encoder/decoder only, not in .xz) * * LZMA1 is the very same thing as what was called just LZMA in LZMA Utils, * 7-Zip, and LZMA SDK. It's called LZMA1 here to prevent developers from * accidentally using LZMA when they actually want LZMA2. - * - * LZMA1 shouldn't be used for new applications unless you _really_ know - * what you are doing. LZMA2 is almost always a better choice. */ #define LZMA_FILTER_LZMA1 LZMA_VLI_C(0x4000000000000001) /** + * \brief LZMA1 Filter ID with extended options (for raw encoder/decoder) + * + * This is like LZMA_FILTER_LZMA1 but with this ID a few extra options + * are supported in the lzma_options_lzma structure: + * + * - A flag to tell the encoder if the end of payload marker (EOPM) alias + * end of stream (EOS) marker must be written at the end of the stream. + * In contrast, LZMA_FILTER_LZMA1 always writes the end marker. + * + * - Decoder needs to be told the uncompressed size of the stream + * or that it is unknown (using the special value UINT64_MAX). + * If the size is known, a flag can be set to allow the presence of + * the end marker anyway. In contrast, LZMA_FILTER_LZMA1 always + * behaves as if the uncompressed size was unknown. + * + * This allows handling file formats where LZMA1 streams are used but where + * the end marker isn't allowed or where it might not (always) be present. + * This extended LZMA1 functionality is provided as a Filter ID for raw + * encoder and decoder instead of adding new encoder and decoder initialization + * functions because this way it is possible to also use extra filters, + * for example, LZMA_FILTER_X86 in a filter chain with LZMA_FILTER_LZMA1EXT, + * which might be needed to handle some file formats. + */ +#define LZMA_FILTER_LZMA1EXT LZMA_VLI_C(0x4000000000000002) + +/** * \brief LZMA2 Filter ID * * Usually you want this instead of LZMA1. Compared to LZMA1, LZMA2 adds @@ -374,6 +397,82 @@ typedef struct { */ uint32_t depth; + /** + * \brief For LZMA_FILTER_LZMA1EXT: Extended flags + * + * This is used only with LZMA_FILTER_LZMA1EXT. + * + * Currently only one flag is supported, LZMA_LZMA1EXT_ALLOW_EOPM: + * + * - Encoder: If the flag is set, then end marker is written just + * like it is with LZMA_FILTER_LZMA1. Without this flag the + * end marker isn't written and the application has to store + * the uncompressed size somewhere outside the compressed stream. + * To decompress streams without the end marker, the appliation + * has to set the correct uncompressed size in ext_size_low and + * ext_size_high. + * + * - Decoder: If the uncompressed size in ext_size_low and + * ext_size_high is set to the special value UINT64_MAX + * (indicating unknown uncompressed size) then this flag is + * ignored and the end marker must always be present, that is, + * the behavior is identical to LZMA_FILTER_LZMA1. + * + * Otherwise, if this flag isn't set, then the input stream + * must not have the end marker; if the end marker is detected + * then it will result in LZMA_DATA_ERROR. This is useful when + * it is known that the stream must not have the end marker and + * strict validation is wanted. + * + * If this flag is set, then it is autodetected if the end marker + * is present after the specified number of uncompressed bytes + * has been decompressed (ext_size_low and ext_size_high). The + * end marker isn't allowed in any other position. This behavior + * is useful when uncompressed size is known but the end marker + * may or may not be present. This is the case, for example, + * in .7z files (valid .7z files that have the end marker in + * LZMA1 streams are rare but they do exist). + */ + uint32_t ext_flags; +# define LZMA_LZMA1EXT_ALLOW_EOPM UINT32_C(0x01) + + /** + * \brief For LZMA_FILTER_LZMA1EXT: Uncompressed size (low bits) + * + * The 64-bit uncompressed size is needed for decompression with + * LZMA_FILTER_LZMA1EXT. The size is ignored by the encoder. + * + * The special value UINT64_MAX indicates that the uncompressed size + * is unknown and that the end of payload marker (also known as + * end of stream marker) must be present to indicate the end of + * the LZMA1 stream. Any other value indicates the expected + * uncompressed size of the LZMA1 stream. (If LZMA1 was used together + * with filters that change the size of the data then the uncompressed + * size of the LZMA1 stream could be different than the final + * uncompressed size of the filtered stream.) + * + * ext_size_low holds the least significant 32 bits of the + * uncompressed size. The most significant 32 bits must be set + * in ext_size_high. The macro lzma_ext_size_set(opt_lzma, u64size) + * can be used to set these members. + * + * The 64-bit uncompressed size is split into two uint32_t variables + * because there were no reserved uint64_t members and using the + * same options structure for LZMA_FILTER_LZMA1, LZMA_FILTER_LZMA1EXT, + * and LZMA_FILTER_LZMA2 was otherwise more convenient than having + * a new options structure for LZMA_FILTER_LZMA1EXT. (Replacing two + * uint32_t members with one uint64_t changes the ABI on some systems + * as the alignment of this struct can increase from 4 bytes to 8.) + */ + uint32_t ext_size_low; + + /** + * \brief For LZMA_FILTER_LZMA1EXT: Uncompressed size (high bits) + * + * This holds the most significant 32 bits of the uncompressed size. + */ + uint32_t ext_size_high; + /* * Reserved space to allow possible future extensions without * breaking the ABI. You should not touch these, because the names @@ -381,9 +480,6 @@ typedef struct { * with the currently supported options, so it is safe to leave these * uninitialized. */ - uint32_t reserved_int1; - uint32_t reserved_int2; - uint32_t reserved_int3; uint32_t reserved_int4; uint32_t reserved_int5; uint32_t reserved_int6; @@ -400,6 +496,19 @@ typedef struct { /** + * \brief Macro to set the 64-bit uncompressed size in ext_size_* + * + * This might be convenient when decoding using LZMA_FILTER_LZMA1EXT. + * This isn't used with LZMA_FILTER_LZMA1 or LZMA_FILTER_LZMA2. + */ +#define lzma_set_ext_size(opt_lzma2, u64size) \ +do { \ + (opt_lzma2).ext_size_low = (uint32_t)(u64size); \ + (opt_lzma2).ext_size_high = (uint32_t)((uint64_t)(u64size) >> 32); \ +} while (0) + + +/** * \brief Set a compression preset to lzma_options_lzma structure * * 0 is the fastest and 9 is the slowest. These match the switches -0 .. -9 diff --git a/src/liblzma/api/lzma/version.h b/src/liblzma/api/lzma/version.h index 087bf134557c..7d37130f240f 100644 --- a/src/liblzma/api/lzma/version.h +++ b/src/liblzma/api/lzma/version.h @@ -21,8 +21,8 @@ * Version number split into components */ #define LZMA_VERSION_MAJOR 5 -#define LZMA_VERSION_MINOR 2 -#define LZMA_VERSION_PATCH 9 +#define LZMA_VERSION_MINOR 4 +#define LZMA_VERSION_PATCH 0 #define LZMA_VERSION_STABILITY LZMA_VERSION_STABILITY_STABLE #ifndef LZMA_VERSION_COMMIT diff --git a/src/liblzma/check/crc32_small.c b/src/liblzma/check/crc32_small.c index 5f8a32868782..186966e99216 100644 --- a/src/liblzma/check/crc32_small.c +++ b/src/liblzma/check/crc32_small.c @@ -16,6 +16,9 @@ uint32_t lzma_crc32_table[1][256]; +#ifdef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +__attribute__((__constructor__)) +#endif static void crc32_init(void) { @@ -37,18 +40,22 @@ crc32_init(void) } +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR extern void lzma_crc32_init(void) { mythread_once(crc32_init); return; } +#endif extern LZMA_API(uint32_t) lzma_crc32(const uint8_t *buf, size_t size, uint32_t crc) { +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR lzma_crc32_init(); +#endif crc = ~crc; diff --git a/src/liblzma/check/crc64_fast.c b/src/liblzma/check/crc64_fast.c index 8af54cda7b5c..e3cbf1b1e91d 100644 --- a/src/liblzma/check/crc64_fast.c +++ b/src/liblzma/check/crc64_fast.c @@ -3,11 +3,25 @@ /// \file crc64.c /// \brief CRC64 calculation /// -/// Calculate the CRC64 using the slice-by-four algorithm. This is the same -/// idea that is used in crc32_fast.c, but for CRC64 we use only four tables +/// There are two methods in this file. crc64_generic uses the +/// the slice-by-four algorithm. This is the same idea that is +/// used in crc32_fast.c, but for CRC64 we use only four tables /// instead of eight to avoid increasing CPU cache usage. +/// +/// crc64_clmul uses 32/64-bit x86 SSSE3, SSE4.1, and CLMUL instructions. +/// It was derived from +/// https://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf +/// and the public domain code from https://github.com/rawrunprotected/crc +/// (URLs were checked on 2022-11-07). +/// +/// FIXME: Builds for 32-bit x86 use crc64_x86.S by default instead +/// of this file and thus CLMUL version isn't available on 32-bit x86 +/// unless configured with --disable-assembler. Even then the lookup table +/// isn't omitted in crc64_table.c since it doesn't know that assembly +/// code has been disabled. // -// Author: Lasse Collin +// Authors: Lasse Collin +// Ilya Kurdyukov // // This file has been put into the public domain. // You can do whatever you want with this file. @@ -15,6 +29,54 @@ /////////////////////////////////////////////////////////////////////////////// #include "check.h" + +#undef CRC_GENERIC +#undef CRC_CLMUL +#undef CRC_USE_GENERIC_FOR_SMALL_INPUTS + +// If CLMUL cannot be used then only the generic slice-by-four is built. +#if !defined(HAVE_USABLE_CLMUL) +# define CRC_GENERIC 1 + +// If CLMUL is allowed unconditionally in the compiler options then the +// generic version can be omitted. Note that this doesn't work with MSVC +// as I don't know how to detect the features here. +// +// NOTE: Keep this this in sync with crc64_table.c. +#elif (defined(__SSSE3__) && defined(__SSE4_1__) && defined(__PCLMUL__)) \ + || (defined(__e2k__) && __iset__ >= 6) +# define CRC_CLMUL 1 + +// Otherwise build both and detect at runtime which version to use. +#else +# define CRC_GENERIC 1 +# define CRC_CLMUL 1 + +/* + // The generic code is much faster with 1-8-byte inputs and has + // similar performance up to 16 bytes at least in microbenchmarks + // (it depends on input buffer alignment too). If both versions are + // built, this #define will use the generic version for inputs up to + // 16 bytes and CLMUL for bigger inputs. It saves a little in code + // size since the special cases for 0-16-byte inputs will be omitted + // from the CLMUL code. +# define CRC_USE_GENERIC_FOR_SMALL_INPUTS 1 +*/ + +# if defined(_MSC_VER) +# include <intrin.h> +# elif defined(HAVE_CPUID_H) +# include <cpuid.h> +# endif +#endif + + +///////////////////////////////// +// Generic slice-by-four CRC64 // +///////////////////////////////// + +#ifdef CRC_GENERIC + #include "crc_macros.h" @@ -26,8 +88,8 @@ // See the comments in crc32_fast.c. They aren't duplicated here. -extern LZMA_API(uint64_t) -lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc) +static uint64_t +crc64_generic(const uint8_t *buf, size_t size, uint64_t crc) { crc = ~crc; @@ -46,10 +108,11 @@ lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc) while (buf < limit) { #ifdef WORDS_BIGENDIAN - const uint32_t tmp = (crc >> 32) + const uint32_t tmp = (uint32_t)(crc >> 32) ^ aligned_read32ne(buf); #else - const uint32_t tmp = crc ^ aligned_read32ne(buf); + const uint32_t tmp = (uint32_t)crc + ^ aligned_read32ne(buf); #endif buf += 4; @@ -70,3 +133,380 @@ lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc) return ~crc; } +#endif + + +///////////////////// +// x86 CLMUL CRC64 // +///////////////////// + +#ifdef CRC_CLMUL + +#include <immintrin.h> + + +/* +// These functions were used to generate the constants +// at the top of crc64_clmul(). +static uint64_t +calc_lo(uint64_t poly) +{ + uint64_t a = poly; + uint64_t b = 0; + + for (unsigned i = 0; i < 64; ++i) { + b = (b >> 1) | (a << 63); + a = (a >> 1) ^ (a & 1 ? poly : 0); + } + + return b; +} + +static uint64_t +calc_hi(uint64_t poly, uint64_t a) +{ + for (unsigned i = 0; i < 64; ++i) + a = (a >> 1) ^ (a & 1 ? poly : 0); + + return a; +} +*/ + + +#define MASK_L(in, mask, r) \ + r = _mm_shuffle_epi8(in, mask) + +#define MASK_H(in, mask, r) \ + r = _mm_shuffle_epi8(in, _mm_xor_si128(mask, vsign)) + +#define MASK_LH(in, mask, low, high) \ + MASK_L(in, mask, low); \ + MASK_H(in, mask, high) + + +// EDG-based compilers (Intel's classic compiler and compiler for E2K) can +// define __GNUC__ but the attribute must not be used with them. +// The new Clang-based ICX needs the attribute. +// +// NOTE: Build systems check for this too, keep them in sync with this. +#if (defined(__GNUC__) || defined(__clang__)) && !defined(__EDG__) +__attribute__((__target__("ssse3,sse4.1,pclmul"))) +#endif +static uint64_t +crc64_clmul(const uint8_t *buf, size_t size, uint64_t crc) +{ + // The prototypes of the intrinsics use signed types while most of + // the values are treated as unsigned here. These warnings in this + // function have been checked and found to be harmless so silence them. +#if TUKLIB_GNUC_REQ(4, 6) || defined(__clang__) +# pragma GCC diagnostic push +# pragma GCC diagnostic ignored "-Wsign-conversion" +# pragma GCC diagnostic ignored "-Wconversion" +#endif + +#ifndef CRC_USE_GENERIC_FOR_SMALL_INPUTS + // The code assumes that there is at least one byte of input. + if (size == 0) + return crc; +#endif + + // const uint64_t poly = 0xc96c5795d7870f42; // CRC polynomial + const uint64_t p = 0x92d8af2baf0e1e85; // (poly << 1) | 1 + const uint64_t mu = 0x9c3e466c172963d5; // (calc_lo(poly) << 1) | 1 + const uint64_t k2 = 0xdabe95afc7875f40; // calc_hi(poly, 1) + const uint64_t k1 = 0xe05dd497ca393ae4; // calc_hi(poly, k2) + const __m128i vfold0 = _mm_set_epi64x(p, mu); + const __m128i vfold1 = _mm_set_epi64x(k2, k1); + + // Create a vector with 8-bit values 0 to 15. This is used to + // construct control masks for _mm_blendv_epi8 and _mm_shuffle_epi8. + const __m128i vramp = _mm_setr_epi32( + 0x03020100, 0x07060504, 0x0b0a0908, 0x0f0e0d0c); + + // This is used to inverse the control mask of _mm_shuffle_epi8 + // so that bytes that wouldn't be picked with the original mask + // will be picked and vice versa. + const __m128i vsign = _mm_set1_epi8(0x80); + + // Memory addresses A to D and the distances between them: + // + // A B C D + // [skip_start][size][skip_end] + // [ size2 ] + // + // A and D are 16-byte aligned. B and C are 1-byte aligned. + // skip_start and skip_end are 0-15 bytes. size is at least 1 byte. + // + // A = aligned_buf will initially point to this address. + // B = The address pointed by the caller-supplied buf. + // C = buf + size == aligned_buf + size2 + // D = buf + size + skip_end == aligned_buf + size2 + skip_end + const size_t skip_start = (size_t)((uintptr_t)buf & 15); + const size_t skip_end = (size_t)(-(uintptr_t)(buf + size) & 15); + const __m128i *aligned_buf = (const __m128i *)( + (uintptr_t)buf & ~(uintptr_t)15); + + // If size2 <= 16 then the whole input fits into a single 16-byte + // vector. If size2 > 16 then at least two 16-byte vectors must + // be processed. If size2 > 16 && size <= 16 then there is only + // one 16-byte vector's worth of input but it is unaligned in memory. + // + // NOTE: There is no integer overflow here if the arguments are valid. + // If this overflowed, buf + size would too. + size_t size2 = skip_start + size; + + // Masks to be used with _mm_blendv_epi8 and _mm_shuffle_epi8: + // The first skip_start or skip_end bytes in the vectors will have + // the high bit (0x80) set. _mm_blendv_epi8 and _mm_shuffle_epi8 + // will produce zeros for these positions. (Bitwise-xor of these + // masks with vsign will produce the opposite behavior.) + const __m128i mask_start + = _mm_sub_epi8(vramp, _mm_set1_epi8(skip_start)); + const __m128i mask_end = _mm_sub_epi8(vramp, _mm_set1_epi8(skip_end)); + + // Get the first 1-16 bytes into data0. If loading less than 16 bytes, + // the bytes are loaded to the high bits of the vector and the least + // significant positions are filled with zeros. + const __m128i data0 = _mm_blendv_epi8(_mm_load_si128(aligned_buf), + _mm_setzero_si128(), mask_start); + ++aligned_buf; + +#if defined(__i386__) || defined(_M_IX86) + const __m128i initial_crc = _mm_set_epi64x(0, ~crc); +#else + // GCC and Clang would produce good code with _mm_set_epi64x + // but MSVC needs _mm_cvtsi64_si128 on x86-64. + const __m128i initial_crc = _mm_cvtsi64_si128(~crc); +#endif + + __m128i v0, v1, v2, v3; + +#ifndef CRC_USE_GENERIC_FOR_SMALL_INPUTS + if (size <= 16) { + // Right-shift initial_crc by 1-16 bytes based on "size" + // and store the result in v1 (high bytes) and v0 (low bytes). + // + // NOTE: The highest 8 bytes of initial_crc are zeros so + // v1 will be filled with zeros if size >= 8. The highest 8 + // bytes of v1 will always become zeros. + // + // [ v1 ][ v0 ] + // [ initial_crc ] size == 1 + // [ initial_crc ] size == 2 + // [ initial_crc ] size == 15 + // [ initial_crc ] size == 16 (all in v0) + const __m128i mask_low = _mm_add_epi8( + vramp, _mm_set1_epi8(size - 16)); + MASK_LH(initial_crc, mask_low, v0, v1); + + if (size2 <= 16) { + // There are 1-16 bytes of input and it is all + // in data0. Copy the input bytes to v3. If there + // are fewer than 16 bytes, the low bytes in v3 + // will be filled with zeros. That is, the input + // bytes are stored to the same position as + // (part of) initial_crc is in v0. + MASK_L(data0, mask_end, v3); + } else { + // There are 2-16 bytes of input but not all bytes + // are in data0. + const __m128i data1 = _mm_load_si128(aligned_buf); + + // Collect the 2-16 input bytes from data0 and data1 + // to v2 and v3, and bitwise-xor them with the + // low bits of initial_crc in v0. Note that the + // the second xor is below this else-block as it + // is shared with the other branch. + MASK_H(data0, mask_end, v2); + MASK_L(data1, mask_end, v3); + v0 = _mm_xor_si128(v0, v2); + } + + v0 = _mm_xor_si128(v0, v3); + v1 = _mm_alignr_epi8(v1, v0, 8); + } else +#endif + { + const __m128i data1 = _mm_load_si128(aligned_buf); + MASK_LH(initial_crc, mask_start, v0, v1); + v0 = _mm_xor_si128(v0, data0); + v1 = _mm_xor_si128(v1, data1); + +#define FOLD \ + v1 = _mm_xor_si128(v1, _mm_clmulepi64_si128(v0, vfold1, 0x00)); \ + v0 = _mm_xor_si128(v1, _mm_clmulepi64_si128(v0, vfold1, 0x11)); + + while (size2 > 32) { + ++aligned_buf; + size2 -= 16; + FOLD + v1 = _mm_load_si128(aligned_buf); + } + + if (size2 < 32) { + MASK_H(v0, mask_end, v2); + MASK_L(v0, mask_end, v0); + MASK_L(v1, mask_end, v3); + v1 = _mm_or_si128(v2, v3); + } + + FOLD + v1 = _mm_srli_si128(v0, 8); +#undef FOLD + } + + v1 = _mm_xor_si128(_mm_clmulepi64_si128(v0, vfold1, 0x10), v1); + v0 = _mm_clmulepi64_si128(v1, vfold0, 0x00); + v2 = _mm_clmulepi64_si128(v0, vfold0, 0x10); + v0 = _mm_xor_si128(_mm_xor_si128(v2, _mm_slli_si128(v0, 8)), v1); + +#if defined(__i386__) || defined(_M_IX86) + return ~(((uint64_t)(uint32_t)_mm_extract_epi32(v0, 3) << 32) | + (uint64_t)(uint32_t)_mm_extract_epi32(v0, 2)); +#else + return ~(uint64_t)_mm_extract_epi64(v0, 1); +#endif + +#if TUKLIB_GNUC_REQ(4, 6) || defined(__clang__) +# pragma GCC diagnostic pop +#endif +} +#endif + + +//////////////////////// +// Detect CPU support // +//////////////////////// + +#if defined(CRC_GENERIC) && defined(CRC_CLMUL) +static inline bool +is_clmul_supported(void) +{ + int success = 1; + uint32_t r[4]; // eax, ebx, ecx, edx + +#if defined(_MSC_VER) + // This needs <intrin.h> with MSVC. ICC has it as a built-in + // on all platforms. + __cpuid(r, 1); +#elif defined(HAVE_CPUID_H) + // Compared to just using __asm__ to run CPUID, this also checks + // that CPUID is supported and saves and restores ebx as that is + // needed with GCC < 5 with position-independent code (PIC). + success = __get_cpuid(1, &r[0], &r[1], &r[2], &r[3]); +#else + // Just a fallback that shouldn't be needed. + __asm__("cpuid\n\t" + : "=a"(r[0]), "=b"(r[1]), "=c"(r[2]), "=d"(r[3]) + : "a"(1), "c"(0)); +#endif + + // Returns true if these are supported: + // CLMUL (bit 1 in ecx) + // SSSE3 (bit 9 in ecx) + // SSE4.1 (bit 19 in ecx) + const uint32_t ecx_mask = (1 << 1) | (1 << 9) | (1 << 19); + return success && (r[2] & ecx_mask) == ecx_mask; + + // Alternative methods that weren't used: + // - ICC's _may_i_use_cpu_feature: the other methods should work too. + // - GCC >= 6 / Clang / ICX __builtin_cpu_supports("pclmul") + // + // CPUID decding is needed with MSVC anyway and older GCC. This keeps + // the feature checks in the build system simpler too. The nice thing + // about __builtin_cpu_supports would be that it generates very short + // code as is it only reads a variable set at startup but a few bytes + // doesn't matter here. +} + + +#ifdef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +# define CRC64_FUNC_INIT +# define CRC64_SET_FUNC_ATTR __attribute__((__constructor__)) +#else +# define CRC64_FUNC_INIT = &crc64_dispatch +# define CRC64_SET_FUNC_ATTR +static uint64_t crc64_dispatch(const uint8_t *buf, size_t size, uint64_t crc); +#endif + + +// Pointer to the the selected CRC64 method. +static uint64_t (*crc64_func)(const uint8_t *buf, size_t size, uint64_t crc) + CRC64_FUNC_INIT; + + +CRC64_SET_FUNC_ATTR +static void +crc64_set_func(void) +{ + crc64_func = is_clmul_supported() ? &crc64_clmul : &crc64_generic; + return; +} + + +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +static uint64_t +crc64_dispatch(const uint8_t *buf, size_t size, uint64_t crc) +{ + // When __attribute__((__constructor__)) isn't supported, set the + // function pointer without any locking. If multiple threads run + // the detection code in parallel, they will all end up setting + // the pointer to the same value. This avoids the use of + // mythread_once() on every call to lzma_crc64() but this likely + // isn't strictly standards compliant. Let's change it if it breaks. + crc64_set_func(); + return crc64_func(buf, size, crc); +} +#endif +#endif + + +extern LZMA_API(uint64_t) +lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc) +{ +#if defined(CRC_GENERIC) && defined(CRC_CLMUL) + // If CLMUL is available, it is the best for non-tiny inputs, + // being over twice as fast as the generic slice-by-four version. + // However, for size <= 16 it's different. In the extreme case + // of size == 1 the generic version can be five times faster. + // At size >= 8 the CLMUL starts to become reasonable. It + // varies depending on the alignment of buf too. + // + // The above doesn't include the overhead of mythread_once(). + // At least on x86-64 GNU/Linux, pthread_once() is very fast but + // it still makes lzma_crc64(buf, 1, crc) 50-100 % slower. When + // size reaches 12-16 bytes the overhead becomes negligible. + // + // So using the generic version for size <= 16 may give better + // performance with tiny inputs but if such inputs happen rarely + // it's not so obvious because then the lookup table of the + // generic version may not be in the processor cache. +#ifdef CRC_USE_GENERIC_FOR_SMALL_INPUTS + if (size <= 16) + return crc64_generic(buf, size, crc); +#endif + +/* +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR + // See crc64_dispatch(). This would be the alternative which uses + // locking and doesn't use crc64_dispatch(). Note that on Windows + // this method needs Vista threads. + mythread_once(crc64_set_func); +#endif +*/ + + return crc64_func(buf, size, crc); + +#elif defined(CRC_CLMUL) + // If CLMUL is used unconditionally without runtime CPU detection + // then omitting the generic version and its 8 KiB lookup table + // makes the library smaller. + // + // FIXME: Lookup table isn't currently omitted on 32-bit x86, + // see crc64_table.c. + return crc64_clmul(buf, size, crc); + +#else + return crc64_generic(buf, size, crc); +#endif +} diff --git a/src/liblzma/check/crc64_small.c b/src/liblzma/check/crc64_small.c index 55d72316bce7..420f7cfbb475 100644 --- a/src/liblzma/check/crc64_small.c +++ b/src/liblzma/check/crc64_small.c @@ -16,6 +16,9 @@ static uint64_t crc64_table[256]; +#ifdef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR +__attribute__((__constructor__)) +#endif static void crc64_init(void) { @@ -40,7 +43,9 @@ crc64_init(void) extern LZMA_API(uint64_t) lzma_crc64(const uint8_t *buf, size_t size, uint64_t crc) { +#ifndef HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR mythread_once(crc64_init); +#endif crc = ~crc; diff --git a/src/liblzma/check/crc64_table.c b/src/liblzma/check/crc64_table.c index 7560eb0a3b15..241adcd4a5cb 100644 --- a/src/liblzma/check/crc64_table.c +++ b/src/liblzma/check/crc64_table.c @@ -12,11 +12,24 @@ #include "common.h" + +// FIXME: Compared to crc64_fast.c this has to check for __x86_64__ too +// so that in 32-bit builds crc64_x86.S won't break due to a missing table. +#if (defined(__x86_64__) && defined(__SSSE3__) \ + && defined(__SSE4_1__) && defined(__PCLMUL__)) \ + || (defined(__e2k__) && __iset__ >= 6) +// No table needed but something has to be exported to keep some toolchains +// happy. Also use a declaration to silence compiler warnings. +extern const char lzma_crc64_dummy; +const char lzma_crc64_dummy; + +#else // Having the declaration here silences clang -Wmissing-variable-declarations. extern const uint64_t lzma_crc64_table[4][256]; -#ifdef WORDS_BIGENDIAN -# include "crc64_table_be.h" -#else -# include "crc64_table_le.h" +# if defined(WORDS_BIGENDIAN) +# include "crc64_table_be.h" +# else +# include "crc64_table_le.h" +# endif #endif diff --git a/src/liblzma/common/alone_decoder.c b/src/liblzma/common/alone_decoder.c index a3ea20a27a9c..1dc85badf941 100644 --- a/src/liblzma/common/alone_decoder.c +++ b/src/liblzma/common/alone_decoder.c @@ -110,12 +110,24 @@ alone_decode(void *coder_ptr, const lzma_allocator *allocator, // Another hack to ditch false positives: Assume that // if the uncompressed size is known, it must be less // than 256 GiB. + // + // FIXME? Without picky we allow > LZMA_VLI_MAX which doesn't + // really matter in this specific situation (> LZMA_VLI_MAX is + // safe in the LZMA decoder) but it's somewhat weird still. if (coder->picky && coder->uncompressed_size != LZMA_VLI_UNKNOWN && coder->uncompressed_size >= (LZMA_VLI_C(1) << 38)) return LZMA_FORMAT_ERROR; + // Use LZMA_FILTER_LZMA1EXT features to specify the + // uncompressed size and that the end marker is allowed + // even when the uncompressed size is known. Both .lzma + // header and LZMA1EXT use UINT64_MAX indicate that size + // is unknown. + coder->options.ext_flags = LZMA_LZMA1EXT_ALLOW_EOPM; + lzma_set_ext_size(coder->options, coder->uncompressed_size); + // Calculate the memory usage so that it is ready // for SEQ_CODER_INIT. coder->memusage = lzma_lzma_decoder_memusage(&coder->options) @@ -132,6 +144,7 @@ alone_decode(void *coder_ptr, const lzma_allocator *allocator, lzma_filter_info filters[2] = { { + .id = LZMA_FILTER_LZMA1EXT, .init = &lzma_lzma_decoder_init, .options = &coder->options, }, { @@ -139,14 +152,8 @@ alone_decode(void *coder_ptr, const lzma_allocator *allocator, } }; - const lzma_ret ret = lzma_next_filter_init(&coder->next, - allocator, filters); - if (ret != LZMA_OK) - return ret; - - // Use a hack to set the uncompressed size. - lzma_lz_decoder_uncompressed(coder->next.coder, - coder->uncompressed_size, true); + return_if_error(lzma_next_filter_init(&coder->next, + allocator, filters)); coder->sequence = SEQ_CODE; break; diff --git a/src/liblzma/common/alone_encoder.c b/src/liblzma/common/alone_encoder.c index 96c1db70cc6a..c9392f376948 100644 --- a/src/liblzma/common/alone_encoder.c +++ b/src/liblzma/common/alone_encoder.c @@ -129,6 +129,7 @@ alone_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, // Initialize the LZMA encoder. const lzma_filter_info filters[2] = { { + .id = LZMA_FILTER_LZMA1, .init = &lzma_lzma_encoder_init, .options = (void *)(options), }, { diff --git a/src/liblzma/common/auto_decoder.c b/src/liblzma/common/auto_decoder.c index f58ab595ee3c..2a5c0894d123 100644 --- a/src/liblzma/common/auto_decoder.c +++ b/src/liblzma/common/auto_decoder.c @@ -1,7 +1,7 @@ /////////////////////////////////////////////////////////////////////////////// // /// \file auto_decoder.c -/// \brief Autodetect between .xz Stream and .lzma (LZMA_Alone) formats +/// \brief Autodetect between .xz, .lzma (LZMA_Alone), and .lz (lzip) // // Author: Lasse Collin // @@ -12,10 +12,13 @@ #include "stream_decoder.h" #include "alone_decoder.h" +#ifdef HAVE_LZIP_DECODER +# include "lzip_decoder.h" +#endif typedef struct { - /// Stream decoder or LZMA_Alone decoder + /// .xz Stream decoder, LZMA_Alone decoder, or lzip decoder lzma_next_coder next; uint64_t memlimit; @@ -46,14 +49,22 @@ auto_decode(void *coder_ptr, const lzma_allocator *allocator, // SEQ_CODE even if we return some LZMA_*_CHECK. coder->sequence = SEQ_CODE; - // Detect the file format. For now this is simple, since if - // it doesn't start with 0xFD (the first magic byte of the - // new format), it has to be LZMA_Alone, or something that - // we don't support at all. + // Detect the file format. .xz files start with 0xFD which + // cannot be the first byte of .lzma (LZMA_Alone) format. + // The .lz format starts with 0x4C which could be the + // first byte of a .lzma file but luckily it would mean + // lc/lp/pb being 4/3/1 which liblzma doesn't support because + // lc + lp > 4. So using just 0x4C to detect .lz is OK here. if (in[*in_pos] == 0xFD) { return_if_error(lzma_stream_decoder_init( &coder->next, allocator, coder->memlimit, coder->flags)); +#ifdef HAVE_LZIP_DECODER + } else if (in[*in_pos] == 0x4C) { + return_if_error(lzma_lzip_decoder_init( + &coder->next, allocator, + coder->memlimit, coder->flags)); +#endif } else { return_if_error(lzma_alone_decoder_init(&coder->next, allocator, coder->memlimit, true)); diff --git a/src/liblzma/common/block_header_decoder.c b/src/liblzma/common/block_header_decoder.c index 2e1135dd639a..c4935dcf46c1 100644 --- a/src/liblzma/common/block_header_decoder.c +++ b/src/liblzma/common/block_header_decoder.c @@ -14,22 +14,6 @@ #include "check.h" -static void -free_properties(lzma_block *block, const lzma_allocator *allocator) -{ - // Free allocated filter options. The last array member is not - // touched after the initialization in the beginning of - // lzma_block_header_decode(), so we don't need to touch that here. - for (size_t i = 0; i < LZMA_FILTERS_MAX; ++i) { - lzma_free(block->filters[i].options, allocator); - block->filters[i].id = LZMA_VLI_UNKNOWN; - block->filters[i].options = NULL; - } - - return; -} - - extern LZMA_API(lzma_ret) lzma_block_header_decode(lzma_block *block, const lzma_allocator *allocator, const uint8_t *in) @@ -39,6 +23,10 @@ lzma_block_header_decode(lzma_block *block, // are invalid or over 63 bits, or if the header is too small // to contain the claimed information. + // Catch unexpected NULL pointers. + if (block == NULL || block->filters == NULL || in == NULL) + return LZMA_PROG_ERROR; + // Initialize the filter options array. This way the caller can // safely free() the options even if an error occurs in this function. for (size_t i = 0; i <= LZMA_FILTERS_MAX; ++i) { @@ -67,8 +55,11 @@ lzma_block_header_decode(lzma_block *block, const size_t in_size = block->header_size - 4; // Verify CRC32 - if (lzma_crc32(in, in_size, 0) != read32le(in + in_size)) + if (lzma_crc32(in, in_size, 0) != read32le(in + in_size)) { +#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION return LZMA_DATA_ERROR; +#endif + } // Check for unsupported flags. if (in[1] & 0x3C) @@ -104,7 +95,7 @@ lzma_block_header_decode(lzma_block *block, &block->filters[i], allocator, in, &in_pos, in_size); if (ret != LZMA_OK) { - free_properties(block, allocator); + lzma_filters_free(block->filters, allocator); return ret; } } @@ -112,7 +103,7 @@ lzma_block_header_decode(lzma_block *block, // Padding while (in_pos < in_size) { if (in[in_pos++] != 0x00) { - free_properties(block, allocator); + lzma_filters_free(block->filters, allocator); // Possibly some new field present so use // LZMA_OPTIONS_ERROR instead of LZMA_DATA_ERROR. diff --git a/src/liblzma/common/common.c b/src/liblzma/common/common.c index 10fc884099a4..a708fdf187a6 100644 --- a/src/liblzma/common/common.c +++ b/src/liblzma/common/common.c @@ -211,7 +211,6 @@ lzma_code(lzma_stream *strm, lzma_action action) || strm->reserved_ptr2 != NULL || strm->reserved_ptr3 != NULL || strm->reserved_ptr4 != NULL - || strm->reserved_int1 != 0 || strm->reserved_int2 != 0 || strm->reserved_int3 != 0 || strm->reserved_int4 != 0 @@ -299,9 +298,7 @@ lzma_code(lzma_stream *strm, lzma_action action) strm->internal->avail_in = strm->avail_in; - // Cast is needed to silence a warning about LZMA_TIMED_OUT, which - // isn't part of lzma_ret enumeration. - switch ((unsigned int)(ret)) { + switch (ret) { case LZMA_OK: // Don't return LZMA_BUF_ERROR when it happens the first time. // This is to avoid returning LZMA_BUF_ERROR when avail_out @@ -322,6 +319,17 @@ lzma_code(lzma_stream *strm, lzma_action action) ret = LZMA_OK; break; + case LZMA_SEEK_NEEDED: + strm->internal->allow_buf_error = false; + + // If LZMA_FINISH was used, reset it back to the + // LZMA_RUN-based state so that new input can be supplied + // by the application. + if (strm->internal->sequence == ISEQ_FINISH) + strm->internal->sequence = ISEQ_RUN; + + break; + case LZMA_STREAM_END: if (strm->internal->sequence == ISEQ_SYNC_FLUSH || strm->internal->sequence == ISEQ_FULL_FLUSH diff --git a/src/liblzma/common/common.h b/src/liblzma/common/common.h index 7fb1732adff7..11fec52c5970 100644 --- a/src/liblzma/common/common.h +++ b/src/liblzma/common/common.h @@ -34,6 +34,14 @@ #include "lzma.h" +// This is for detecting modern GCC and Clang attributes +// like __symver__ in GCC >= 10. +#ifdef __has_attribute +# define lzma_has_attribute(attr) __has_attribute(attr) +#else +# define lzma_has_attribute(attr) 0 +#endif + // The extra symbol versioning in the C files may only be used when // building a shared library. If HAVE_SYMBOL_VERSIONS_LINUX is defined // to 2 then symbol versioning is done only if also PIC is defined. @@ -63,7 +71,12 @@ // since 2000). When using @@ instead of @@@, the internal name must not be // the same as the external name to avoid problems in some situations. This // is why "#define foo_52 foo" is needed for the default symbol versions. -# if TUKLIB_GNUC_REQ(10, 0) && !defined(__INTEL_COMPILER) +// +// __has_attribute is supported before GCC 10 and it is supported in Clang 14 +// too (which doesn't support __symver__) so use it to detect if __symver__ +// is available. This should be far more reliable than looking at compiler +// version macros as nowadays especially __GNUC__ is defined by many compilers. +# if lzma_has_attribute(__symver__) # define LZMA_SYMVER_API(extnamever, type, intname) \ extern __attribute__((__symver__(extnamever))) \ LZMA_API(type) intname @@ -107,14 +120,15 @@ #define LZMA_FILTER_RESERVED_START (LZMA_VLI_C(1) << 62) -/// Supported flags that can be passed to lzma_stream_decoder() -/// or lzma_auto_decoder(). +/// Supported flags that can be passed to lzma_stream_decoder(), +/// lzma_auto_decoder(), or lzma_stream_decoder_mt(). #define LZMA_SUPPORTED_FLAGS \ ( LZMA_TELL_NO_CHECK \ | LZMA_TELL_UNSUPPORTED_CHECK \ | LZMA_TELL_ANY_CHECK \ | LZMA_IGNORE_CHECK \ - | LZMA_CONCATENATED ) + | LZMA_CONCATENATED \ + | LZMA_FAIL_FAST ) /// Largest valid lzma_action value as unsigned integer. @@ -123,9 +137,12 @@ /// Special return value (lzma_ret) to indicate that a timeout was reached /// and lzma_code() must not return LZMA_BUF_ERROR. This is converted to -/// LZMA_OK in lzma_code(). This is not in the lzma_ret enumeration because -/// there's no need to have it in the public API. -#define LZMA_TIMED_OUT 32 +/// LZMA_OK in lzma_code(). +#define LZMA_TIMED_OUT LZMA_RET_INTERNAL1 + +/// Special return value (lzma_ret) for use in stream_decoder_mt.c to +/// indicate Index was detected instead of a Block Header. +#define LZMA_INDEX_DETECTED LZMA_RET_INTERNAL2 typedef struct lzma_next_coder_s lzma_next_coder; @@ -158,8 +175,11 @@ typedef void (*lzma_end_function)( /// an array of lzma_filter_info structures. This array is used with /// lzma_next_filter_init to initialize the filter chain. struct lzma_filter_info_s { - /// Filter ID. This is used only by the encoder - /// with lzma_filters_update(). + /// Filter ID. This can be used to share the same initiazation + /// function *and* data structures with different Filter IDs + /// (LZMA_FILTER_LZMA1EXT does it), and also by the encoder + /// with lzma_filters_update() if filter chain is updated + /// in the middle of a raw stream or Block (LZMA_SYNC_FLUSH). lzma_vli id; /// Pointer to function used to initialize the filter. @@ -213,6 +233,16 @@ struct lzma_next_coder_s { lzma_ret (*update)(void *coder, const lzma_allocator *allocator, const lzma_filter *filters, const lzma_filter *reversed_filters); + + /// Set how many bytes of output this coder may produce at maximum. + /// On success LZMA_OK must be returned. + /// If the filter chain as a whole cannot support this feature, + /// this must return LZMA_OPTIONS_ERROR. + /// If no input has been given to the coder and the requested limit + /// is too small, this must return LZMA_BUF_ERROR. If input has been + /// seen, LZMA_OK is allowed too. + lzma_ret (*set_out_limit)(void *coder, uint64_t *uncomp_size, + uint64_t out_limit); }; @@ -228,6 +258,7 @@ struct lzma_next_coder_s { .get_check = NULL, \ .memconfig = NULL, \ .update = NULL, \ + .set_out_limit = NULL, \ } diff --git a/src/liblzma/common/file_info.c b/src/liblzma/common/file_info.c new file mode 100644 index 000000000000..a6b7e145ae04 --- /dev/null +++ b/src/liblzma/common/file_info.c @@ -0,0 +1,855 @@ +/////////////////////////////////////////////////////////////////////////////// +// +/// \file file_info.c +/// \brief Decode .xz file information into a lzma_index structure +// +// Author: Lasse Collin +// +// This file has been put into the public domain. +// You can do whatever you want with this file. +// +/////////////////////////////////////////////////////////////////////////////// + +#include "index_decoder.h" + + +typedef struct { + enum { + SEQ_MAGIC_BYTES, + SEQ_PADDING_SEEK, + SEQ_PADDING_DECODE, + SEQ_FOOTER, + SEQ_INDEX_INIT, + SEQ_INDEX_DECODE, + SEQ_HEADER_DECODE, + SEQ_HEADER_COMPARE, + } sequence; + + /// Absolute position of in[*in_pos] in the file. All code that + /// modifies *in_pos also updates this. seek_to_pos() needs this + /// to determine if we need to request the application to seek for + /// us or if we can do the seeking internally by adjusting *in_pos. + uint64_t file_cur_pos; + + /// This refers to absolute positions of interesting parts of the + /// input file. Sometimes it points to the *beginning* of a specific + /// field and sometimes to the *end* of a field. The current target + /// position at each moment is explained in the comments. + uint64_t file_target_pos; + + /// Size of the .xz file (from the application). + uint64_t file_size; + + /// Index decoder + lzma_next_coder index_decoder; + + /// Number of bytes remaining in the Index field that is currently + /// being decoded. + lzma_vli index_remaining; + + /// The Index decoder will store the decoded Index in this pointer. + lzma_index *this_index; + + /// Amount of Stream Padding in the current Stream. + lzma_vli stream_padding; + + /// The final combined index is collected here. + lzma_index *combined_index; + + /// Pointer from the application where to store the index information + /// after successful decoding. + lzma_index **dest_index; + + /// Pointer to lzma_stream.seek_pos to be used when returning + /// LZMA_SEEK_NEEDED. This is set by seek_to_pos() when needed. + uint64_t *external_seek_pos; + + /// Memory usage limit + uint64_t memlimit; + + /// Stream Flags from the very beginning of the file. + lzma_stream_flags first_header_flags; + + /// Stream Flags from Stream Header of the current Stream. + lzma_stream_flags header_flags; + + /// Stream Flags from Stream Footer of the current Stream. + lzma_stream_flags footer_flags; + + size_t temp_pos; + size_t temp_size; + uint8_t temp[8192]; + +} lzma_file_info_coder; + + +/// Copies data from in[*in_pos] into coder->temp until +/// coder->temp_pos == coder->temp_size. This also keeps coder->file_cur_pos +/// in sync with *in_pos. Returns true if more input is needed. +static bool +fill_temp(lzma_file_info_coder *coder, const uint8_t *restrict in, + size_t *restrict in_pos, size_t in_size) +{ + coder->file_cur_pos += lzma_bufcpy(in, in_pos, in_size, + coder->temp, &coder->temp_pos, coder->temp_size); + return coder->temp_pos < coder->temp_size; +} + + +/// Seeks to the absolute file position specified by target_pos. +/// This tries to do the seeking by only modifying *in_pos, if possible. +/// The main benefit of this is that if one passes the whole file at once +/// to lzma_code(), the decoder will never need to return LZMA_SEEK_NEEDED +/// as all the seeking can be done by adjusting *in_pos in this function. +/// +/// Returns true if an external seek is needed and the caller must return +/// LZMA_SEEK_NEEDED. +static bool +seek_to_pos(lzma_file_info_coder *coder, uint64_t target_pos, + size_t in_start, size_t *in_pos, size_t in_size) +{ + // The input buffer doesn't extend beyond the end of the file. + // This has been checked by file_info_decode() already. + assert(coder->file_size - coder->file_cur_pos >= in_size - *in_pos); + + const uint64_t pos_min = coder->file_cur_pos - (*in_pos - in_start); + const uint64_t pos_max = coder->file_cur_pos + (in_size - *in_pos); + + bool external_seek_needed; + + if (target_pos >= pos_min && target_pos <= pos_max) { + // The requested position is available in the current input + // buffer or right after it. That is, in a corner case we + // end up setting *in_pos == in_size and thus will immediately + // need new input bytes from the application. + *in_pos += (size_t)(target_pos - coder->file_cur_pos); + external_seek_needed = false; + } else { + // Ask the application to seek the input file. + *coder->external_seek_pos = target_pos; + external_seek_needed = true; + + // Mark the whole input buffer as used. This way + // lzma_stream.total_in will have a better estimate + // of the amount of data read. It still won't be perfect + // as the value will depend on the input buffer size that + // the application uses, but it should be good enough for + // those few who want an estimate. + *in_pos = in_size; + } + + // After seeking (internal or external) the current position + // will match the requested target position. + coder->file_cur_pos = target_pos; + + return external_seek_needed; +} + + +/// The caller sets coder->file_target_pos so that it points to the *end* +/// of the desired file position. This function then determines how far +/// backwards from that position we can seek. After seeking fill_temp() +/// can be used to read data into coder->temp. When fill_temp() has finished, +/// coder->temp[coder->temp_size] will match coder->file_target_pos. +/// +/// This also validates that coder->target_file_pos is sane in sense that +/// we aren't trying to seek too far backwards (too close or beyond the +/// beginning of the file). +static lzma_ret +reverse_seek(lzma_file_info_coder *coder, + size_t in_start, size_t *in_pos, size_t in_size) +{ + // Check that there is enough data before the target position + // to contain at least Stream Header and Stream Footer. If there + // isn't, the file cannot be valid. + if (coder->file_target_pos < 2 * LZMA_STREAM_HEADER_SIZE) + return LZMA_DATA_ERROR; + + coder->temp_pos = 0; + + // The Stream Header at the very beginning of the file gets handled + // specially in SEQ_MAGIC_BYTES and thus we will never need to seek + // there. By not seeking to the first LZMA_STREAM_HEADER_SIZE bytes + // we avoid a useless external seek after SEQ_MAGIC_BYTES if the + // application uses an extremely small input buffer and the input + // file is very small. + if (coder->file_target_pos - LZMA_STREAM_HEADER_SIZE + < sizeof(coder->temp)) + coder->temp_size = (size_t)(coder->file_target_pos + - LZMA_STREAM_HEADER_SIZE); + else + coder->temp_size = sizeof(coder->temp); + + // The above if-statements guarantee this. This is important because + // the Stream Header/Footer decoders assume that there's at least + // LZMA_STREAM_HEADER_SIZE bytes in coder->temp. + assert(coder->temp_size >= LZMA_STREAM_HEADER_SIZE); + + if (seek_to_pos(coder, coder->file_target_pos - coder->temp_size, + in_start, in_pos, in_size)) + return LZMA_SEEK_NEEDED; + + return LZMA_OK; +} + + +/// Gets the number of zero-bytes at the end of the buffer. +static size_t +get_padding_size(const uint8_t *buf, size_t buf_size) +{ + size_t padding = 0; + while (buf_size > 0 && buf[--buf_size] == 0x00) + ++padding; + + return padding; +} + + +/// With the Stream Header at the very beginning of the file, LZMA_FORMAT_ERROR +/// is used to tell the application that Magic Bytes didn't match. In other +/// Stream Header/Footer fields (in the middle/end of the file) it could be +/// a bit confusing to return LZMA_FORMAT_ERROR as we already know that there +/// is a valid Stream Header at the beginning of the file. For those cases +/// this function is used to convert LZMA_FORMAT_ERROR to LZMA_DATA_ERROR. +static lzma_ret +hide_format_error(lzma_ret ret) +{ + if (ret == LZMA_FORMAT_ERROR) + ret = LZMA_DATA_ERROR; + + return ret; +} + + +/// Calls the Index decoder and updates coder->index_remaining. +/// This is a separate function because the input can be either directly +/// from the application or from coder->temp. +static lzma_ret +decode_index(lzma_file_info_coder *coder, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, bool update_file_cur_pos) +{ + const size_t in_start = *in_pos; + + const lzma_ret ret = coder->index_decoder.code( + coder->index_decoder.coder, + allocator, in, in_pos, in_size, + NULL, NULL, 0, LZMA_RUN); + + coder->index_remaining -= *in_pos - in_start; + + if (update_file_cur_pos) + coder->file_cur_pos += *in_pos - in_start; + + return ret; +} + + +static lzma_ret +file_info_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, + uint8_t *restrict out lzma_attribute((__unused__)), + size_t *restrict out_pos lzma_attribute((__unused__)), + size_t out_size lzma_attribute((__unused__)), + lzma_action action lzma_attribute((__unused__))) +{ + lzma_file_info_coder *coder = coder_ptr; + const size_t in_start = *in_pos; + + // If the caller provides input past the end of the file, trim + // the extra bytes from the buffer so that we won't read too far. + assert(coder->file_size >= coder->file_cur_pos); + if (coder->file_size - coder->file_cur_pos < in_size - in_start) + in_size = in_start + + (size_t)(coder->file_size - coder->file_cur_pos); + + while (true) + switch (coder->sequence) { + case SEQ_MAGIC_BYTES: + // Decode the Stream Header at the beginning of the file + // first to check if the Magic Bytes match. The flags + // are stored in coder->first_header_flags so that we + // don't need to seek to it again. + // + // Check that the file is big enough to contain at least + // Stream Header. + if (coder->file_size < LZMA_STREAM_HEADER_SIZE) + return LZMA_FORMAT_ERROR; + + // Read the Stream Header field into coder->temp. + if (fill_temp(coder, in, in_pos, in_size)) + return LZMA_OK; + + // This is the only Stream Header/Footer decoding where we + // want to return LZMA_FORMAT_ERROR if the Magic Bytes don't + // match. Elsewhere it will be converted to LZMA_DATA_ERROR. + return_if_error(lzma_stream_header_decode( + &coder->first_header_flags, coder->temp)); + + // Now that we know that the Magic Bytes match, check the + // file size. It's better to do this here after checking the + // Magic Bytes since this way we can give LZMA_FORMAT_ERROR + // instead of LZMA_DATA_ERROR when the Magic Bytes don't + // match in a file that is too big or isn't a multiple of + // four bytes. + if (coder->file_size > LZMA_VLI_MAX || (coder->file_size & 3)) + return LZMA_DATA_ERROR; + + // Start looking for Stream Padding and Stream Footer + // at the end of the file. + coder->file_target_pos = coder->file_size; + + // Fall through + + case SEQ_PADDING_SEEK: + coder->sequence = SEQ_PADDING_DECODE; + return_if_error(reverse_seek( + coder, in_start, in_pos, in_size)); + + // Fall through + + case SEQ_PADDING_DECODE: { + // Copy to coder->temp first. This keeps the code simpler if + // the application only provides input a few bytes at a time. + if (fill_temp(coder, in, in_pos, in_size)) + return LZMA_OK; + + // Scan the buffer backwards to get the size of the + // Stream Padding field (if any). + const size_t new_padding = get_padding_size( + coder->temp, coder->temp_size); + coder->stream_padding += new_padding; + + // Set the target position to the beginning of Stream Padding + // that has been observed so far. If all Stream Padding has + // been seen, then the target position will be at the end + // of the Stream Footer field. + coder->file_target_pos -= new_padding; + + if (new_padding == coder->temp_size) { + // The whole buffer was padding. Seek backwards in + // the file to get more input. + coder->sequence = SEQ_PADDING_SEEK; + break; + } + + // Size of Stream Padding must be a multiple of 4 bytes. + if (coder->stream_padding & 3) + return LZMA_DATA_ERROR; + + coder->sequence = SEQ_FOOTER; + + // Calculate the amount of non-padding data in coder->temp. + coder->temp_size -= new_padding; + coder->temp_pos = coder->temp_size; + + // We can avoid an external seek if the whole Stream Footer + // is already in coder->temp. In that case SEQ_FOOTER won't + // read more input and will find the Stream Footer from + // coder->temp[coder->temp_size - LZMA_STREAM_HEADER_SIZE]. + // + // Otherwise we will need to seek. The seeking is done so + // that Stream Footer wil be at the end of coder->temp. + // This way it's likely that we also get a complete Index + // field into coder->temp without needing a separate seek + // for that (unless the Index field is big). + if (coder->temp_size < LZMA_STREAM_HEADER_SIZE) + return_if_error(reverse_seek( + coder, in_start, in_pos, in_size)); + } + + // Fall through + + case SEQ_FOOTER: + // Copy the Stream Footer field into coder->temp. + // If Stream Footer was already available in coder->temp + // in SEQ_PADDING_DECODE, then this does nothing. + if (fill_temp(coder, in, in_pos, in_size)) + return LZMA_OK; + + // Make coder->file_target_pos and coder->temp_size point + // to the beginning of Stream Footer and thus to the end + // of the Index field. coder->temp_pos will be updated + // a bit later. + coder->file_target_pos -= LZMA_STREAM_HEADER_SIZE; + coder->temp_size -= LZMA_STREAM_HEADER_SIZE; + + // Decode Stream Footer. + return_if_error(hide_format_error(lzma_stream_footer_decode( + &coder->footer_flags, + coder->temp + coder->temp_size))); + + // Check that we won't seek past the beginning of the file. + // + // LZMA_STREAM_HEADER_SIZE is added because there must be + // space for Stream Header too even though we won't seek + // there before decoding the Index field. + // + // There's no risk of integer overflow here because + // Backward Size cannot be greater than 2^34. + if (coder->file_target_pos < coder->footer_flags.backward_size + + LZMA_STREAM_HEADER_SIZE) + return LZMA_DATA_ERROR; + + // Set the target position to the beginning of the Index field. + coder->file_target_pos -= coder->footer_flags.backward_size; + coder->sequence = SEQ_INDEX_INIT; + + // We can avoid an external seek if the whole Index field is + // already available in coder->temp. + if (coder->temp_size >= coder->footer_flags.backward_size) { + // Set coder->temp_pos to point to the beginning + // of the Index. + coder->temp_pos = coder->temp_size + - coder->footer_flags.backward_size; + } else { + // These are set to zero to indicate that there's no + // useful data (Index or anything else) in coder->temp. + coder->temp_pos = 0; + coder->temp_size = 0; + + // Seek to the beginning of the Index field. + if (seek_to_pos(coder, coder->file_target_pos, + in_start, in_pos, in_size)) + return LZMA_SEEK_NEEDED; + } + + // Fall through + + case SEQ_INDEX_INIT: { + // Calculate the amount of memory already used by the earlier + // Indexes so that we know how big memory limit to pass to + // the Index decoder. + // + // NOTE: When there are multiple Streams, the separate + // lzma_index structures can use more RAM (as measured by + // lzma_index_memused()) than the final combined lzma_index. + // Thus memlimit may need to be slightly higher than the final + // calculated memory usage will be. This is perhaps a bit + // confusing to the application, but I think it shouldn't + // cause problems in practice. + uint64_t memused = 0; + if (coder->combined_index != NULL) { + memused = lzma_index_memused(coder->combined_index); + assert(memused <= coder->memlimit); + if (memused > coder->memlimit) // Extra sanity check + return LZMA_PROG_ERROR; + } + + // Initialize the Index decoder. + return_if_error(lzma_index_decoder_init( + &coder->index_decoder, allocator, + &coder->this_index, + coder->memlimit - memused)); + + coder->index_remaining = coder->footer_flags.backward_size; + coder->sequence = SEQ_INDEX_DECODE; + } + + // Fall through + + case SEQ_INDEX_DECODE: { + // Decode (a part of) the Index. If the whole Index is already + // in coder->temp, read it from there. Otherwise read from + // in[*in_pos] onwards. Note that index_decode() updates + // coder->index_remaining and optionally coder->file_cur_pos. + lzma_ret ret; + if (coder->temp_size != 0) { + assert(coder->temp_size - coder->temp_pos + == coder->index_remaining); + ret = decode_index(coder, allocator, coder->temp, + &coder->temp_pos, coder->temp_size, + false); + } else { + // Don't give the decoder more input than the known + // remaining size of the Index field. + size_t in_stop = in_size; + if (in_size - *in_pos > coder->index_remaining) + in_stop = *in_pos + + (size_t)(coder->index_remaining); + + ret = decode_index(coder, allocator, + in, in_pos, in_stop, true); + } + + switch (ret) { + case LZMA_OK: + // If the Index docoder asks for more input when we + // have already given it as much input as Backward Size + // indicated, the file is invalid. + if (coder->index_remaining == 0) + return LZMA_DATA_ERROR; + + // We cannot get here if we were reading Index from + // coder->temp because when reading from coder->temp + // we give the Index decoder exactly + // coder->index_remaining bytes of input. + assert(coder->temp_size == 0); + + return LZMA_OK; + + case LZMA_STREAM_END: + // If the decoding seems to be successful, check also + // that the Index decoder consumed as much input as + // indicated by the Backward Size field. + if (coder->index_remaining != 0) + return LZMA_DATA_ERROR; + + break; + + default: + return ret; + } + + // Calculate how much the Index tells us to seek backwards + // (relative to the beginning of the Index): Total size of + // all Blocks plus the size of the Stream Header field. + // No integer overflow here because lzma_index_total_size() + // cannot return a value greater than LZMA_VLI_MAX. + const uint64_t seek_amount + = lzma_index_total_size(coder->this_index) + + LZMA_STREAM_HEADER_SIZE; + + // Check that Index is sane in sense that seek_amount won't + // make us seek past the beginning of the file when locating + // the Stream Header. + // + // coder->file_target_pos still points to the beginning of + // the Index field. + if (coder->file_target_pos < seek_amount) + return LZMA_DATA_ERROR; + + // Set the target to the beginning of Stream Header. + coder->file_target_pos -= seek_amount; + + if (coder->file_target_pos == 0) { + // We would seek to the beginning of the file, but + // since we already decoded that Stream Header in + // SEQ_MAGIC_BYTES, we can use the cached value from + // coder->first_header_flags to avoid the seek. + coder->header_flags = coder->first_header_flags; + coder->sequence = SEQ_HEADER_COMPARE; + break; + } + + coder->sequence = SEQ_HEADER_DECODE; + + // Make coder->file_target_pos point to the end of + // the Stream Header field. + coder->file_target_pos += LZMA_STREAM_HEADER_SIZE; + + // If coder->temp_size is non-zero, it points to the end + // of the Index field. Then the beginning of the Index + // field is at coder->temp[coder->temp_size + // - coder->footer_flags.backward_size]. + assert(coder->temp_size == 0 || coder->temp_size + >= coder->footer_flags.backward_size); + + // If coder->temp contained the whole Index, see if it has + // enough data to contain also the Stream Header. If so, + // we avoid an external seek. + // + // NOTE: This can happen only with small .xz files and only + // for the non-first Stream as the Stream Flags of the first + // Stream are cached and already handled a few lines above. + // So this isn't as useful as the other seek-avoidance cases. + if (coder->temp_size != 0 && coder->temp_size + - coder->footer_flags.backward_size + >= seek_amount) { + // Make temp_pos and temp_size point to the *end* of + // Stream Header so that SEQ_HEADER_DECODE will find + // the start of Stream Header from coder->temp[ + // coder->temp_size - LZMA_STREAM_HEADER_SIZE]. + coder->temp_pos = coder->temp_size + - coder->footer_flags.backward_size + - seek_amount + + LZMA_STREAM_HEADER_SIZE; + coder->temp_size = coder->temp_pos; + } else { + // Seek so that Stream Header will be at the end of + // coder->temp. With typical multi-Stream files we + // will usually also get the Stream Footer and Index + // of the *previous* Stream in coder->temp and thus + // won't need a separate seek for them. + return_if_error(reverse_seek(coder, + in_start, in_pos, in_size)); + } + } + + // Fall through + + case SEQ_HEADER_DECODE: + // Copy the Stream Header field into coder->temp. + // If Stream Header was already available in coder->temp + // in SEQ_INDEX_DECODE, then this does nothing. + if (fill_temp(coder, in, in_pos, in_size)) + return LZMA_OK; + + // Make all these point to the beginning of Stream Header. + coder->file_target_pos -= LZMA_STREAM_HEADER_SIZE; + coder->temp_size -= LZMA_STREAM_HEADER_SIZE; + coder->temp_pos = coder->temp_size; + + // Decode the Stream Header. + return_if_error(hide_format_error(lzma_stream_header_decode( + &coder->header_flags, + coder->temp + coder->temp_size))); + + coder->sequence = SEQ_HEADER_COMPARE; + + // Fall through + + case SEQ_HEADER_COMPARE: + // Compare Stream Header against Stream Footer. They must + // match. + return_if_error(lzma_stream_flags_compare( + &coder->header_flags, &coder->footer_flags)); + + // Store the decoded Stream Flags into the Index. Use the + // Footer Flags because it contains Backward Size, although + // it shouldn't matter in practice. + if (lzma_index_stream_flags(coder->this_index, + &coder->footer_flags) != LZMA_OK) + return LZMA_PROG_ERROR; + + // Store also the size of the Stream Padding field. It is + // needed to calculate the offsets of the Streams correctly. + if (lzma_index_stream_padding(coder->this_index, + coder->stream_padding) != LZMA_OK) + return LZMA_PROG_ERROR; + + // Reset it so that it's ready for the next Stream. + coder->stream_padding = 0; + + // Append the earlier decoded Indexes after this_index. + if (coder->combined_index != NULL) + return_if_error(lzma_index_cat(coder->this_index, + coder->combined_index, allocator)); + + coder->combined_index = coder->this_index; + coder->this_index = NULL; + + // If the whole file was decoded, tell the caller that we + // are finished. + if (coder->file_target_pos == 0) { + // The combined index must indicate the same file + // size as was told to us at initialization. + assert(lzma_index_file_size(coder->combined_index) + == coder->file_size); + + // Make the combined index available to + // the application. + *coder->dest_index = coder->combined_index; + coder->combined_index = NULL; + + // Mark the input buffer as used since we may have + // done internal seeking and thus don't know how + // many input bytes were actually used. This way + // lzma_stream.total_in gets a slightly better + // estimate of the amount of input used. + *in_pos = in_size; + return LZMA_STREAM_END; + } + + // We didn't hit the beginning of the file yet, so continue + // reading backwards in the file. If we have unprocessed + // data in coder->temp, use it before requesting more data + // from the application. + // + // coder->file_target_pos, coder->temp_size, and + // coder->temp_pos all point to the beginning of Stream Header + // and thus the end of the previous Stream in the file. + coder->sequence = coder->temp_size > 0 + ? SEQ_PADDING_DECODE : SEQ_PADDING_SEEK; + break; + + default: + assert(0); + return LZMA_PROG_ERROR; + } +} + + +static lzma_ret +file_info_decoder_memconfig(void *coder_ptr, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit) +{ + lzma_file_info_coder *coder = coder_ptr; + + // The memory usage calculation comes from three things: + // + // (1) The Indexes that have already been decoded and processed into + // coder->combined_index. + // + // (2) The latest Index in coder->this_index that has been decoded but + // not yet put into coder->combined_index. + // + // (3) The latest Index that we have started decoding but haven't + // finished and thus isn't available in coder->this_index yet. + // Memory usage and limit information needs to be communicated + // from/to coder->index_decoder. + // + // Care has to be taken to not do both (2) and (3) when calculating + // the memory usage. + uint64_t combined_index_memusage = 0; + uint64_t this_index_memusage = 0; + + // (1) If we have already successfully decoded one or more Indexes, + // get their memory usage. + if (coder->combined_index != NULL) + combined_index_memusage = lzma_index_memused( + coder->combined_index); + + // Choose between (2), (3), or neither. + if (coder->this_index != NULL) { + // (2) The latest Index is available. Use its memory usage. + this_index_memusage = lzma_index_memused(coder->this_index); + + } else if (coder->sequence == SEQ_INDEX_DECODE) { + // (3) The Index decoder is activate and hasn't yet stored + // the new index in coder->this_index. Get the memory usage + // information from the Index decoder. + // + // NOTE: If the Index decoder doesn't yet know how much memory + // it will eventually need, it will return a tiny value here. + uint64_t dummy; + if (coder->index_decoder.memconfig(coder->index_decoder.coder, + &this_index_memusage, &dummy, 0) + != LZMA_OK) { + assert(0); + return LZMA_PROG_ERROR; + } + } + + // Now we know the total memory usage/requirement. If we had neither + // old Indexes nor a new Index, this will be zero which isn't + // acceptable as lzma_memusage() has to return non-zero on success + // and even with an empty .xz file we will end up with a lzma_index + // that takes some memory. + *memusage = combined_index_memusage + this_index_memusage; + if (*memusage == 0) + *memusage = lzma_index_memusage(1, 0); + + *old_memlimit = coder->memlimit; + + // If requested, set a new memory usage limit. + if (new_memlimit != 0) { + if (new_memlimit < *memusage) + return LZMA_MEMLIMIT_ERROR; + + // In the condition (3) we need to tell the Index decoder + // its new memory usage limit. + if (coder->this_index == NULL + && coder->sequence == SEQ_INDEX_DECODE) { + const uint64_t idec_new_memlimit = new_memlimit + - combined_index_memusage; + + assert(this_index_memusage > 0); + assert(idec_new_memlimit > 0); + + uint64_t dummy1; + uint64_t dummy2; + + if (coder->index_decoder.memconfig( + coder->index_decoder.coder, + &dummy1, &dummy2, idec_new_memlimit) + != LZMA_OK) { + assert(0); + return LZMA_PROG_ERROR; + } + } + + coder->memlimit = new_memlimit; + } + + return LZMA_OK; +} + + +static void +file_info_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_file_info_coder *coder = coder_ptr; + + lzma_next_end(&coder->index_decoder, allocator); + lzma_index_end(coder->this_index, allocator); + lzma_index_end(coder->combined_index, allocator); + + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +lzma_file_info_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, uint64_t *seek_pos, + lzma_index **dest_index, + uint64_t memlimit, uint64_t file_size) +{ + lzma_next_coder_init(&lzma_file_info_decoder_init, next, allocator); + + if (dest_index == NULL) + return LZMA_PROG_ERROR; + + lzma_file_info_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_file_info_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &file_info_decode; + next->end = &file_info_decoder_end; + next->memconfig = &file_info_decoder_memconfig; + + coder->index_decoder = LZMA_NEXT_CODER_INIT; + coder->this_index = NULL; + coder->combined_index = NULL; + } + + coder->sequence = SEQ_MAGIC_BYTES; + coder->file_cur_pos = 0; + coder->file_target_pos = 0; + coder->file_size = file_size; + + lzma_index_end(coder->this_index, allocator); + coder->this_index = NULL; + + lzma_index_end(coder->combined_index, allocator); + coder->combined_index = NULL; + + coder->stream_padding = 0; + + coder->dest_index = dest_index; + coder->external_seek_pos = seek_pos; + + // If memlimit is 0, make it 1 to ensure that lzma_memlimit_get() + // won't return 0 (which would indicate an error). + coder->memlimit = my_max(1, memlimit); + + // Prepare these for reading the first Stream Header into coder->temp. + coder->temp_pos = 0; + coder->temp_size = LZMA_STREAM_HEADER_SIZE; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_file_info_decoder(lzma_stream *strm, lzma_index **dest_index, + uint64_t memlimit, uint64_t file_size) +{ + lzma_next_strm_init(lzma_file_info_decoder_init, strm, &strm->seek_pos, + dest_index, memlimit, file_size); + + // We allow LZMA_FINISH in addition to LZMA_RUN for convenience. + // lzma_code() is able to handle the LZMA_FINISH + LZMA_SEEK_NEEDED + // combination in a sane way. Applications still need to be careful + // if they use LZMA_FINISH so that they remember to reset it back + // to LZMA_RUN after seeking if needed. + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/filter_common.c b/src/liblzma/common/filter_common.c index 590be7303feb..fa0927cf9b9f 100644 --- a/src/liblzma/common/filter_common.c +++ b/src/liblzma/common/filter_common.c @@ -42,6 +42,13 @@ static const struct { .last_ok = true, .changes_size = true, }, + { + .id = LZMA_FILTER_LZMA1EXT, + .options_size = sizeof(lzma_options_lzma), + .non_last_ok = false, + .last_ok = true, + .changes_size = true, + }, #endif #if defined(HAVE_ENCODER_LZMA2) || defined(HAVE_DECODER_LZMA2) { @@ -97,6 +104,15 @@ static const struct { .changes_size = false, }, #endif +#if defined(HAVE_ENCODER_ARM64) || defined(HAVE_DECODER_ARM64) + { + .id = LZMA_FILTER_ARM64, + .options_size = sizeof(lzma_options_bcj), + .non_last_ok = true, + .last_ok = false, + .changes_size = false, + }, +#endif #if defined(HAVE_ENCODER_SPARC) || defined(HAVE_DECODER_SPARC) { .id = LZMA_FILTER_SPARC, @@ -196,8 +212,34 @@ error: } -static lzma_ret -validate_chain(const lzma_filter *filters, size_t *count) +extern LZMA_API(void) +lzma_filters_free(lzma_filter *filters, const lzma_allocator *allocator) +{ + if (filters == NULL) + return; + + for (size_t i = 0; filters[i].id != LZMA_VLI_UNKNOWN; ++i) { + if (i == LZMA_FILTERS_MAX) { + // The API says that LZMA_FILTERS_MAX + 1 is the + // maximum allowed size including the terminating + // element. Thus, we should never get here but in + // case there is a bug and we do anyway, don't go + // past the (probable) end of the array. + assert(0); + break; + } + + lzma_free(filters[i].options, allocator); + filters[i].options = NULL; + filters[i].id = LZMA_VLI_UNKNOWN; + } + + return; +} + + +extern lzma_ret +lzma_validate_chain(const lzma_filter *filters, size_t *count) { // There must be at least one filter. if (filters == NULL || filters[0].id == LZMA_VLI_UNKNOWN) @@ -251,7 +293,7 @@ lzma_raw_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, { // Do some basic validation and get the number of filters. size_t count; - return_if_error(validate_chain(options, &count)); + return_if_error(lzma_validate_chain(options, &count)); // Set the filter functions and copy the options pointer. lzma_filter_info filters[LZMA_FILTERS_MAX + 1]; @@ -304,7 +346,7 @@ lzma_raw_coder_memusage(lzma_filter_find coder_find, // The chain has to have at least one filter. { size_t tmp; - if (validate_chain(filters, &tmp) != LZMA_OK) + if (lzma_validate_chain(filters, &tmp) != LZMA_OK) return UINT64_MAX; } diff --git a/src/liblzma/common/filter_common.h b/src/liblzma/common/filter_common.h index 9390305c26c1..2e47bb69f7f6 100644 --- a/src/liblzma/common/filter_common.h +++ b/src/liblzma/common/filter_common.h @@ -35,6 +35,9 @@ typedef struct { typedef const lzma_filter_coder *(*lzma_filter_find)(lzma_vli id); +extern lzma_ret lzma_validate_chain(const lzma_filter *filters, size_t *count); + + extern lzma_ret lzma_raw_coder_init( lzma_next_coder *next, const lzma_allocator *allocator, const lzma_filter *filters, diff --git a/src/liblzma/common/filter_decoder.c b/src/liblzma/common/filter_decoder.c index c75b0a89c30f..fa53f5bdbad0 100644 --- a/src/liblzma/common/filter_decoder.c +++ b/src/liblzma/common/filter_decoder.c @@ -50,6 +50,12 @@ static const lzma_filter_decoder decoders[] = { .memusage = &lzma_lzma_decoder_memusage, .props_decode = &lzma_lzma_props_decode, }, + { + .id = LZMA_FILTER_LZMA1EXT, + .init = &lzma_lzma_decoder_init, + .memusage = &lzma_lzma_decoder_memusage, + .props_decode = &lzma_lzma_props_decode, + }, #endif #ifdef HAVE_DECODER_LZMA2 { @@ -99,6 +105,14 @@ static const lzma_filter_decoder decoders[] = { .props_decode = &lzma_simple_props_decode, }, #endif +#ifdef HAVE_DECODER_ARM64 + { + .id = LZMA_FILTER_ARM64, + .init = &lzma_simple_arm64_decoder_init, + .memusage = NULL, + .props_decode = &lzma_simple_props_decode, + }, +#endif #ifdef HAVE_DECODER_SPARC { .id = LZMA_FILTER_SPARC, diff --git a/src/liblzma/common/filter_encoder.c b/src/liblzma/common/filter_encoder.c index f0762d9a0afa..978b7a6bb5b8 100644 --- a/src/liblzma/common/filter_encoder.c +++ b/src/liblzma/common/filter_encoder.c @@ -64,6 +64,15 @@ static const lzma_filter_encoder encoders[] = { .props_size_fixed = 5, .props_encode = &lzma_lzma_props_encode, }, + { + .id = LZMA_FILTER_LZMA1EXT, + .init = &lzma_lzma_encoder_init, + .memusage = &lzma_lzma_encoder_memusage, + .block_size = NULL, // Not needed for LZMA1 + .props_size_get = NULL, + .props_size_fixed = 5, + .props_encode = &lzma_lzma_props_encode, + }, #endif #ifdef HAVE_ENCODER_LZMA2 { @@ -126,6 +135,16 @@ static const lzma_filter_encoder encoders[] = { .props_encode = &lzma_simple_props_encode, }, #endif +#ifdef HAVE_ENCODER_ARM64 + { + .id = LZMA_FILTER_ARM64, + .init = &lzma_simple_arm64_encoder_init, + .memusage = NULL, + .block_size = NULL, + .props_size_get = &lzma_simple_props_size, + .props_encode = &lzma_simple_props_encode, + }, +#endif #ifdef HAVE_ENCODER_SPARC { .id = LZMA_FILTER_SPARC, diff --git a/src/liblzma/common/index_decoder.c b/src/liblzma/common/index_decoder.c index cc07a1b8c533..b26898853381 100644 --- a/src/liblzma/common/index_decoder.c +++ b/src/liblzma/common/index_decoder.c @@ -10,7 +10,7 @@ // /////////////////////////////////////////////////////////////////////////////// -#include "index.h" +#include "index_decoder.h" #include "check.h" @@ -180,8 +180,11 @@ index_decode(void *coder_ptr, const lzma_allocator *allocator, return LZMA_OK; if (((coder->crc32 >> (coder->pos * 8)) & 0xFF) - != in[(*in_pos)++]) + != in[(*in_pos)++]) { +#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION return LZMA_DATA_ERROR; +#endif + } } while (++coder->pos < 4); @@ -265,11 +268,11 @@ index_decoder_reset(lzma_index_coder *coder, const lzma_allocator *allocator, } -static lzma_ret -index_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, +extern lzma_ret +lzma_index_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, lzma_index **i, uint64_t memlimit) { - lzma_next_coder_init(&index_decoder_init, next, allocator); + lzma_next_coder_init(&lzma_index_decoder_init, next, allocator); if (i == NULL) return LZMA_PROG_ERROR; @@ -296,7 +299,7 @@ index_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, extern LZMA_API(lzma_ret) lzma_index_decoder(lzma_stream *strm, lzma_index **i, uint64_t memlimit) { - lzma_next_strm_init(index_decoder_init, strm, i, memlimit); + lzma_next_strm_init(lzma_index_decoder_init, strm, i, memlimit); strm->internal->supported_actions[LZMA_RUN] = true; strm->internal->supported_actions[LZMA_FINISH] = true; diff --git a/src/liblzma/common/index_decoder.h b/src/liblzma/common/index_decoder.h new file mode 100644 index 000000000000..1af433b58b56 --- /dev/null +++ b/src/liblzma/common/index_decoder.h @@ -0,0 +1,24 @@ +/////////////////////////////////////////////////////////////////////////////// +// +/// \file index_decoder.h +/// \brief Decodes the Index field +// +// Author: Lasse Collin +// +// This file has been put into the public domain. +// You can do whatever you want with this file. +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_INDEX_DECODER_H +#define LZMA_INDEX_DECODER_H + +#include "index.h" + + +extern lzma_ret lzma_index_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + lzma_index **i, uint64_t memlimit); + + +#endif diff --git a/src/liblzma/common/index_hash.c b/src/liblzma/common/index_hash.c index 5c9d314039da..34df85d72f94 100644 --- a/src/liblzma/common/index_hash.c +++ b/src/liblzma/common/index_hash.c @@ -312,8 +312,11 @@ lzma_index_hash_decode(lzma_index_hash *index_hash, const uint8_t *in, return LZMA_OK; if (((index_hash->crc32 >> (index_hash->pos * 8)) - & 0xFF) != in[(*in_pos)++]) + & 0xFF) != in[(*in_pos)++]) { +#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION return LZMA_DATA_ERROR; +#endif + } } while (++index_hash->pos < 4); diff --git a/src/liblzma/common/lzip_decoder.c b/src/liblzma/common/lzip_decoder.c new file mode 100644 index 000000000000..20794f9466f0 --- /dev/null +++ b/src/liblzma/common/lzip_decoder.c @@ -0,0 +1,414 @@ +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzip_decoder.c +/// \brief Decodes .lz (lzip) files +// +// Author: Michał Górny +// Lasse Collin +// +// This file has been put into the public domain. +// You can do whatever you want with this file. +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lzip_decoder.h" +#include "lzma_decoder.h" +#include "check.h" + + +// .lz format version 0 lacks the 64-bit Member size field in the footer. +#define LZIP_V0_FOOTER_SIZE 12 +#define LZIP_V1_FOOTER_SIZE 20 +#define LZIP_FOOTER_SIZE_MAX LZIP_V1_FOOTER_SIZE + +// lc/lp/pb are hardcoded in the .lz format. +#define LZIP_LC 3 +#define LZIP_LP 0 +#define LZIP_PB 2 + + +typedef struct { + enum { + SEQ_ID_STRING, + SEQ_VERSION, + SEQ_DICT_SIZE, + SEQ_CODER_INIT, + SEQ_LZMA_STREAM, + SEQ_MEMBER_FOOTER, + } sequence; + + /// .lz member format version + uint32_t version; + + /// CRC32 of the uncompressed data in the .lz member + uint32_t crc32; + + /// Uncompressed size of the .lz member + uint64_t uncompressed_size; + + /// Compressed size of the .lz member + uint64_t member_size; + + /// Memory usage limit + uint64_t memlimit; + + /// Amount of memory actually needed + uint64_t memusage; + + /// If true, LZMA_GET_CHECK is returned after decoding the header + /// fields. As all files use CRC32 this is redundant but it's + /// implemented anyway since the initialization functions supports + /// all other flags in addition to LZMA_TELL_ANY_CHECK. + bool tell_any_check; + + /// If true, we won't calculate or verify the CRC32 of + /// the uncompressed data. + bool ignore_check; + + /// If true, we will decode concatenated .lz members and stop if + /// non-.lz data is seen after at least one member has been + /// successfully decoded. + bool concatenated; + + /// When decoding concatenated .lz members, this is true as long as + /// we are decoding the first .lz member. This is needed to avoid + /// incorrect LZMA_FORMAT_ERROR in case there is non-.lz data at + /// the end of the file. + bool first_member; + + /// Reading position in the header and footer fields + size_t pos; + + /// Buffer to hold the .lz footer fields + uint8_t buffer[LZIP_FOOTER_SIZE_MAX]; + + /// Options decoded from the .lz header that needed to initialize + /// the LZMA1 decoder. + lzma_options_lzma options; + + /// LZMA1 decoder + lzma_next_coder lzma_decoder; + +} lzma_lzip_coder; + + +static lzma_ret +lzip_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_lzip_coder *coder = coder_ptr; + + while (true) + switch (coder->sequence) { + case SEQ_ID_STRING: { + // The "ID string" or magic bytes are "LZIP" in US-ASCII. + const uint8_t lzip_id_string[4] = { 0x4C, 0x5A, 0x49, 0x50 }; + + while (coder->pos < sizeof(lzip_id_string)) { + if (*in_pos >= in_size) { + // If we are on the 2nd+ concatenated member + // and the input ends before we can read + // the magic bytes, we discard the bytes that + // were already read (up to 3) and finish. + // See the reasoning below. + return !coder->first_member + && action == LZMA_FINISH + ? LZMA_STREAM_END : LZMA_OK; + } + + if (in[*in_pos] != lzip_id_string[coder->pos]) { + // The .lz format allows putting non-.lz data + // at the end of the file. If we have seen + // at least one valid .lz member already, + // then we won't consume the byte at *in_pos + // and will return LZMA_STREAM_END. This way + // apps can easily locate and read the non-.lz + // data after the .lz member(s). + // + // NOTE: If the first 1-3 bytes of the non-.lz + // data match the .lz ID string then the first + // 1-3 bytes of the junk will get ignored by + // us. If apps want to properly locate the + // trailing data they must ensure that the + // first byte of their custom data isn't the + // same as the first byte of .lz ID string. + // With the liblzma API we cannot rewind the + // input position across calls to lzma_code(). + return !coder->first_member + ? LZMA_STREAM_END : LZMA_FORMAT_ERROR; + } + + ++*in_pos; + ++coder->pos; + } + + coder->pos = 0; + + coder->crc32 = 0; + coder->uncompressed_size = 0; + coder->member_size = sizeof(lzip_id_string); + + coder->sequence = SEQ_VERSION; + } + + // Fall through + + case SEQ_VERSION: + if (*in_pos >= in_size) + return LZMA_OK; + + coder->version = in[(*in_pos)++]; + + // We support version 0 and unextended version 1. + if (coder->version > 1) + return LZMA_OPTIONS_ERROR; + + ++coder->member_size; + coder->sequence = SEQ_DICT_SIZE; + + // .lz versions 0 and 1 use CRC32 as the integrity check + // so if the application wanted to know that + // (LZMA_TELL_ANY_CHECK) we can tell it now. + if (coder->tell_any_check) + return LZMA_GET_CHECK; + + // Fall through + + case SEQ_DICT_SIZE: { + if (*in_pos >= in_size) + return LZMA_OK; + + const uint32_t ds = in[(*in_pos)++]; + ++coder->member_size; + + // The five lowest bits are for the base-2 logarithm of + // the dictionary size and the highest three bits are + // the fractional part (0/16 to 7/16) that will be + // substracted to get the final value. + // + // For example, with 0xB5: + // b2log = 21 + // fracnum = 5 + // dict_size = 2^21 - 2^21 * 5 / 16 = 1408 KiB + const uint32_t b2log = ds & 0x1F; + const uint32_t fracnum = ds >> 5; + + // The format versions 0 and 1 allow dictionary size in the + // range [4 KiB, 512 MiB]. + if (b2log < 12 || b2log > 29 || (b2log == 12 && fracnum > 0)) + return LZMA_DATA_ERROR; + + // 2^[b2log] - 2^[b2log] * [fracnum] / 16 + // = 2^[b2log] - [fracnum] * 2^([b2log] - 4) + coder->options.dict_size = (UINT32_C(1) << b2log) + - (fracnum << (b2log - 4)); + + assert(coder->options.dict_size >= 4096); + assert(coder->options.dict_size <= (UINT32_C(512) << 20)); + + coder->options.preset_dict = NULL; + coder->options.lc = LZIP_LC; + coder->options.lp = LZIP_LP; + coder->options.pb = LZIP_PB; + + // Calculate the memory usage. + coder->memusage = lzma_lzma_decoder_memusage(&coder->options) + + LZMA_MEMUSAGE_BASE; + + // Initialization is a separate step because if we return + // LZMA_MEMLIMIT_ERROR we need to be able to restart after + // the memlimit has been increased. + coder->sequence = SEQ_CODER_INIT; + } + + // Fall through + + case SEQ_CODER_INIT: { + if (coder->memusage > coder->memlimit) + return LZMA_MEMLIMIT_ERROR; + + const lzma_filter_info filters[2] = { + { + .id = LZMA_FILTER_LZMA1, + .init = &lzma_lzma_decoder_init, + .options = &coder->options, + }, { + .init = NULL, + } + }; + + return_if_error(lzma_next_filter_init(&coder->lzma_decoder, + allocator, filters)); + + coder->crc32 = 0; + coder->sequence = SEQ_LZMA_STREAM; + } + + // Fall through + + case SEQ_LZMA_STREAM: { + const size_t in_start = *in_pos; + const size_t out_start = *out_pos; + + const lzma_ret ret = coder->lzma_decoder.code( + coder->lzma_decoder.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, + action); + + const size_t out_used = *out_pos - out_start; + + coder->member_size += *in_pos - in_start; + coder->uncompressed_size += out_used; + + if (!coder->ignore_check) + coder->crc32 = lzma_crc32(out + out_start, out_used, + coder->crc32); + + if (ret != LZMA_STREAM_END) + return ret; + + coder->sequence = SEQ_MEMBER_FOOTER; + } + + // Fall through + + case SEQ_MEMBER_FOOTER: { + // The footer of .lz version 0 lacks the Member size field. + // This is the only difference between version 0 and + // unextended version 1 formats. + const size_t footer_size = coder->version == 0 + ? LZIP_V0_FOOTER_SIZE + : LZIP_V1_FOOTER_SIZE; + + // Copy the CRC32, Data size, and Member size fields to + // the internal buffer. + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + footer_size); + + // Return if we didn't get the whole footer yet. + if (coder->pos < footer_size) + return LZMA_OK; + + coder->pos = 0; + coder->member_size += footer_size; + + // Check that the footer fields match the observed data. + if (!coder->ignore_check + && coder->crc32 != read32le(&coder->buffer[0])) + return LZMA_DATA_ERROR; + + if (coder->uncompressed_size != read64le(&coder->buffer[4])) + return LZMA_DATA_ERROR; + + if (coder->version > 0) { + // .lz version 0 has no Member size field. + if (coder->member_size != read64le(&coder->buffer[12])) + return LZMA_DATA_ERROR; + } + + // Decoding is finished if we weren't requested to decode + // more than one .lz member. + if (!coder->concatenated) + return LZMA_STREAM_END; + + coder->first_member = false; + coder->sequence = SEQ_ID_STRING; + break; + } + + default: + assert(0); + return LZMA_PROG_ERROR; + } + + // Never reached +} + + +static void +lzip_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_lzip_coder *coder = coder_ptr; + lzma_next_end(&coder->lzma_decoder, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_check +lzip_decoder_get_check(const void *coder_ptr lzma_attribute((__unused__))) +{ + return LZMA_CHECK_CRC32; +} + + +static lzma_ret +lzip_decoder_memconfig(void *coder_ptr, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit) +{ + lzma_lzip_coder *coder = coder_ptr; + + *memusage = coder->memusage; + *old_memlimit = coder->memlimit; + + if (new_memlimit != 0) { + if (new_memlimit < coder->memusage) + return LZMA_MEMLIMIT_ERROR; + + coder->memlimit = new_memlimit; + } + + return LZMA_OK; +} + + +extern lzma_ret +lzma_lzip_decoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t memlimit, uint32_t flags) +{ + lzma_next_coder_init(&lzma_lzip_decoder_init, next, allocator); + + if (flags & ~LZMA_SUPPORTED_FLAGS) + return LZMA_OPTIONS_ERROR; + + lzma_lzip_coder *coder = next->coder; + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_lzip_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = &lzip_decode; + next->end = &lzip_decoder_end; + next->get_check = &lzip_decoder_get_check; + next->memconfig = &lzip_decoder_memconfig; + + coder->lzma_decoder = LZMA_NEXT_CODER_INIT; + } + + coder->sequence = SEQ_ID_STRING; + coder->memlimit = my_max(1, memlimit); + coder->memusage = LZMA_MEMUSAGE_BASE; + coder->tell_any_check = (flags & LZMA_TELL_ANY_CHECK) != 0; + coder->ignore_check = (flags & LZMA_IGNORE_CHECK) != 0; + coder->concatenated = (flags & LZMA_CONCATENATED) != 0; + coder->first_member = true; + coder->pos = 0; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_lzip_decoder(lzma_stream *strm, uint64_t memlimit, uint32_t flags) +{ + lzma_next_strm_init(lzma_lzip_decoder_init, strm, memlimit, flags); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/lzip_decoder.h b/src/liblzma/common/lzip_decoder.h new file mode 100644 index 000000000000..33a01c352ce3 --- /dev/null +++ b/src/liblzma/common/lzip_decoder.h @@ -0,0 +1,22 @@ +/////////////////////////////////////////////////////////////////////////////// +// +/// \file lzip_decoder.h +/// \brief Decodes .lz (lzip) files +// +// Author: Michał Górny +// +// This file has been put into the public domain. +// You can do whatever you want with this file. +// +/////////////////////////////////////////////////////////////////////////////// + +#ifndef LZMA_LZIP_DECODER_H +#define LZMA_LZIP_DECODER_H + +#include "common.h" + +extern lzma_ret lzma_lzip_decoder_init( + lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t memlimit, uint32_t flags); + +#endif diff --git a/src/liblzma/common/memcmplen.h b/src/liblzma/common/memcmplen.h index b76a0b6327ed..5a481a02c9d8 100644 --- a/src/liblzma/common/memcmplen.h +++ b/src/liblzma/common/memcmplen.h @@ -51,10 +51,6 @@ lzma_memcmplen(const uint8_t *buf1, const uint8_t *buf2, || (defined(__INTEL_COMPILER) && defined(__x86_64__)) \ || (defined(__INTEL_COMPILER) && defined(_M_X64)) \ || (defined(_MSC_VER) && defined(_M_X64))) - // NOTE: This will use 64-bit unaligned access which - // TUKLIB_FAST_UNALIGNED_ACCESS wasn't meant to permit, but - // it's convenient here at least as long as it's x86-64 only. - // // I keep this x86-64 only for now since that's where I know this // to be a good method. This may be fine on other 64-bit CPUs too. // On big endian one should use xor instead of subtraction and switch @@ -83,8 +79,9 @@ lzma_memcmplen(const uint8_t *buf1, const uint8_t *buf2, && (defined(__SSE2__) \ || (defined(_MSC_VER) && defined(_M_IX86_FP) \ && _M_IX86_FP >= 2)) - // NOTE: Like above, this will use 128-bit unaligned access which - // TUKLIB_FAST_UNALIGNED_ACCESS wasn't meant to permit. + // NOTE: This will use 128-bit unaligned access which + // TUKLIB_FAST_UNALIGNED_ACCESS wasn't meant to permit, + // but it's convenient here since this is x86-only. // // SSE2 version for 32-bit and 64-bit x86. On x86-64 the above // version is sometimes significantly faster and sometimes diff --git a/src/liblzma/common/microlzma_decoder.c b/src/liblzma/common/microlzma_decoder.c new file mode 100644 index 000000000000..e473373daaae --- /dev/null +++ b/src/liblzma/common/microlzma_decoder.c @@ -0,0 +1,221 @@ +/////////////////////////////////////////////////////////////////////////////// +// +/// \file microlzma_decoder.c +/// \brief Decode MicroLZMA format +// +// Author: Lasse Collin +// +// This file has been put into the public domain. +// You can do whatever you want with this file. +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lzma_decoder.h" +#include "lz_decoder.h" + + +typedef struct { + /// LZMA1 decoder + lzma_next_coder lzma; + + /// Compressed size of the stream as given by the application. + /// This must be exactly correct. + /// + /// This will be decremented when input is read. + uint64_t comp_size; + + /// Uncompressed size of the stream as given by the application. + /// This may be less than the actual uncompressed size if + /// uncomp_size_is_exact is false. + /// + /// This will be decremented when output is produced. + lzma_vli uncomp_size; + + /// LZMA dictionary size as given by the application + uint32_t dict_size; + + /// If true, the exact uncompressed size is known. If false, + /// uncomp_size may be smaller than the real uncompressed size; + /// uncomp_size may never be bigger than the real uncompressed size. + bool uncomp_size_is_exact; + + /// True once the first byte of the MicroLZMA stream + /// has been processed. + bool props_decoded; +} lzma_microlzma_coder; + + +static lzma_ret +microlzma_decode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_microlzma_coder *coder = coder_ptr; + + // Remember the in start position so that we can update comp_size. + const size_t in_start = *in_pos; + + // Remember the out start position so that we can update uncomp_size. + const size_t out_start = *out_pos; + + // Limit the amount of input so that the decoder won't read more than + // comp_size. This is required when uncomp_size isn't exact because + // in that case the LZMA decoder will try to decode more input even + // when it has no output space (it can be looking for EOPM). + if (in_size - *in_pos > coder->comp_size) + in_size = *in_pos + (size_t)(coder->comp_size); + + // When the exact uncompressed size isn't known, we must limit + // the available output space to prevent the LZMA decoder from + // trying to decode too much. + if (!coder->uncomp_size_is_exact + && out_size - *out_pos > coder->uncomp_size) + out_size = *out_pos + (size_t)(coder->uncomp_size); + + if (!coder->props_decoded) { + // There must be at least one byte of input to decode + // the properties byte. + if (*in_pos >= in_size) + return LZMA_OK; + + lzma_options_lzma options = { + .dict_size = coder->dict_size, + .preset_dict = NULL, + .preset_dict_size = 0, + .ext_flags = 0, // EOPM not allowed when size is known + .ext_size_low = UINT32_MAX, // Unknown size by default + .ext_size_high = UINT32_MAX, + }; + + if (coder->uncomp_size_is_exact) + lzma_set_ext_size(options, coder->uncomp_size); + + // The properties are stored as bitwise-negation + // of the typical encoding. + if (lzma_lzma_lclppb_decode(&options, ~in[*in_pos])) + return LZMA_OPTIONS_ERROR; + + ++*in_pos; + + // Initialize the decoder. + lzma_filter_info filters[2] = { + { + .id = LZMA_FILTER_LZMA1EXT, + .init = &lzma_lzma_decoder_init, + .options = &options, + }, { + .init = NULL, + } + }; + + return_if_error(lzma_next_filter_init(&coder->lzma, + allocator, filters)); + + // Pass one dummy 0x00 byte to the LZMA decoder since that + // is what it expects the first byte to be. + const uint8_t dummy_in = 0; + size_t dummy_in_pos = 0; + if (coder->lzma.code(coder->lzma.coder, allocator, + &dummy_in, &dummy_in_pos, 1, + out, out_pos, out_size, LZMA_RUN) != LZMA_OK) + return LZMA_PROG_ERROR; + + assert(dummy_in_pos == 1); + coder->props_decoded = true; + } + + // The rest is normal LZMA decoding. + lzma_ret ret = coder->lzma.code(coder->lzma.coder, allocator, + in, in_pos, in_size, + out, out_pos, out_size, action); + + // Update the remaining compressed size. + assert(coder->comp_size >= *in_pos - in_start); + coder->comp_size -= *in_pos - in_start; + + if (coder->uncomp_size_is_exact) { + // After successful decompression of the complete stream + // the compressed size must match. + if (ret == LZMA_STREAM_END && coder->comp_size != 0) + ret = LZMA_DATA_ERROR; + } else { + // Update the amount of output remaining. + assert(coder->uncomp_size >= *out_pos - out_start); + coder->uncomp_size -= *out_pos - out_start; + + // - We must not get LZMA_STREAM_END because the stream + // shouldn't have EOPM. + // - We must use uncomp_size to determine when to + // return LZMA_STREAM_END. + if (ret == LZMA_STREAM_END) + ret = LZMA_DATA_ERROR; + else if (coder->uncomp_size == 0) + ret = LZMA_STREAM_END; + } + + return ret; +} + + +static void +microlzma_decoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_microlzma_coder *coder = coder_ptr; + lzma_next_end(&coder->lzma, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +microlzma_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + uint64_t comp_size, + uint64_t uncomp_size, bool uncomp_size_is_exact, + uint32_t dict_size) +{ + lzma_next_coder_init(µlzma_decoder_init, next, allocator); + + lzma_microlzma_coder *coder = next->coder; + + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_microlzma_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = µlzma_decode; + next->end = µlzma_decoder_end; + + coder->lzma = LZMA_NEXT_CODER_INIT; + } + + // The public API is uint64_t but the internal LZ decoder API uses + // lzma_vli. + if (uncomp_size > LZMA_VLI_MAX) + return LZMA_OPTIONS_ERROR; + + coder->comp_size = comp_size; + coder->uncomp_size = uncomp_size; + coder->uncomp_size_is_exact = uncomp_size_is_exact; + coder->dict_size = dict_size; + + coder->props_decoded = false; + + return LZMA_OK; +} + + +extern LZMA_API(lzma_ret) +lzma_microlzma_decoder(lzma_stream *strm, uint64_t comp_size, + uint64_t uncomp_size, lzma_bool uncomp_size_is_exact, + uint32_t dict_size) +{ + lzma_next_strm_init(microlzma_decoder_init, strm, comp_size, + uncomp_size, uncomp_size_is_exact, dict_size); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/microlzma_encoder.c b/src/liblzma/common/microlzma_encoder.c new file mode 100644 index 000000000000..d3ef0632ddb3 --- /dev/null +++ b/src/liblzma/common/microlzma_encoder.c @@ -0,0 +1,140 @@ +/////////////////////////////////////////////////////////////////////////////// +// +/// \file microlzma_encoder.c +/// \brief Encode into MicroLZMA format +// +// Author: Lasse Collin +// +// This file has been put into the public domain. +// You can do whatever you want with this file. +// +/////////////////////////////////////////////////////////////////////////////// + +#include "lzma_encoder.h" + + +typedef struct { + /// LZMA1 encoder + lzma_next_coder lzma; + + /// LZMA properties byte (lc/lp/pb) + uint8_t props; +} lzma_microlzma_coder; + + +static lzma_ret +microlzma_encode(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, uint8_t *restrict out, + size_t *restrict out_pos, size_t out_size, lzma_action action) +{ + lzma_microlzma_coder *coder = coder_ptr; + + // Remember *out_pos so that we can overwrite the first byte with + // the LZMA properties byte. + const size_t out_start = *out_pos; + + // Remember *in_pos so that we can set it based on how many + // uncompressed bytes were actually encoded. + const size_t in_start = *in_pos; + + // Set the output size limit based on the available output space. + // We know that the encoder supports set_out_limit() so + // LZMA_OPTIONS_ERROR isn't possible. LZMA_BUF_ERROR is possible + // but lzma_code() has an assertion to not allow it to be returned + // from here and I don't want to change that for now, so + // LZMA_BUF_ERROR becomes LZMA_PROG_ERROR. + uint64_t uncomp_size; + if (coder->lzma.set_out_limit(coder->lzma.coder, + &uncomp_size, out_size - *out_pos) != LZMA_OK) + return LZMA_PROG_ERROR; + + // set_out_limit fails if this isn't true. + assert(out_size - *out_pos >= 6); + + // Encode as much as possible. + const lzma_ret ret = coder->lzma.code(coder->lzma.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, action); + + if (ret != LZMA_STREAM_END) { + if (ret == LZMA_OK) { + assert(0); + return LZMA_PROG_ERROR; + } + + return ret; + } + + // The first output byte is bitwise-negation of the properties byte. + // We know that there is space for this byte because set_out_limit + // and the actual encoding succeeded. + out[out_start] = (uint8_t)(~coder->props); + + // The LZMA encoder likely read more input than it was able to encode. + // Set *in_pos based on uncomp_size. + assert(uncomp_size <= in_size - in_start); + *in_pos = in_start + (size_t)(uncomp_size); + + return ret; +} + + +static void +microlzma_encoder_end(void *coder_ptr, const lzma_allocator *allocator) +{ + lzma_microlzma_coder *coder = coder_ptr; + lzma_next_end(&coder->lzma, allocator); + lzma_free(coder, allocator); + return; +} + + +static lzma_ret +microlzma_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_options_lzma *options) +{ + lzma_next_coder_init(µlzma_encoder_init, next, allocator); + + lzma_microlzma_coder *coder = next->coder; + + if (coder == NULL) { + coder = lzma_alloc(sizeof(lzma_microlzma_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + next->code = µlzma_encode; + next->end = µlzma_encoder_end; + + coder->lzma = LZMA_NEXT_CODER_INIT; + } + + // Encode the properties byte. Bitwise-negation of it will be the + // first output byte. + return_if_error(lzma_lzma_lclppb_encode(options, &coder->props)); + + // Initialize the LZMA encoder. + const lzma_filter_info filters[2] = { + { + .id = LZMA_FILTER_LZMA1, + .init = &lzma_lzma_encoder_init, + .options = (void *)(options), + }, { + .init = NULL, + } + }; + + return lzma_next_filter_init(&coder->lzma, allocator, filters); +} + + +extern LZMA_API(lzma_ret) +lzma_microlzma_encoder(lzma_stream *strm, const lzma_options_lzma *options) +{ + lzma_next_strm_init(microlzma_encoder_init, strm, options); + + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; + +} diff --git a/src/liblzma/common/outqueue.c b/src/liblzma/common/outqueue.c index 2dc8a38d1be3..71e8648a294d 100644 --- a/src/liblzma/common/outqueue.c +++ b/src/liblzma/common/outqueue.c @@ -13,84 +13,121 @@ #include "outqueue.h" -/// This is to ease integer overflow checking: We may allocate up to -/// 2 * LZMA_THREADS_MAX buffers and we need some extra memory for other -/// data structures (that's the second /2). -#define BUF_SIZE_MAX (UINT64_MAX / LZMA_THREADS_MAX / 2 / 2) +/// Get the maximum number of buffers that may be allocated based +/// on the number of threads. For now this is twice the number of threads. +/// It's a compromise between RAM usage and keeping the worker threads busy +/// when buffers finish out of order. +#define GET_BUFS_LIMIT(threads) (2 * (threads)) -static lzma_ret -get_options(uint64_t *bufs_alloc_size, uint32_t *bufs_count, - uint64_t buf_size_max, uint32_t threads) +extern uint64_t +lzma_outq_memusage(uint64_t buf_size_max, uint32_t threads) { - if (threads > LZMA_THREADS_MAX || buf_size_max > BUF_SIZE_MAX) - return LZMA_OPTIONS_ERROR; - - // The number of buffers is twice the number of threads. - // This wastes RAM but keeps the threads busy when buffers - // finish out of order. + // This is to ease integer overflow checking: We may allocate up to + // GET_BUFS_LIMIT(LZMA_THREADS_MAX) buffers and we need some extra + // memory for other data structures too (that's the /2). // - // NOTE: If this is changed, update BUF_SIZE_MAX too. - *bufs_count = threads * 2; - *bufs_alloc_size = *bufs_count * buf_size_max; + // lzma_outq_prealloc_buf() will still accept bigger buffers than this. + const uint64_t limit + = UINT64_MAX / GET_BUFS_LIMIT(LZMA_THREADS_MAX) / 2; - return LZMA_OK; + if (threads > LZMA_THREADS_MAX || buf_size_max > limit) + return UINT64_MAX; + + return GET_BUFS_LIMIT(threads) + * lzma_outq_outbuf_memusage(buf_size_max); } -extern uint64_t -lzma_outq_memusage(uint64_t buf_size_max, uint32_t threads) +static void +move_head_to_cache(lzma_outq *outq, const lzma_allocator *allocator) { - uint64_t bufs_alloc_size; - uint32_t bufs_count; + assert(outq->head != NULL); + assert(outq->tail != NULL); + assert(outq->bufs_in_use > 0); - if (get_options(&bufs_alloc_size, &bufs_count, buf_size_max, threads) - != LZMA_OK) - return UINT64_MAX; + lzma_outbuf *buf = outq->head; + outq->head = buf->next; + if (outq->head == NULL) + outq->tail = NULL; - return sizeof(lzma_outq) + bufs_count * sizeof(lzma_outbuf) - + bufs_alloc_size; + if (outq->cache != NULL && outq->cache->allocated != buf->allocated) + lzma_outq_clear_cache(outq, allocator); + + buf->next = outq->cache; + outq->cache = buf; + + --outq->bufs_in_use; + outq->mem_in_use -= lzma_outq_outbuf_memusage(buf->allocated); + + return; +} + + +static void +free_one_cached_buffer(lzma_outq *outq, const lzma_allocator *allocator) +{ + assert(outq->cache != NULL); + + lzma_outbuf *buf = outq->cache; + outq->cache = buf->next; + + --outq->bufs_allocated; + outq->mem_allocated -= lzma_outq_outbuf_memusage(buf->allocated); + + lzma_free(buf, allocator); + return; +} + + +extern void +lzma_outq_clear_cache(lzma_outq *outq, const lzma_allocator *allocator) +{ + while (outq->cache != NULL) + free_one_cached_buffer(outq, allocator); + + return; +} + + +extern void +lzma_outq_clear_cache2(lzma_outq *outq, const lzma_allocator *allocator, + size_t keep_size) +{ + if (outq->cache == NULL) + return; + + // Free all but one. + while (outq->cache->next != NULL) + free_one_cached_buffer(outq, allocator); + + // Free the last one only if its size doesn't equal to keep_size. + if (outq->cache->allocated != keep_size) + free_one_cached_buffer(outq, allocator); + + return; } extern lzma_ret lzma_outq_init(lzma_outq *outq, const lzma_allocator *allocator, - uint64_t buf_size_max, uint32_t threads) + uint32_t threads) { - uint64_t bufs_alloc_size; - uint32_t bufs_count; - - // Set bufs_count and bufs_alloc_size. - return_if_error(get_options(&bufs_alloc_size, &bufs_count, - buf_size_max, threads)); - - // Allocate memory if needed. - if (outq->buf_size_max != buf_size_max - || outq->bufs_allocated != bufs_count) { - lzma_outq_end(outq, allocator); - -#if SIZE_MAX < UINT64_MAX - if (bufs_alloc_size > SIZE_MAX) - return LZMA_MEM_ERROR; -#endif - - outq->bufs = lzma_alloc(bufs_count * sizeof(lzma_outbuf), - allocator); - outq->bufs_mem = lzma_alloc((size_t)(bufs_alloc_size), - allocator); - - if (outq->bufs == NULL || outq->bufs_mem == NULL) { - lzma_outq_end(outq, allocator); - return LZMA_MEM_ERROR; - } - } + if (threads > LZMA_THREADS_MAX) + return LZMA_OPTIONS_ERROR; + + const uint32_t bufs_limit = GET_BUFS_LIMIT(threads); + + // Clear head/tail. + while (outq->head != NULL) + move_head_to_cache(outq, allocator); - // Initialize the rest of the main structure. Initialization of - // outq->bufs[] is done when they are actually needed. - outq->buf_size_max = (size_t)(buf_size_max); - outq->bufs_allocated = bufs_count; - outq->bufs_pos = 0; - outq->bufs_used = 0; + // If new buf_limit is lower than the old one, we may need to free + // a few cached buffers. + while (bufs_limit < outq->bufs_allocated) + free_one_cached_buffer(outq, allocator); + + outq->bufs_limit = bufs_limit; outq->read_pos = 0; return LZMA_OK; @@ -100,33 +137,81 @@ lzma_outq_init(lzma_outq *outq, const lzma_allocator *allocator, extern void lzma_outq_end(lzma_outq *outq, const lzma_allocator *allocator) { - lzma_free(outq->bufs, allocator); - outq->bufs = NULL; - - lzma_free(outq->bufs_mem, allocator); - outq->bufs_mem = NULL; + while (outq->head != NULL) + move_head_to_cache(outq, allocator); + lzma_outq_clear_cache(outq, allocator); return; } -extern lzma_outbuf * -lzma_outq_get_buf(lzma_outq *outq) +extern lzma_ret +lzma_outq_prealloc_buf(lzma_outq *outq, const lzma_allocator *allocator, + size_t size) { // Caller must have checked it with lzma_outq_has_buf(). - assert(outq->bufs_used < outq->bufs_allocated); + assert(outq->bufs_in_use < outq->bufs_limit); + + // If there already is appropriately-sized buffer in the cache, + // we need to do nothing. + if (outq->cache != NULL && outq->cache->allocated == size) + return LZMA_OK; + + if (size > SIZE_MAX - sizeof(lzma_outbuf)) + return LZMA_MEM_ERROR; - // Initialize the new buffer. - lzma_outbuf *buf = &outq->bufs[outq->bufs_pos]; - buf->buf = outq->bufs_mem + outq->bufs_pos * outq->buf_size_max; - buf->size = 0; + const size_t alloc_size = lzma_outq_outbuf_memusage(size); + + // The cache may have buffers but their size is wrong. + lzma_outq_clear_cache(outq, allocator); + + outq->cache = lzma_alloc(alloc_size, allocator); + if (outq->cache == NULL) + return LZMA_MEM_ERROR; + + outq->cache->next = NULL; + outq->cache->allocated = size; + + ++outq->bufs_allocated; + outq->mem_allocated += alloc_size; + + return LZMA_OK; +} + + +extern lzma_outbuf * +lzma_outq_get_buf(lzma_outq *outq, void *worker) +{ + // Caller must have used lzma_outq_prealloc_buf() to ensure these. + assert(outq->bufs_in_use < outq->bufs_limit); + assert(outq->bufs_in_use < outq->bufs_allocated); + assert(outq->cache != NULL); + + lzma_outbuf *buf = outq->cache; + outq->cache = buf->next; + buf->next = NULL; + + if (outq->tail != NULL) { + assert(outq->head != NULL); + outq->tail->next = buf; + } else { + assert(outq->head == NULL); + outq->head = buf; + } + + outq->tail = buf; + + buf->worker = worker; buf->finished = false; + buf->finish_ret = LZMA_STREAM_END; + buf->pos = 0; + buf->decoder_in_pos = 0; - // Update the queue state. - if (++outq->bufs_pos == outq->bufs_allocated) - outq->bufs_pos = 0; + buf->unpadded_size = 0; + buf->uncompressed_size = 0; - ++outq->bufs_used; + ++outq->bufs_in_use; + outq->mem_in_use += lzma_outq_outbuf_memusage(buf->allocated); return buf; } @@ -135,50 +220,68 @@ lzma_outq_get_buf(lzma_outq *outq) extern bool lzma_outq_is_readable(const lzma_outq *outq) { - uint32_t i = outq->bufs_pos - outq->bufs_used; - if (outq->bufs_pos < outq->bufs_used) - i += outq->bufs_allocated; + if (outq->head == NULL) + return false; - return outq->bufs[i].finished; + return outq->read_pos < outq->head->pos || outq->head->finished; } extern lzma_ret -lzma_outq_read(lzma_outq *restrict outq, uint8_t *restrict out, - size_t *restrict out_pos, size_t out_size, +lzma_outq_read(lzma_outq *restrict outq, + const lzma_allocator *restrict allocator, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size, lzma_vli *restrict unpadded_size, lzma_vli *restrict uncompressed_size) { // There must be at least one buffer from which to read. - if (outq->bufs_used == 0) + if (outq->bufs_in_use == 0) return LZMA_OK; // Get the buffer. - uint32_t i = outq->bufs_pos - outq->bufs_used; - if (outq->bufs_pos < outq->bufs_used) - i += outq->bufs_allocated; - - lzma_outbuf *buf = &outq->bufs[i]; - - // If it isn't finished yet, we cannot read from it. - if (!buf->finished) - return LZMA_OK; + lzma_outbuf *buf = outq->head; // Copy from the buffer to output. - lzma_bufcpy(buf->buf, &outq->read_pos, buf->size, + // + // FIXME? In threaded decoder it may be bad to do this copy while + // the mutex is being held. + lzma_bufcpy(buf->buf, &outq->read_pos, buf->pos, out, out_pos, out_size); // Return if we didn't get all the data from the buffer. - if (outq->read_pos < buf->size) + if (!buf->finished || outq->read_pos < buf->pos) return LZMA_OK; // The buffer was finished. Tell the caller its size information. - *unpadded_size = buf->unpadded_size; - *uncompressed_size = buf->uncompressed_size; + if (unpadded_size != NULL) + *unpadded_size = buf->unpadded_size; + + if (uncompressed_size != NULL) + *uncompressed_size = buf->uncompressed_size; + + // Remember the return value. + const lzma_ret finish_ret = buf->finish_ret; // Free this buffer for further use. - --outq->bufs_used; + move_head_to_cache(outq, allocator); outq->read_pos = 0; - return LZMA_STREAM_END; + return finish_ret; +} + + +extern void +lzma_outq_enable_partial_output(lzma_outq *outq, + void (*enable_partial_output)(void *worker)) +{ + if (outq->head != NULL && !outq->head->finished + && outq->head->worker != NULL) { + enable_partial_output(outq->head->worker); + + // Set it to NULL since calling it twice is pointless. + outq->head->worker = NULL; + } + + return; } diff --git a/src/liblzma/common/outqueue.h b/src/liblzma/common/outqueue.h index 079634de4587..596911e95ee1 100644 --- a/src/liblzma/common/outqueue.h +++ b/src/liblzma/common/outqueue.h @@ -14,16 +14,36 @@ /// Output buffer for a single thread -typedef struct { - /// Pointer to the output buffer of lzma_outq.buf_size_max bytes - uint8_t *buf; - - /// Amount of data written to buf - size_t size; +typedef struct lzma_outbuf_s lzma_outbuf; +struct lzma_outbuf_s { + /// Pointer to the next buffer. This is used for the cached buffers. + /// The worker thread must not modify this. + lzma_outbuf *next; + + /// This initialized by lzma_outq_get_buf() and + /// is used by lzma_outq_enable_partial_output(). + /// The worker thread must not modify this. + void *worker; + + /// Amount of memory allocated for buf[]. + /// The worker thread must not modify this. + size_t allocated; + + /// Writing position in the worker thread or, in other words, the + /// amount of finished data written to buf[] which can be copied out + /// + /// \note This is read by another thread and thus access + /// to this variable needs a mutex. + size_t pos; - /// Additional size information - lzma_vli unpadded_size; - lzma_vli uncompressed_size; + /// Decompression: Position in the input buffer in the worker thread + /// that matches the output "pos" above. This is used to detect if + /// more output might be possible from the worker thread: if it has + /// consumed all its input, then more output isn't possible. + /// + /// \note This is read by another thread and thus access + /// to this variable needs a mutex. + size_t decoder_in_pos; /// True when no more data will be written into this buffer. /// @@ -31,32 +51,55 @@ typedef struct { /// to this variable needs a mutex. bool finished; -} lzma_outbuf; + /// Return value for lzma_outq_read() when the last byte from + /// a finished buffer has been read. Defaults to LZMA_STREAM_END. + /// This must *not* be LZMA_OK. The idea is to allow a decoder to + /// pass an error code to the main thread, setting the code here + /// together with finished = true. + lzma_ret finish_ret; + + /// Additional size information. lzma_outq_read() may read these + /// when "finished" is true. + lzma_vli unpadded_size; + lzma_vli uncompressed_size; + + /// Buffer of "allocated" bytes + uint8_t buf[]; +}; typedef struct { - /// Array of buffers that are used cyclically. - lzma_outbuf *bufs; + /// Linked list of buffers in use. The next output byte will be + /// read from the head and buffers for the next thread will be + /// appended to the tail. tail->next is always NULL. + lzma_outbuf *head; + lzma_outbuf *tail; - /// Memory allocated for all the buffers - uint8_t *bufs_mem; + /// Number of bytes read from head->buf[] in lzma_outq_read() + size_t read_pos; - /// Amount of buffer space available in each buffer - size_t buf_size_max; + /// Linked list of allocated buffers that aren't currently used. + /// This way buffers of similar size can be reused and don't + /// need to be reallocated every time. For simplicity, all + /// cached buffers in the list have the same allocated size. + lzma_outbuf *cache; - /// Number of buffers allocated - uint32_t bufs_allocated; + /// Total amount of memory allocated for buffers + uint64_t mem_allocated; - /// Position in the bufs array. The next buffer to be taken - /// into use is bufs[bufs_pos]. - uint32_t bufs_pos; + /// Amount of memory used by the buffers that are in use in + /// the head...tail linked list. + uint64_t mem_in_use; - /// Number of buffers in use - uint32_t bufs_used; + /// Number of buffers in use in the head...tail list. If and only if + /// this is zero, the pointers head and tail above are NULL. + uint32_t bufs_in_use; - /// Position in the buffer in lzma_outq_read() - size_t read_pos; + /// Number of buffers allocated (in use + cached) + uint32_t bufs_allocated; + /// Maximum allowed number of allocated buffers + uint32_t bufs_limit; } lzma_outq; @@ -76,32 +119,60 @@ extern uint64_t lzma_outq_memusage(uint64_t buf_size_max, uint32_t threads); /// function knows that there are no previous /// allocations to free. /// \param allocator Pointer to allocator or NULL -/// \param buf_size_max Maximum amount of data that a single buffer -/// in the queue may need to store. /// \param threads Number of buffers that may be in use /// concurrently. Note that more than this number -/// of buffers will actually get allocated to +/// of buffers may actually get allocated to /// improve performance when buffers finish -/// out of order. +/// out of order. The actual maximum number of +/// allocated buffers is derived from the number +/// of threads. /// /// \return - LZMA_OK /// - LZMA_MEM_ERROR /// -extern lzma_ret lzma_outq_init( - lzma_outq *outq, const lzma_allocator *allocator, - uint64_t buf_size_max, uint32_t threads); +extern lzma_ret lzma_outq_init(lzma_outq *outq, + const lzma_allocator *allocator, uint32_t threads); /// \brief Free the memory associated with the output queue extern void lzma_outq_end(lzma_outq *outq, const lzma_allocator *allocator); +/// \brief Free all cached buffers that consume memory but aren't in use +extern void lzma_outq_clear_cache( + lzma_outq *outq, const lzma_allocator *allocator); + + +/// \brief Like lzma_outq_clear_cache() but might keep one buffer +/// +/// One buffer is not freed if its size is equal to keep_size. +/// This is useful if the caller knows that it will soon need a buffer of +/// keep_size bytes. This way it won't be freed and immediately reallocated. +extern void lzma_outq_clear_cache2( + lzma_outq *outq, const lzma_allocator *allocator, + size_t keep_size); + + +/// \brief Preallocate a new buffer into cache +/// +/// Splitting the buffer allocation into a separate function makes it +/// possible to ensure that way lzma_outq_get_buf() cannot fail. +/// If the preallocated buffer isn't actually used (for example, some +/// other error occurs), the caller has to do nothing as the buffer will +/// be used later or cleared from the cache when not needed. +/// +/// \return LZMA_OK on success, LZMA_MEM_ERROR if allocation fails +/// +extern lzma_ret lzma_outq_prealloc_buf( + lzma_outq *outq, const lzma_allocator *allocator, size_t size); + + /// \brief Get a new buffer /// -/// lzma_outq_has_buf() must be used to check that there is a buffer +/// lzma_outq_prealloc_buf() must be used to ensure that there is a buffer /// available before calling lzma_outq_get_buf(). /// -extern lzma_outbuf *lzma_outq_get_buf(lzma_outq *outq); +extern lzma_outbuf *lzma_outq_get_buf(lzma_outq *outq, void *worker); /// \brief Test if there is data ready to be read @@ -126,17 +197,32 @@ extern bool lzma_outq_is_readable(const lzma_outq *outq); /// \return - LZMA: All OK. Either no data was available or the buffer /// being read didn't become empty yet. /// - LZMA_STREAM_END: The buffer being read was finished. -/// *unpadded_size and *uncompressed_size were set. +/// *unpadded_size and *uncompressed_size were set if they +/// were not NULL. /// -/// \note This reads lzma_outbuf.finished variables and thus call -/// to this function needs to be protected with a mutex. +/// \note This reads lzma_outbuf.finished and .pos variables and thus +/// calls to this function need to be protected with a mutex. /// extern lzma_ret lzma_outq_read(lzma_outq *restrict outq, + const lzma_allocator *restrict allocator, uint8_t *restrict out, size_t *restrict out_pos, size_t out_size, lzma_vli *restrict unpadded_size, lzma_vli *restrict uncompressed_size); +/// \brief Enable partial output from a worker thread +/// +/// If the buffer at the head of the output queue isn't finished, +/// this will call enable_partial_output on the worker associated with +/// that output buffer. +/// +/// \note This reads a lzma_outbuf.finished variable and thus +/// calls to this function need to be protected with a mutex. +/// +extern void lzma_outq_enable_partial_output(lzma_outq *outq, + void (*enable_partial_output)(void *worker)); + + /// \brief Test if there is at least one buffer free /// /// This must be used before getting a new buffer with lzma_outq_get_buf(). @@ -144,7 +230,7 @@ extern lzma_ret lzma_outq_read(lzma_outq *restrict outq, static inline bool lzma_outq_has_buf(const lzma_outq *outq) { - return outq->bufs_used < outq->bufs_allocated; + return outq->bufs_in_use < outq->bufs_limit; } @@ -152,5 +238,17 @@ lzma_outq_has_buf(const lzma_outq *outq) static inline bool lzma_outq_is_empty(const lzma_outq *outq) { - return outq->bufs_used == 0; + return outq->bufs_in_use == 0; +} + + +/// \brief Get the amount of memory needed for a single lzma_outbuf +/// +/// \note Caller must check that the argument is significantly less +/// than SIZE_MAX to avoid an integer overflow! +static inline uint64_t +lzma_outq_outbuf_memusage(size_t buf_size) +{ + assert(buf_size <= SIZE_MAX - sizeof(lzma_outbuf)); + return sizeof(lzma_outbuf) + buf_size; } diff --git a/src/liblzma/common/stream_decoder.c b/src/liblzma/common/stream_decoder.c index 6885d0c7c8cf..dcf7c1499f7e 100644 --- a/src/liblzma/common/stream_decoder.c +++ b/src/liblzma/common/stream_decoder.c @@ -243,9 +243,7 @@ stream_decode(void *coder_ptr, const lzma_allocator *allocator, // Free the allocated filter options since they are needed // only to initialize the Block decoder. - for (size_t i = 0; i < LZMA_FILTERS_MAX; ++i) - lzma_free(filters[i].options, allocator); - + lzma_filters_free(filters, allocator); coder->block_options.filters = NULL; // Check if memory usage calculation and Block decoder diff --git a/src/liblzma/common/stream_decoder_mt.c b/src/liblzma/common/stream_decoder_mt.c new file mode 100644 index 000000000000..5733c764892d --- /dev/null +++ b/src/liblzma/common/stream_decoder_mt.c @@ -0,0 +1,2016 @@ +/////////////////////////////////////////////////////////////////////////////// +// +/// \file stream_decoder_mt.c +/// \brief Multithreaded .xz Stream decoder +// +// Authors: Sebastian Andrzej Siewior +// Lasse Collin +// +// This file has been put into the public domain. +// You can do whatever you want with this file. +// +/////////////////////////////////////////////////////////////////////////////// + +#include "common.h" +#include "block_decoder.h" +#include "stream_decoder.h" +#include "index.h" +#include "outqueue.h" + + +typedef enum { + /// Waiting for work. + /// Main thread may change this to THR_RUN or THR_EXIT. + THR_IDLE, + + /// Decoding is in progress. + /// Main thread may change this to THR_STOP or THR_EXIT. + /// The worker thread may change this to THR_IDLE. + THR_RUN, + + /// The main thread wants the thread to stop whatever it was doing + /// but not exit. Main thread may change this to THR_EXIT. + /// The worker thread may change this to THR_IDLE. + THR_STOP, + + /// The main thread wants the thread to exit. + THR_EXIT, + +} worker_state; + + +typedef enum { + /// Partial updates (storing of worker thread progress + /// to lzma_outbuf) are disabled. + PARTIAL_DISABLED, + + /// Main thread requests partial updates to be enabled but + /// no partial update has been done by the worker thread yet. + /// + /// Changing from PARTIAL_DISABLED to PARTIAL_START requires + /// use of the worker-thread mutex. Other transitions don't + /// need a mutex. + PARTIAL_START, + + /// Partial updates are enabled and the worker thread has done + /// at least one partial update. + PARTIAL_ENABLED, + +} partial_update_mode; + + +struct worker_thread { + /// Worker state is protected with our mutex. + worker_state state; + + /// Input buffer that will contain the whole Block except Block Header. + uint8_t *in; + + /// Amount of memory allocated for "in" + size_t in_size; + + /// Number of bytes written to "in" by the main thread + size_t in_filled; + + /// Number of bytes consumed from "in" by the worker thread. + size_t in_pos; + + /// Amount of uncompressed data that has been decoded. This local + /// copy is needed because updating outbuf->pos requires locking + /// the main mutex (coder->mutex). + size_t out_pos; + + /// Pointer to the main structure is needed to (1) lock the main + /// mutex (coder->mutex) when updating outbuf->pos and (2) when + /// putting this thread back to the stack of free threads. + struct lzma_stream_coder *coder; + + /// The allocator is set by the main thread. Since a copy of the + /// pointer is kept here, the application must not change the + /// allocator before calling lzma_end(). + const lzma_allocator *allocator; + + /// Output queue buffer to which the uncompressed data is written. + lzma_outbuf *outbuf; + + /// Amount of compressed data that has already been decompressed. + /// This is updated from in_pos when our mutex is locked. + /// This is size_t, not uint64_t, because per-thread progress + /// is limited to sizes of allocated buffers. + size_t progress_in; + + /// Like progress_in but for uncompressed data. + size_t progress_out; + + /// Updating outbuf->pos requires locking the main mutex + /// (coder->mutex). Since the main thread will only read output + /// from the oldest outbuf in the queue, only the worker thread + /// that is associated with the oldest outbuf needs to update its + /// outbuf->pos. This avoids useless mutex contention that would + /// happen if all worker threads were frequently locking the main + /// mutex to update their outbuf->pos. + /// + /// Only when partial_update is something else than PARTIAL_DISABLED, + /// this worker thread will update outbuf->pos after each call to + /// the Block decoder. + partial_update_mode partial_update; + + /// Block decoder + lzma_next_coder block_decoder; + + /// Thread-specific Block options are needed because the Block + /// decoder modifies the struct given to it at initialization. + lzma_block block_options; + + /// Filter chain memory usage + uint64_t mem_filters; + + /// Next structure in the stack of free worker threads. + struct worker_thread *next; + + mythread_mutex mutex; + mythread_cond cond; + + /// The ID of this thread is used to join the thread + /// when it's not needed anymore. + mythread thread_id; +}; + + +struct lzma_stream_coder { + enum { + SEQ_STREAM_HEADER, + SEQ_BLOCK_HEADER, + SEQ_BLOCK_INIT, + SEQ_BLOCK_THR_INIT, + SEQ_BLOCK_THR_RUN, + SEQ_BLOCK_DIRECT_INIT, + SEQ_BLOCK_DIRECT_RUN, + SEQ_INDEX_WAIT_OUTPUT, + SEQ_INDEX_DECODE, + SEQ_STREAM_FOOTER, + SEQ_STREAM_PADDING, + SEQ_ERROR, + } sequence; + + /// Block decoder + lzma_next_coder block_decoder; + + /// Every Block Header will be decoded into this structure. + /// This is also used to initialize a Block decoder when in + /// direct mode. In threaded mode, a thread-specific copy will + /// be made for decoder initialization because the Block decoder + /// will modify the structure given to it. + lzma_block block_options; + + /// Buffer to hold a filter chain for Block Header decoding and + /// initialization. These are freed after successful Block decoder + /// initialization or at stream_decoder_mt_end(). The thread-specific + /// copy of block_options won't hold a pointer to filters[] after + /// initialization. + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + + /// Stream Flags from Stream Header + lzma_stream_flags stream_flags; + + /// Index is hashed so that it can be compared to the sizes of Blocks + /// with O(1) memory usage. + lzma_index_hash *index_hash; + + + /// Maximum wait time if cannot use all the input and cannot + /// fill the output buffer. This is in milliseconds. + uint32_t timeout; + + + /// Error code from a worker thread. + /// + /// \note Use mutex. + lzma_ret thread_error; + + /// Error code to return after pending output has been copied out. If + /// set in read_output_and_wait(), this is a mirror of thread_error. + /// If set in stream_decode_mt() then it's, for example, error that + /// occurred when decoding Block Header. + lzma_ret pending_error; + + /// Number of threads that will be created at maximum. + uint32_t threads_max; + + /// Number of thread structures that have been initialized from + /// "threads", and thus the number of worker threads actually + /// created so far. + uint32_t threads_initialized; + + /// Array of allocated thread-specific structures. When no threads + /// are in use (direct mode) this is NULL. In threaded mode this + /// points to an array of threads_max number of worker_thread structs. + struct worker_thread *threads; + + /// Stack of free threads. When a thread finishes, it puts itself + /// back into this stack. This starts as empty because threads + /// are created only when actually needed. + /// + /// \note Use mutex. + struct worker_thread *threads_free; + + /// The most recent worker thread to which the main thread writes + /// the new input from the application. + struct worker_thread *thr; + + /// Output buffer queue for decompressed data from the worker threads + /// + /// \note Use mutex with operations that need it. + lzma_outq outq; + + mythread_mutex mutex; + mythread_cond cond; + + + /// Memory usage that will not be exceeded in multi-threaded mode. + /// Single-threaded mode can exceed this even by a large amount. + uint64_t memlimit_threading; + + /// Memory usage limit that should never be exceeded. + /// LZMA_MEMLIMIT_ERROR will be returned if decoding isn't possible + /// even in single-threaded mode without exceeding this limit. + uint64_t memlimit_stop; + + /// Amount of memory in use by the direct mode decoder + /// (coder->block_decoder). In threaded mode this is 0. + uint64_t mem_direct_mode; + + /// Amount of memory needed by the running worker threads. + /// This doesn't include the memory needed by the output buffer. + /// + /// \note Use mutex. + uint64_t mem_in_use; + + /// Amount of memory used by the idle (cached) threads. + /// + /// \note Use mutex. + uint64_t mem_cached; + + + /// Amount of memory needed for the filter chain of the next Block. + uint64_t mem_next_filters; + + /// Amount of memory needed for the thread-specific input buffer + /// for the next Block. + uint64_t mem_next_in; + + /// Amount of memory actually needed to decode the next Block + /// in threaded mode. This is + /// mem_next_filters + mem_next_in + memory needed for lzma_outbuf. + uint64_t mem_next_block; + + + /// Amount of compressed data in Stream Header + Blocks that have + /// already been finished. + /// + /// \note Use mutex. + uint64_t progress_in; + + /// Amount of uncompressed data in Blocks that have already + /// been finished. + /// + /// \note Use mutex. + uint64_t progress_out; + + + /// If true, LZMA_NO_CHECK is returned if the Stream has + /// no integrity check. + bool tell_no_check; + + /// If true, LZMA_UNSUPPORTED_CHECK is returned if the Stream has + /// an integrity check that isn't supported by this liblzma build. + bool tell_unsupported_check; + + /// If true, LZMA_GET_CHECK is returned after decoding Stream Header. + bool tell_any_check; + + /// If true, we will tell the Block decoder to skip calculating + /// and verifying the integrity check. + bool ignore_check; + + /// If true, we will decode concatenated Streams that possibly have + /// Stream Padding between or after them. LZMA_STREAM_END is returned + /// once the application isn't giving us any new input (LZMA_FINISH), + /// and we aren't in the middle of a Stream, and possible + /// Stream Padding is a multiple of four bytes. + bool concatenated; + + /// If true, we will return any errors immediately instead of first + /// producing all output before the location of the error. + bool fail_fast; + + + /// When decoding concatenated Streams, this is true as long as we + /// are decoding the first Stream. This is needed to avoid misleading + /// LZMA_FORMAT_ERROR in case the later Streams don't have valid magic + /// bytes. + bool first_stream; + + /// This is used to track if the previous call to stream_decode_mt() + /// had output space (*out_pos < out_size) and managed to fill the + /// output buffer (*out_pos == out_size). This may be set to true + /// in read_output_and_wait(). This is read and then reset to false + /// at the beginning of stream_decode_mt(). + /// + /// This is needed to support applications that call lzma_code() in + /// such a way that more input is provided only when lzma_code() + /// didn't fill the output buffer completely. Basically, this makes + /// it easier to convert such applications from single-threaded + /// decoder to multi-threaded decoder. + bool out_was_filled; + + /// Write position in buffer[] and position in Stream Padding + size_t pos; + + /// Buffer to hold Stream Header, Block Header, and Stream Footer. + /// Block Header has biggest maximum size. + uint8_t buffer[LZMA_BLOCK_HEADER_SIZE_MAX]; +}; + + +/// Enables updating of outbuf->pos. This is a callback function that is +/// used with lzma_outq_enable_partial_output(). +static void +worker_enable_partial_update(void *thr_ptr) +{ + struct worker_thread *thr = thr_ptr; + + mythread_sync(thr->mutex) { + thr->partial_update = PARTIAL_START; + mythread_cond_signal(&thr->cond); + } +} + + +/// Things do to at THR_STOP or when finishing a Block. +/// This is called with thr->mutex locked. +static void +worker_stop(struct worker_thread *thr) +{ + // Update memory usage counters. + thr->coder->mem_in_use -= thr->in_size; + thr->in_size = 0; // thr->in was freed above. + + thr->coder->mem_in_use -= thr->mem_filters; + thr->coder->mem_cached += thr->mem_filters; + + // Put this thread to the stack of free threads. + thr->next = thr->coder->threads_free; + thr->coder->threads_free = thr; + + mythread_cond_signal(&thr->coder->cond); + return; +} + + +static MYTHREAD_RET_TYPE +worker_decoder(void *thr_ptr) +{ + struct worker_thread *thr = thr_ptr; + size_t in_filled; + partial_update_mode partial_update; + lzma_ret ret; + +next_loop_lock: + + mythread_mutex_lock(&thr->mutex); +next_loop_unlocked: + + if (thr->state == THR_IDLE) { + mythread_cond_wait(&thr->cond, &thr->mutex); + goto next_loop_unlocked; + } + + if (thr->state == THR_EXIT) { + mythread_mutex_unlock(&thr->mutex); + + lzma_free(thr->in, thr->allocator); + lzma_next_end(&thr->block_decoder, thr->allocator); + + mythread_mutex_destroy(&thr->mutex); + mythread_cond_destroy(&thr->cond); + + return MYTHREAD_RET_VALUE; + } + + if (thr->state == THR_STOP) { + thr->state = THR_IDLE; + mythread_mutex_unlock(&thr->mutex); + + mythread_sync(thr->coder->mutex) { + worker_stop(thr); + } + + goto next_loop_lock; + } + + assert(thr->state == THR_RUN); + + // Update progress info for get_progress(). + thr->progress_in = thr->in_pos; + thr->progress_out = thr->out_pos; + + // If we don't have any new input, wait for a signal from the main + // thread except if partial output has just been enabled. In that + // case we will do one normal run so that the partial output info + // gets passed to the main thread. The call to block_decoder.code() + // is useless but harmless as it can occur only once per Block. + in_filled = thr->in_filled; + partial_update = thr->partial_update; + + if (in_filled == thr->in_pos && partial_update != PARTIAL_START) { + mythread_cond_wait(&thr->cond, &thr->mutex); + goto next_loop_unlocked; + } + + mythread_mutex_unlock(&thr->mutex); + + // Pass the input in small chunks to the Block decoder. + // This way we react reasonably fast if we are told to stop/exit, + // and (when partial update is enabled) we tell about our progress + // to the main thread frequently enough. + const size_t chunk_size = 16384; + if ((in_filled - thr->in_pos) > chunk_size) + in_filled = thr->in_pos + chunk_size; + + ret = thr->block_decoder.code( + thr->block_decoder.coder, thr->allocator, + thr->in, &thr->in_pos, in_filled, + thr->outbuf->buf, &thr->out_pos, + thr->outbuf->allocated, LZMA_RUN); + + if (ret == LZMA_OK) { + if (partial_update != PARTIAL_DISABLED) { + // The main thread uses thr->mutex to change from + // PARTIAL_DISABLED to PARTIAL_START. The main thread + // doesn't care about this variable after that so we + // can safely change it here to PARTIAL_ENABLED + // without a mutex. + thr->partial_update = PARTIAL_ENABLED; + + // The main thread is reading decompressed data + // from thr->outbuf. Tell the main thread about + // our progress. + // + // NOTE: It's possible that we consumed input without + // producing any new output so it's possible that + // only in_pos has changed. In case of PARTIAL_START + // it is possible that neither in_pos nor out_pos has + // changed. + mythread_sync(thr->coder->mutex) { + thr->outbuf->pos = thr->out_pos; + thr->outbuf->decoder_in_pos = thr->in_pos; + mythread_cond_signal(&thr->coder->cond); + } + } + + goto next_loop_lock; + } + + // Either we finished successfully (LZMA_STREAM_END) or an error + // occurred. Both cases are handled almost identically. The error + // case requires updating thr->coder->thread_error. + // + // The sizes are in the Block Header and the Block decoder + // checks that they match, thus we know these: + assert(ret != LZMA_STREAM_END || thr->in_pos == thr->in_size); + assert(ret != LZMA_STREAM_END + || thr->out_pos == thr->block_options.uncompressed_size); + + // Free the input buffer. Don't update in_size as we need + // it later to update thr->coder->mem_in_use. + lzma_free(thr->in, thr->allocator); + thr->in = NULL; + + mythread_sync(thr->mutex) { + if (thr->state != THR_EXIT) + thr->state = THR_IDLE; + } + + mythread_sync(thr->coder->mutex) { + // Move our progress info to the main thread. + thr->coder->progress_in += thr->in_pos; + thr->coder->progress_out += thr->out_pos; + thr->progress_in = 0; + thr->progress_out = 0; + + // Mark the outbuf as finished. + thr->outbuf->pos = thr->out_pos; + thr->outbuf->decoder_in_pos = thr->in_pos; + thr->outbuf->finished = true; + thr->outbuf->finish_ret = ret; + thr->outbuf = NULL; + + // If an error occurred, tell it to the main thread. + if (ret != LZMA_STREAM_END + && thr->coder->thread_error == LZMA_OK) + thr->coder->thread_error = ret; + + worker_stop(thr); + } + + goto next_loop_lock; +} + + +/// Tells the worker threads to exit and waits for them to terminate. +static void +threads_end(struct lzma_stream_coder *coder, const lzma_allocator *allocator) +{ + for (uint32_t i = 0; i < coder->threads_initialized; ++i) { + mythread_sync(coder->threads[i].mutex) { + coder->threads[i].state = THR_EXIT; + mythread_cond_signal(&coder->threads[i].cond); + } + } + + for (uint32_t i = 0; i < coder->threads_initialized; ++i) + mythread_join(coder->threads[i].thread_id); + + lzma_free(coder->threads, allocator); + coder->threads_initialized = 0; + coder->threads = NULL; + coder->threads_free = NULL; + + // The threads don't update these when they exit. Do it here. + coder->mem_in_use = 0; + coder->mem_cached = 0; + + return; +} + + +static void +threads_stop(struct lzma_stream_coder *coder) +{ + for (uint32_t i = 0; i < coder->threads_initialized; ++i) { + mythread_sync(coder->threads[i].mutex) { + // The state must be changed conditionally because + // THR_IDLE -> THR_STOP is not a valid state change. + if (coder->threads[i].state != THR_IDLE) { + coder->threads[i].state = THR_STOP; + mythread_cond_signal(&coder->threads[i].cond); + } + } + } + + return; +} + + +/// Initialize a new worker_thread structure and create a new thread. +static lzma_ret +initialize_new_thread(struct lzma_stream_coder *coder, + const lzma_allocator *allocator) +{ + // Allocate the coder->threads array if needed. It's done here instead + // of when initializing the decoder because we don't need this if we + // use the direct mode (we may even free coder->threads in the middle + // of the file if we switch from threaded to direct mode). + if (coder->threads == NULL) { + coder->threads = lzma_alloc( + coder->threads_max * sizeof(struct worker_thread), + allocator); + + if (coder->threads == NULL) + return LZMA_MEM_ERROR; + } + + // Pick a free structure. + assert(coder->threads_initialized < coder->threads_max); + struct worker_thread *thr + = &coder->threads[coder->threads_initialized]; + + if (mythread_mutex_init(&thr->mutex)) + goto error_mutex; + + if (mythread_cond_init(&thr->cond)) + goto error_cond; + + thr->state = THR_IDLE; + thr->in = NULL; + thr->in_size = 0; + thr->allocator = allocator; + thr->coder = coder; + thr->outbuf = NULL; + thr->block_decoder = LZMA_NEXT_CODER_INIT; + thr->mem_filters = 0; + + if (mythread_create(&thr->thread_id, worker_decoder, thr)) + goto error_thread; + + ++coder->threads_initialized; + coder->thr = thr; + + return LZMA_OK; + +error_thread: + mythread_cond_destroy(&thr->cond); + +error_cond: + mythread_mutex_destroy(&thr->mutex); + +error_mutex: + return LZMA_MEM_ERROR; +} + + +static lzma_ret +get_thread(struct lzma_stream_coder *coder, const lzma_allocator *allocator) +{ + // If there is a free structure on the stack, use it. + mythread_sync(coder->mutex) { + if (coder->threads_free != NULL) { + coder->thr = coder->threads_free; + coder->threads_free = coder->threads_free->next; + + // The thread is no longer in the cache so substract + // it from the cached memory usage. Don't add it + // to mem_in_use though; the caller will handle it + // since it knows how much memory it will actually + // use (the filter chain might change). + coder->mem_cached -= coder->thr->mem_filters; + } + } + + if (coder->thr == NULL) { + assert(coder->threads_initialized < coder->threads_max); + + // Initialize a new thread. + return_if_error(initialize_new_thread(coder, allocator)); + } + + coder->thr->in_filled = 0; + coder->thr->in_pos = 0; + coder->thr->out_pos = 0; + + coder->thr->progress_in = 0; + coder->thr->progress_out = 0; + + coder->thr->partial_update = PARTIAL_DISABLED; + + return LZMA_OK; +} + + +static lzma_ret +read_output_and_wait(struct lzma_stream_coder *coder, + const lzma_allocator *allocator, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size, + bool *input_is_possible, + bool waiting_allowed, + mythread_condtime *wait_abs, bool *has_blocked) +{ + lzma_ret ret = LZMA_OK; + + mythread_sync(coder->mutex) { + do { + // Get as much output from the queue as is possible + // without blocking. + const size_t out_start = *out_pos; + do { + ret = lzma_outq_read(&coder->outq, allocator, + out, out_pos, out_size, + NULL, NULL); + + // If a Block was finished, tell the worker + // thread of the next Block (if it is still + // running) to start telling the main thread + // when new output is available. + if (ret == LZMA_STREAM_END) + lzma_outq_enable_partial_output( + &coder->outq, + &worker_enable_partial_update); + + // Loop until a Block wasn't finished. + // It's important to loop around even if + // *out_pos == out_size because there could + // be an empty Block that will return + // LZMA_STREAM_END without needing any + // output space. + } while (ret == LZMA_STREAM_END); + + // Check if lzma_outq_read reported an error from + // the Block decoder. + if (ret != LZMA_OK) + break; + + // If the output buffer is now full but it wasn't full + // when this function was called, set out_was_filled. + // This way the next call to stream_decode_mt() knows + // that some output was produced and no output space + // remained in the previous call to stream_decode_mt(). + if (*out_pos == out_size && *out_pos != out_start) + coder->out_was_filled = true; + + // Check if any thread has indicated an error. + if (coder->thread_error != LZMA_OK) { + // If LZMA_FAIL_FAST was used, report errors + // from worker threads immediately. + if (coder->fail_fast) { + ret = coder->thread_error; + break; + } + + // Otherwise set pending_error. The value we + // set here will not actually get used other + // than working as a flag that an error has + // occurred. This is because in SEQ_ERROR + // all output before the error will be read + // first by calling this function, and once we + // reach the location of the (first) error the + // error code from the above lzma_outq_read() + // will be returned to the application. + // + // Use LZMA_PROG_ERROR since the value should + // never leak to the application. It's + // possible that pending_error has already + // been set but that doesn't matter: if we get + // here, pending_error only works as a flag. + coder->pending_error = LZMA_PROG_ERROR; + } + + // Check if decoding of the next Block can be started. + // The memusage of the active threads must be low + // enough, there must be a free buffer slot in the + // output queue, and there must be a free thread + // (that can be either created or an existing one + // reused). + // + // NOTE: This is checked after reading the output + // above because reading the output can free a slot in + // the output queue and also reduce active memusage. + // + // NOTE: If output queue is empty, then input will + // always be possible. + if (input_is_possible != NULL + && coder->memlimit_threading + - coder->mem_in_use + - coder->outq.mem_in_use + >= coder->mem_next_block + && lzma_outq_has_buf(&coder->outq) + && (coder->threads_initialized + < coder->threads_max + || coder->threads_free + != NULL)) { + *input_is_possible = true; + break; + } + + // If the caller doesn't want us to block, return now. + if (!waiting_allowed) + break; + + // This check is needed only when input_is_possible + // is NULL. We must return if we aren't waiting for + // input to become possible and there is no more + // output coming from the queue. + if (lzma_outq_is_empty(&coder->outq)) { + assert(input_is_possible == NULL); + break; + } + + // If there is more data available from the queue, + // our out buffer must be full and we need to return + // so that the application can provide more output + // space. + // + // NOTE: In general lzma_outq_is_readable() can return + // true also when there are no more bytes available. + // This can happen when a Block has finished without + // providing any new output. We know that this is not + // the case because in the beginning of this loop we + // tried to read as much as possible even when we had + // no output space left and the mutex has been locked + // all the time (so worker threads cannot have changed + // anything). Thus there must be actual pending output + // in the queue. + if (lzma_outq_is_readable(&coder->outq)) { + assert(*out_pos == out_size); + break; + } + + // If the application stops providing more input + // in the middle of a Block, there will eventually + // be one worker thread left that is stuck waiting for + // more input (that might never arrive) and a matching + // outbuf which the worker thread cannot finish due + // to lack of input. We must detect this situation, + // otherwise we would end up waiting indefinitely + // (if no timeout is in use) or keep returning + // LZMA_TIMED_OUT while making no progress. Thus, the + // application would never get LZMA_BUF_ERROR from + // lzma_code() which would tell the application that + // no more progress is possible. No LZMA_BUF_ERROR + // means that, for example, truncated .xz files could + // cause an infinite loop. + // + // A worker thread doing partial updates will + // store not only the output position in outbuf->pos + // but also the matching input position in + // outbuf->decoder_in_pos. Here we check if that + // input position matches the amount of input that + // the worker thread has been given (in_filled). + // If so, we must return and not wait as no more + // output will be coming without first getting more + // input to the worker thread. If the application + // keeps calling lzma_code() without providing more + // input, it will eventually get LZMA_BUF_ERROR. + // + // NOTE: We can read partial_update and in_filled + // without thr->mutex as only the main thread + // modifies these variables. decoder_in_pos requires + // coder->mutex which we are already holding. + if (coder->thr != NULL && coder->thr->partial_update + != PARTIAL_DISABLED) { + // There is exactly one outbuf in the queue. + assert(coder->thr->outbuf == coder->outq.head); + assert(coder->thr->outbuf == coder->outq.tail); + + if (coder->thr->outbuf->decoder_in_pos + == coder->thr->in_filled) + break; + } + + // Wait for input or output to become possible. + if (coder->timeout != 0) { + // See the comment in stream_encoder_mt.c + // about why mythread_condtime_set() is used + // like this. + // + // FIXME? + // In contrast to the encoder, this calls + // _condtime_set while the mutex is locked. + if (!*has_blocked) { + *has_blocked = true; + mythread_condtime_set(wait_abs, + &coder->cond, + coder->timeout); + } + + if (mythread_cond_timedwait(&coder->cond, + &coder->mutex, + wait_abs) != 0) { + ret = LZMA_TIMED_OUT; + break; + } + } else { + mythread_cond_wait(&coder->cond, + &coder->mutex); + } + } while (ret == LZMA_OK); + } + + // If we are returning an error, then the application cannot get + // more output from us and thus keeping the threads running is + // useless and waste of CPU time. + if (ret != LZMA_OK && ret != LZMA_TIMED_OUT) + threads_stop(coder); + + return ret; +} + + +static lzma_ret +decode_block_header(struct lzma_stream_coder *coder, + const lzma_allocator *allocator, const uint8_t *restrict in, + size_t *restrict in_pos, size_t in_size) +{ + if (*in_pos >= in_size) + return LZMA_OK; + + if (coder->pos == 0) { + // Detect if it's Index. + if (in[*in_pos] == 0x00) + return LZMA_INDEX_DETECTED; + + // Calculate the size of the Block Header. Note that + // Block Header decoder wants to see this byte too + // so don't advance *in_pos. + coder->block_options.header_size + = lzma_block_header_size_decode( + in[*in_pos]); + } + + // Copy the Block Header to the internal buffer. + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + coder->block_options.header_size); + + // Return if we didn't get the whole Block Header yet. + if (coder->pos < coder->block_options.header_size) + return LZMA_OK; + + coder->pos = 0; + + // Version 1 is needed to support the .ignore_check option. + coder->block_options.version = 1; + + // Block Header decoder will initialize all members of this array + // so we don't need to do it here. + coder->block_options.filters = coder->filters; + + // Decode the Block Header. + return_if_error(lzma_block_header_decode(&coder->block_options, + allocator, coder->buffer)); + + // If LZMA_IGNORE_CHECK was used, this flag needs to be set. + // It has to be set after lzma_block_header_decode() because + // it always resets this to false. + coder->block_options.ignore_check = coder->ignore_check; + + // coder->block_options is ready now. + return LZMA_STREAM_END; +} + + +/// Get the size of the Compressed Data + Block Padding + Check. +static size_t +comp_blk_size(const struct lzma_stream_coder *coder) +{ + return vli_ceil4(coder->block_options.compressed_size) + + lzma_check_size(coder->stream_flags.check); +} + + +/// Returns true if the size (compressed or uncompressed) is such that +/// threaded decompression cannot be used. Sizes that are too big compared +/// to SIZE_MAX must be rejected to avoid integer overflows and truncations +/// when lzma_vli is assigned to a size_t. +static bool +is_direct_mode_needed(lzma_vli size) +{ + return size == LZMA_VLI_UNKNOWN || size > SIZE_MAX / 3; +} + + +static lzma_ret +stream_decoder_reset(struct lzma_stream_coder *coder, + const lzma_allocator *allocator) +{ + // Initialize the Index hash used to verify the Index. + coder->index_hash = lzma_index_hash_init(coder->index_hash, allocator); + if (coder->index_hash == NULL) + return LZMA_MEM_ERROR; + + // Reset the rest of the variables. + coder->sequence = SEQ_STREAM_HEADER; + coder->pos = 0; + + return LZMA_OK; +} + + +static lzma_ret +stream_decode_mt(void *coder_ptr, const lzma_allocator *allocator, + const uint8_t *restrict in, size_t *restrict in_pos, + size_t in_size, + uint8_t *restrict out, size_t *restrict out_pos, + size_t out_size, lzma_action action) +{ + struct lzma_stream_coder *coder = coder_ptr; + + mythread_condtime wait_abs; + bool has_blocked = false; + + // Determine if in SEQ_BLOCK_HEADER and SEQ_BLOCK_THR_RUN we should + // tell read_output_and_wait() to wait until it can fill the output + // buffer (or a timeout occurs). Two conditions must be met: + // + // (1) If the caller provided no new input. The reason for this + // can be, for example, the end of the file or that there is + // a pause in the input stream and more input is available + // a little later. In this situation we should wait for output + // because otherwise we would end up in a busy-waiting loop where + // we make no progress and the application just calls us again + // without providing any new input. This would then result in + // LZMA_BUF_ERROR even though more output would be available + // once the worker threads decode more data. + // + // (2) Even if (1) is true, we will not wait if the previous call to + // this function managed to produce some output and the output + // buffer became full. This is for compatibility with applications + // that call lzma_code() in such a way that new input is provided + // only when the output buffer didn't become full. Without this + // trick such applications would have bad performance (bad + // parallelization due to decoder not getting input fast enough). + // + // NOTE: Such loops might require that timeout is disabled (0) + // if they assume that output-not-full implies that all input has + // been consumed. If and only if timeout is enabled, we may return + // when output isn't full *and* not all input has been consumed. + // + // However, if LZMA_FINISH is used, the above is ignored and we always + // wait (timeout can still cause us to return) because we know that + // we won't get any more input. This matters if the input file is + // truncated and we are doing single-shot decoding, that is, + // timeout = 0 and LZMA_FINISH is used on the first call to + // lzma_code() and the output buffer is known to be big enough + // to hold all uncompressed data: + // + // - If LZMA_FINISH wasn't handled specially, we could return + // LZMA_OK before providing all output that is possible with the + // truncated input. The rest would be available if lzma_code() was + // called again but then it's not single-shot decoding anymore. + // + // - By handling LZMA_FINISH specially here, the first call will + // produce all the output, matching the behavior of the + // single-threaded decoder. + // + // So it's a very specific corner case but also easy to avoid. Note + // that this special handling of LZMA_FINISH has no effect for + // single-shot decoding when the input file is valid (not truncated); + // premature LZMA_OK wouldn't be possible as long as timeout = 0. + const bool waiting_allowed = action == LZMA_FINISH + || (*in_pos == in_size && !coder->out_was_filled); + coder->out_was_filled = false; + + while (true) + switch (coder->sequence) { + case SEQ_STREAM_HEADER: { + // Copy the Stream Header to the internal buffer. + const size_t in_old = *in_pos; + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + LZMA_STREAM_HEADER_SIZE); + coder->progress_in += *in_pos - in_old; + + // Return if we didn't get the whole Stream Header yet. + if (coder->pos < LZMA_STREAM_HEADER_SIZE) + return LZMA_OK; + + coder->pos = 0; + + // Decode the Stream Header. + const lzma_ret ret = lzma_stream_header_decode( + &coder->stream_flags, coder->buffer); + if (ret != LZMA_OK) + return ret == LZMA_FORMAT_ERROR && !coder->first_stream + ? LZMA_DATA_ERROR : ret; + + // If we are decoding concatenated Streams, and the later + // Streams have invalid Header Magic Bytes, we give + // LZMA_DATA_ERROR instead of LZMA_FORMAT_ERROR. + coder->first_stream = false; + + // Copy the type of the Check so that Block Header and Block + // decoders see it. + coder->block_options.check = coder->stream_flags.check; + + // Even if we return LZMA_*_CHECK below, we want + // to continue from Block Header decoding. + coder->sequence = SEQ_BLOCK_HEADER; + + // Detect if there's no integrity check or if it is + // unsupported if those were requested by the application. + if (coder->tell_no_check && coder->stream_flags.check + == LZMA_CHECK_NONE) + return LZMA_NO_CHECK; + + if (coder->tell_unsupported_check + && !lzma_check_is_supported( + coder->stream_flags.check)) + return LZMA_UNSUPPORTED_CHECK; + + if (coder->tell_any_check) + return LZMA_GET_CHECK; + } + + // Fall through + + case SEQ_BLOCK_HEADER: { + const size_t in_old = *in_pos; + const lzma_ret ret = decode_block_header(coder, allocator, + in, in_pos, in_size); + coder->progress_in += *in_pos - in_old; + + if (ret == LZMA_OK) { + // We didn't decode the whole Block Header yet. + // + // Read output from the queue before returning. This + // is important because it is possible that the + // application doesn't have any new input available + // immediately. If we didn't try to copy output from + // the output queue here, lzma_code() could end up + // returning LZMA_BUF_ERROR even though queued output + // is available. + // + // If the lzma_code() call provided at least one input + // byte, only copy as much data from the output queue + // as is available immediately. This way the + // application will be able to provide more input + // without a delay. + // + // On the other hand, if lzma_code() was called with + // an empty input buffer(*), treat it specially: try + // to fill the output buffer even if it requires + // waiting for the worker threads to provide output + // (timeout, if specified, can still cause us to + // return). + // + // - This way the application will be able to get all + // data that can be decoded from the input provided + // so far. + // + // - We avoid both premature LZMA_BUF_ERROR and + // busy-waiting where the application repeatedly + // calls lzma_code() which immediately returns + // LZMA_OK without providing new data. + // + // - If the queue becomes empty, we won't wait + // anything and will return LZMA_OK immediately + // (coder->timeout is completely ignored). + // + // (*) See the comment at the beginning of this + // function how waiting_allowed is determined + // and why there is an exception to the rule + // of "called with an empty input buffer". + assert(*in_pos == in_size); + + // If LZMA_FINISH was used we know that we won't get + // more input, so the file must be truncated if we + // get here. If worker threads don't detect any + // errors, eventually there will be no more output + // while we keep returning LZMA_OK which gets + // converted to LZMA_BUF_ERROR in lzma_code(). + // + // If fail-fast is enabled then we will return + // immediately using LZMA_DATA_ERROR instead of + // LZMA_OK or LZMA_BUF_ERROR. Rationale for the + // error code: + // + // - Worker threads may have a large amount of + // not-yet-decoded input data and we don't + // know for sure if all data is valid. Bad + // data there would result in LZMA_DATA_ERROR + // when fail-fast isn't used. + // + // - Immediate LZMA_BUF_ERROR would be a bit weird + // considering the older liblzma code. lzma_code() + // even has an assertion to prevent coders from + // returning LZMA_BUF_ERROR directly. + // + // The downside of this is that with fail-fast apps + // cannot always distinguish between corrupt and + // truncated files. + if (action == LZMA_FINISH && coder->fail_fast) { + // We won't produce any more output. Stop + // the unfinished worker threads so they + // won't waste CPU time. + threads_stop(coder); + return LZMA_DATA_ERROR; + } + + // read_output_and_wait() will call threads_stop() + // if needed so with that we can use return_if_error. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, waiting_allowed, + &wait_abs, &has_blocked)); + + if (coder->pending_error != LZMA_OK) { + coder->sequence = SEQ_ERROR; + break; + } + + return LZMA_OK; + } + + if (ret == LZMA_INDEX_DETECTED) { + coder->sequence = SEQ_INDEX_WAIT_OUTPUT; + break; + } + + // See if an error occurred. + if (ret != LZMA_STREAM_END) { + // NOTE: Here and in all other places where + // pending_error is set, it may overwrite the value + // (LZMA_PROG_ERROR) set by read_output_and_wait(). + // That function might overwrite value set here too. + // These are fine because when read_output_and_wait() + // sets pending_error, it actually works as a flag + // variable only ("some error has occurred") and the + // actual value of pending_error is not used in + // SEQ_ERROR. In such cases SEQ_ERROR will eventually + // get the correct error code from the return value of + // a later read_output_and_wait() call. + coder->pending_error = ret; + coder->sequence = SEQ_ERROR; + break; + } + + // Calculate the memory usage of the filters / Block decoder. + coder->mem_next_filters = lzma_raw_decoder_memusage( + coder->filters); + + if (coder->mem_next_filters == UINT64_MAX) { + // One or more unknown Filter IDs. + coder->pending_error = LZMA_OPTIONS_ERROR; + coder->sequence = SEQ_ERROR; + break; + } + + coder->sequence = SEQ_BLOCK_INIT; + } + + // Fall through + + case SEQ_BLOCK_INIT: { + // Check if decoding is possible at all with the current + // memlimit_stop which we must never exceed. + // + // This needs to be the first thing in SEQ_BLOCK_INIT + // to make it possible to restart decoding after increasing + // memlimit_stop with lzma_memlimit_set(). + if (coder->mem_next_filters > coder->memlimit_stop) { + // Flush pending output before returning + // LZMA_MEMLIMIT_ERROR. If the application doesn't + // want to increase the limit, at least it will get + // all the output possible so far. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, true, &wait_abs, &has_blocked)); + + if (!lzma_outq_is_empty(&coder->outq)) + return LZMA_OK; + + return LZMA_MEMLIMIT_ERROR; + } + + // Check if the size information is available in Block Header. + // If it is, check if the sizes are small enough that we don't + // need to worry *too* much about integer overflows later in + // the code. If these conditions are not met, we must use the + // single-threaded direct mode. + if (is_direct_mode_needed(coder->block_options.compressed_size) + || is_direct_mode_needed( + coder->block_options.uncompressed_size)) { + coder->sequence = SEQ_BLOCK_DIRECT_INIT; + break; + } + + // Calculate the amount of memory needed for the input and + // output buffers in threaded mode. + // + // These cannot overflow because we already checked that + // the sizes are small enough using is_direct_mode_needed(). + coder->mem_next_in = comp_blk_size(coder); + const uint64_t mem_buffers = coder->mem_next_in + + lzma_outq_outbuf_memusage( + coder->block_options.uncompressed_size); + + // Add the amount needed by the filters. + // Avoid integer overflows. + if (UINT64_MAX - mem_buffers < coder->mem_next_filters) { + // Use direct mode if the memusage would overflow. + // This is a theoretical case that shouldn't happen + // in practice unless the input file is weird (broken + // or malicious). + coder->sequence = SEQ_BLOCK_DIRECT_INIT; + break; + } + + // Amount of memory needed to decode this Block in + // threaded mode: + coder->mem_next_block = coder->mem_next_filters + mem_buffers; + + // If this alone would exceed memlimit_threading, then we must + // use the single-threaded direct mode. + if (coder->mem_next_block > coder->memlimit_threading) { + coder->sequence = SEQ_BLOCK_DIRECT_INIT; + break; + } + + // Use the threaded mode. Free the direct mode decoder in + // case it has been initialized. + lzma_next_end(&coder->block_decoder, allocator); + coder->mem_direct_mode = 0; + + // Since we already know what the sizes are supposed to be, + // we can already add them to the Index hash. The Block + // decoder will verify the values while decoding. + const lzma_ret ret = lzma_index_hash_append(coder->index_hash, + lzma_block_unpadded_size( + &coder->block_options), + coder->block_options.uncompressed_size); + if (ret != LZMA_OK) { + coder->pending_error = ret; + coder->sequence = SEQ_ERROR; + break; + } + + coder->sequence = SEQ_BLOCK_THR_INIT; + } + + // Fall through + + case SEQ_BLOCK_THR_INIT: { + // We need to wait for a multiple conditions to become true + // until we can initialize the Block decoder and let a worker + // thread decode it: + // + // - Wait for the memory usage of the active threads to drop + // so that starting the decoding of this Block won't make + // us go over memlimit_threading. + // + // - Wait for at least one free output queue slot. + // + // - Wait for a free worker thread. + // + // While we wait, we must copy decompressed data to the out + // buffer and catch possible decoder errors. + // + // read_output_and_wait() does all the above. + bool block_can_start = false; + + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + &block_can_start, true, + &wait_abs, &has_blocked)); + + if (coder->pending_error != LZMA_OK) { + coder->sequence = SEQ_ERROR; + break; + } + + if (!block_can_start) { + // It's not a timeout because return_if_error handles + // it already. Output queue cannot be empty either + // because in that case block_can_start would have + // been true. Thus the output buffer must be full and + // the queue isn't empty. + assert(*out_pos == out_size); + assert(!lzma_outq_is_empty(&coder->outq)); + return LZMA_OK; + } + + // We know that we can start decoding this Block without + // exceeding memlimit_threading. However, to stay below + // memlimit_threading may require freeing some of the + // cached memory. + // + // Get a local copy of variables that require locking the + // mutex. It is fine if the worker threads modify the real + // values after we read these as those changes can only be + // towards more favorable conditions (less memory in use, + // more in cache). + uint64_t mem_in_use; + uint64_t mem_cached; + struct worker_thread *thr = NULL; // Init to silence warning. + + mythread_sync(coder->mutex) { + mem_in_use = coder->mem_in_use; + mem_cached = coder->mem_cached; + thr = coder->threads_free; + } + + // The maximum amount of memory that can be held by other + // threads and cached buffers while allowing us to start + // decoding the next Block. + const uint64_t mem_max = coder->memlimit_threading + - coder->mem_next_block; + + // If the existing allocations are so large that starting + // to decode this Block might exceed memlimit_threads, + // try to free memory from the output queue cache first. + // + // NOTE: This math assumes the worst case. It's possible + // that the limit wouldn't be exceeded if the existing cached + // allocations are reused. + if (mem_in_use + mem_cached + coder->outq.mem_allocated + > mem_max) { + // Clear the outq cache except leave one buffer in + // the cache if its size is correct. That way we + // don't free and almost immediately reallocate + // an identical buffer. + lzma_outq_clear_cache2(&coder->outq, allocator, + coder->block_options.uncompressed_size); + } + + // If there is at least one worker_thread in the cache and + // the existing allocations are so large that starting to + // decode this Block might exceed memlimit_threads, free + // memory by freeing cached Block decoders. + // + // NOTE: The comparison is different here than above. + // Here we don't care about cached buffers in outq anymore + // and only look at memory actually in use. This is because + // if there is something in outq cache, it's a single buffer + // that can be used as is. We ensured this in the above + // if-block. + uint64_t mem_freed = 0; + if (thr != NULL && mem_in_use + mem_cached + + coder->outq.mem_in_use > mem_max) { + // Don't free the first Block decoder if its memory + // usage isn't greater than what this Block will need. + // Typically the same filter chain is used for all + // Blocks so this way the allocations can be reused + // when get_thread() picks the first worker_thread + // from the cache. + if (thr->mem_filters <= coder->mem_next_filters) + thr = thr->next; + + while (thr != NULL) { + lzma_next_end(&thr->block_decoder, allocator); + mem_freed += thr->mem_filters; + thr->mem_filters = 0; + thr = thr->next; + } + } + + // Update the memory usage counters. Note that coder->mem_* + // may have changed since we read them so we must substract + // or add the changes. + mythread_sync(coder->mutex) { + coder->mem_cached -= mem_freed; + + // Memory needed for the filters and the input buffer. + // The output queue takes care of its own counter so + // we don't touch it here. + // + // NOTE: After this, coder->mem_in_use + + // coder->mem_cached might count the same thing twice. + // If so, this will get corrected in get_thread() when + // a worker_thread is picked from coder->free_threads + // and its memory usage is substracted from mem_cached. + coder->mem_in_use += coder->mem_next_in + + coder->mem_next_filters; + } + + // Allocate memory for the output buffer in the output queue. + lzma_ret ret = lzma_outq_prealloc_buf( + &coder->outq, allocator, + coder->block_options.uncompressed_size); + if (ret != LZMA_OK) { + threads_stop(coder); + return ret; + } + + // Set up coder->thr. + ret = get_thread(coder, allocator); + if (ret != LZMA_OK) { + threads_stop(coder); + return ret; + } + + // The new Block decoder memory usage is already counted in + // coder->mem_in_use. Store it in the thread too. + coder->thr->mem_filters = coder->mem_next_filters; + + // Initialize the Block decoder. + coder->thr->block_options = coder->block_options; + ret = lzma_block_decoder_init( + &coder->thr->block_decoder, allocator, + &coder->thr->block_options); + + // Free the allocated filter options since they are needed + // only to initialize the Block decoder. + lzma_filters_free(coder->filters, allocator); + coder->thr->block_options.filters = NULL; + + // Check if memory usage calculation and Block encoder + // initialization succeeded. + if (ret != LZMA_OK) { + coder->pending_error = ret; + coder->sequence = SEQ_ERROR; + break; + } + + // Allocate the input buffer. + coder->thr->in_size = coder->mem_next_in; + coder->thr->in = lzma_alloc(coder->thr->in_size, allocator); + if (coder->thr->in == NULL) { + threads_stop(coder); + return LZMA_MEM_ERROR; + } + + // Get the preallocated output buffer. + coder->thr->outbuf = lzma_outq_get_buf( + &coder->outq, coder->thr); + + // Start the decoder. + mythread_sync(coder->thr->mutex) { + assert(coder->thr->state == THR_IDLE); + coder->thr->state = THR_RUN; + mythread_cond_signal(&coder->thr->cond); + } + + // Enable output from the thread that holds the oldest output + // buffer in the output queue (if such a thread exists). + mythread_sync(coder->mutex) { + lzma_outq_enable_partial_output(&coder->outq, + &worker_enable_partial_update); + } + + coder->sequence = SEQ_BLOCK_THR_RUN; + } + + // Fall through + + case SEQ_BLOCK_THR_RUN: { + if (action == LZMA_FINISH && coder->fail_fast) { + // We know that we won't get more input and that + // the caller wants fail-fast behavior. If we see + // that we don't have enough input to finish this + // Block, return LZMA_DATA_ERROR immediately. + // See SEQ_BLOCK_HEADER for the error code rationale. + const size_t in_avail = in_size - *in_pos; + const size_t in_needed = coder->thr->in_size + - coder->thr->in_filled; + if (in_avail < in_needed) { + threads_stop(coder); + return LZMA_DATA_ERROR; + } + } + + // Copy input to the worker thread. + size_t cur_in_filled = coder->thr->in_filled; + lzma_bufcpy(in, in_pos, in_size, coder->thr->in, + &cur_in_filled, coder->thr->in_size); + + // Tell the thread how much we copied. + mythread_sync(coder->thr->mutex) { + coder->thr->in_filled = cur_in_filled; + + // NOTE: Most of the time we are copying input faster + // than the thread can decode so most of the time + // calling mythread_cond_signal() is useless but + // we cannot make it conditional because thr->in_pos + // is updated without a mutex. And the overhead should + // be very much negligible anyway. + mythread_cond_signal(&coder->thr->cond); + } + + // Read output from the output queue. Just like in + // SEQ_BLOCK_HEADER, we wait to fill the output buffer + // only if waiting_allowed was set to true in the beginning + // of this function (see the comment there). + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, waiting_allowed, + &wait_abs, &has_blocked)); + + if (coder->pending_error != LZMA_OK) { + coder->sequence = SEQ_ERROR; + break; + } + + // Return if the input didn't contain the whole Block. + if (coder->thr->in_filled < coder->thr->in_size) { + assert(*in_pos == in_size); + return LZMA_OK; + } + + // The whole Block has been copied to the thread-specific + // buffer. Continue from the next Block Header or Index. + coder->thr = NULL; + coder->sequence = SEQ_BLOCK_HEADER; + break; + } + + case SEQ_BLOCK_DIRECT_INIT: { + // Wait for the threads to finish and that all decoded data + // has been copied to the output. That is, wait until the + // output queue becomes empty. + // + // NOTE: No need to check for coder->pending_error as + // we aren't consuming any input until the queue is empty + // and if there is a pending error, read_output_and_wait() + // will eventually return it before the queue is empty. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, true, &wait_abs, &has_blocked)); + if (!lzma_outq_is_empty(&coder->outq)) + return LZMA_OK; + + // Free the cached output buffers. + lzma_outq_clear_cache(&coder->outq, allocator); + + // Get rid of the worker threads, including the coder->threads + // array. + threads_end(coder, allocator); + + // Initialize the Block decoder. + const lzma_ret ret = lzma_block_decoder_init( + &coder->block_decoder, allocator, + &coder->block_options); + + // Free the allocated filter options since they are needed + // only to initialize the Block decoder. + lzma_filters_free(coder->filters, allocator); + coder->block_options.filters = NULL; + + // Check if Block decoder initialization succeeded. + if (ret != LZMA_OK) + return ret; + + // Make the memory usage visible to _memconfig(). + coder->mem_direct_mode = coder->mem_next_filters; + + coder->sequence = SEQ_BLOCK_DIRECT_RUN; + } + + // Fall through + + case SEQ_BLOCK_DIRECT_RUN: { + const size_t in_old = *in_pos; + const size_t out_old = *out_pos; + const lzma_ret ret = coder->block_decoder.code( + coder->block_decoder.coder, allocator, + in, in_pos, in_size, out, out_pos, out_size, + action); + coder->progress_in += *in_pos - in_old; + coder->progress_out += *out_pos - out_old; + + if (ret != LZMA_STREAM_END) + return ret; + + // Block decoded successfully. Add the new size pair to + // the Index hash. + return_if_error(lzma_index_hash_append(coder->index_hash, + lzma_block_unpadded_size( + &coder->block_options), + coder->block_options.uncompressed_size)); + + coder->sequence = SEQ_BLOCK_HEADER; + break; + } + + case SEQ_INDEX_WAIT_OUTPUT: + // Flush the output from all worker threads so that we can + // decode the Index without thinking about threading. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, true, &wait_abs, &has_blocked)); + + if (!lzma_outq_is_empty(&coder->outq)) + return LZMA_OK; + + coder->sequence = SEQ_INDEX_DECODE; + + // Fall through + + case SEQ_INDEX_DECODE: { + // If we don't have any input, don't call + // lzma_index_hash_decode() since it would return + // LZMA_BUF_ERROR, which we must not do here. + if (*in_pos >= in_size) + return LZMA_OK; + + // Decode the Index and compare it to the hash calculated + // from the sizes of the Blocks (if any). + const size_t in_old = *in_pos; + const lzma_ret ret = lzma_index_hash_decode(coder->index_hash, + in, in_pos, in_size); + coder->progress_in += *in_pos - in_old; + if (ret != LZMA_STREAM_END) + return ret; + + coder->sequence = SEQ_STREAM_FOOTER; + } + + // Fall through + + case SEQ_STREAM_FOOTER: { + // Copy the Stream Footer to the internal buffer. + const size_t in_old = *in_pos; + lzma_bufcpy(in, in_pos, in_size, coder->buffer, &coder->pos, + LZMA_STREAM_HEADER_SIZE); + coder->progress_in += *in_pos - in_old; + + // Return if we didn't get the whole Stream Footer yet. + if (coder->pos < LZMA_STREAM_HEADER_SIZE) + return LZMA_OK; + + coder->pos = 0; + + // Decode the Stream Footer. The decoder gives + // LZMA_FORMAT_ERROR if the magic bytes don't match, + // so convert that return code to LZMA_DATA_ERROR. + lzma_stream_flags footer_flags; + const lzma_ret ret = lzma_stream_footer_decode( + &footer_flags, coder->buffer); + if (ret != LZMA_OK) + return ret == LZMA_FORMAT_ERROR + ? LZMA_DATA_ERROR : ret; + + // Check that Index Size stored in the Stream Footer matches + // the real size of the Index field. + if (lzma_index_hash_size(coder->index_hash) + != footer_flags.backward_size) + return LZMA_DATA_ERROR; + + // Compare that the Stream Flags fields are identical in + // both Stream Header and Stream Footer. + return_if_error(lzma_stream_flags_compare( + &coder->stream_flags, &footer_flags)); + + if (!coder->concatenated) + return LZMA_STREAM_END; + + coder->sequence = SEQ_STREAM_PADDING; + } + + // Fall through + + case SEQ_STREAM_PADDING: + assert(coder->concatenated); + + // Skip over possible Stream Padding. + while (true) { + if (*in_pos >= in_size) { + // Unless LZMA_FINISH was used, we cannot + // know if there's more input coming later. + if (action != LZMA_FINISH) + return LZMA_OK; + + // Stream Padding must be a multiple of + // four bytes. + return coder->pos == 0 + ? LZMA_STREAM_END + : LZMA_DATA_ERROR; + } + + // If the byte is not zero, it probably indicates + // beginning of a new Stream (or the file is corrupt). + if (in[*in_pos] != 0x00) + break; + + ++*in_pos; + ++coder->progress_in; + coder->pos = (coder->pos + 1) & 3; + } + + // Stream Padding must be a multiple of four bytes (empty + // Stream Padding is OK). + if (coder->pos != 0) { + ++*in_pos; + ++coder->progress_in; + return LZMA_DATA_ERROR; + } + + // Prepare to decode the next Stream. + return_if_error(stream_decoder_reset(coder, allocator)); + break; + + case SEQ_ERROR: + if (!coder->fail_fast) { + // Let the application get all data before the point + // where the error was detected. This matches the + // behavior of single-threaded use. + // + // FIXME? Some errors (LZMA_MEM_ERROR) don't get here, + // they are returned immediately. Thus in rare cases + // the output will be less than in the single-threaded + // mode. Maybe this doesn't matter much in practice. + return_if_error(read_output_and_wait(coder, allocator, + out, out_pos, out_size, + NULL, true, &wait_abs, &has_blocked)); + + // We get here only if the error happened in the main + // thread, for example, unsupported Block Header. + if (!lzma_outq_is_empty(&coder->outq)) + return LZMA_OK; + } + + // We only get here if no errors were detected by the worker + // threads. Errors from worker threads would have already been + // returned by the call to read_output_and_wait() above. + return coder->pending_error; + + default: + assert(0); + return LZMA_PROG_ERROR; + } + + // Never reached +} + + +static void +stream_decoder_mt_end(void *coder_ptr, const lzma_allocator *allocator) +{ + struct lzma_stream_coder *coder = coder_ptr; + + threads_end(coder, allocator); + lzma_outq_end(&coder->outq, allocator); + + lzma_next_end(&coder->block_decoder, allocator); + lzma_filters_free(coder->filters, allocator); + lzma_index_hash_end(coder->index_hash, allocator); + + lzma_free(coder, allocator); + return; +} + + +static lzma_check +stream_decoder_mt_get_check(const void *coder_ptr) +{ + const struct lzma_stream_coder *coder = coder_ptr; + return coder->stream_flags.check; +} + + +static lzma_ret +stream_decoder_mt_memconfig(void *coder_ptr, uint64_t *memusage, + uint64_t *old_memlimit, uint64_t new_memlimit) +{ + // NOTE: This function gets/sets memlimit_stop. For now, + // memlimit_threading cannot be modified after initialization. + // + // *memusage will include cached memory too. Excluding cached memory + // would be misleading and it wouldn't help the applications to + // know how much memory is actually needed to decompress the file + // because the higher the number of threads and the memlimits are + // the more memory the decoder may use. + // + // Setting a new limit includes the cached memory too and too low + // limits will be rejected. Alternative could be to free the cached + // memory immediately if that helps to bring the limit down but + // the current way is the simplest. It's unlikely that limit needs + // to be lowered in the middle of a file anyway; the typical reason + // to want a new limit is to increase after LZMA_MEMLIMIT_ERROR + // and even such use isn't common. + struct lzma_stream_coder *coder = coder_ptr; + + mythread_sync(coder->mutex) { + *memusage = coder->mem_direct_mode + + coder->mem_in_use + + coder->mem_cached + + coder->outq.mem_allocated; + } + + // If no filter chains are allocated, *memusage may be zero. + // Always return at least LZMA_MEMUSAGE_BASE. + if (*memusage < LZMA_MEMUSAGE_BASE) + *memusage = LZMA_MEMUSAGE_BASE; + + *old_memlimit = coder->memlimit_stop; + + if (new_memlimit != 0) { + if (new_memlimit < *memusage) + return LZMA_MEMLIMIT_ERROR; + + coder->memlimit_stop = new_memlimit; + } + + return LZMA_OK; +} + + +static void +stream_decoder_mt_get_progress(void *coder_ptr, + uint64_t *progress_in, uint64_t *progress_out) +{ + struct lzma_stream_coder *coder = coder_ptr; + + // Lock coder->mutex to prevent finishing threads from moving their + // progress info from the worker_thread structure to lzma_stream_coder. + mythread_sync(coder->mutex) { + *progress_in = coder->progress_in; + *progress_out = coder->progress_out; + + for (size_t i = 0; i < coder->threads_initialized; ++i) { + mythread_sync(coder->threads[i].mutex) { + *progress_in += coder->threads[i].progress_in; + *progress_out += coder->threads[i] + .progress_out; + } + } + } + + return; +} + + +static lzma_ret +stream_decoder_mt_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_mt *options) +{ + struct lzma_stream_coder *coder; + + if (options->threads == 0 || options->threads > LZMA_THREADS_MAX) + return LZMA_OPTIONS_ERROR; + + if (options->flags & ~LZMA_SUPPORTED_FLAGS) + return LZMA_OPTIONS_ERROR; + + lzma_next_coder_init(&stream_decoder_mt_init, next, allocator); + + coder = next->coder; + if (!coder) { + coder = lzma_alloc(sizeof(struct lzma_stream_coder), allocator); + if (coder == NULL) + return LZMA_MEM_ERROR; + + next->coder = coder; + + if (mythread_mutex_init(&coder->mutex)) { + lzma_free(coder, allocator); + return LZMA_MEM_ERROR; + } + + if (mythread_cond_init(&coder->cond)) { + mythread_mutex_destroy(&coder->mutex); + lzma_free(coder, allocator); + return LZMA_MEM_ERROR; + } + + next->code = &stream_decode_mt; + next->end = &stream_decoder_mt_end; + next->get_check = &stream_decoder_mt_get_check; + next->memconfig = &stream_decoder_mt_memconfig; + next->get_progress = &stream_decoder_mt_get_progress; + + coder->filters[0].id = LZMA_VLI_UNKNOWN; + memzero(&coder->outq, sizeof(coder->outq)); + + coder->block_decoder = LZMA_NEXT_CODER_INIT; + coder->mem_direct_mode = 0; + + coder->index_hash = NULL; + coder->threads = NULL; + coder->threads_free = NULL; + coder->threads_initialized = 0; + } + + // Cleanup old filter chain if one remains after unfinished decoding + // of a previous Stream. + lzma_filters_free(coder->filters, allocator); + + // By allocating threads from scratch we can start memory-usage + // accounting from scratch, too. Changes in filter and block sizes may + // affect number of threads. + // + // FIXME? Reusing should be easy but unlike the single-threaded + // decoder, with some types of input file combinations reusing + // could leave quite a lot of memory allocated but unused (first + // file could allocate a lot, the next files could use fewer + // threads and some of the allocations from the first file would not + // get freed unless memlimit_threading forces us to clear caches). + // + // NOTE: The direct mode decoder isn't freed here if one exists. + // It will be reused or freed as needed in the main loop. + threads_end(coder, allocator); + + // All memusage counters start at 0 (including mem_direct_mode). + // The little extra that is needed for the structs in this file + // get accounted well enough by the filter chain memory usage + // which adds LZMA_MEMUSAGE_BASE for each chain. However, + // stream_decoder_mt_memconfig() has to handle this specially so that + // it will never return less than LZMA_MEMUSAGE_BASE as memory usage. + coder->mem_in_use = 0; + coder->mem_cached = 0; + coder->mem_next_block = 0; + + coder->progress_in = 0; + coder->progress_out = 0; + + coder->sequence = SEQ_STREAM_HEADER; + coder->thread_error = LZMA_OK; + coder->pending_error = LZMA_OK; + coder->thr = NULL; + + coder->timeout = options->timeout; + + coder->memlimit_threading = my_max(1, options->memlimit_threading); + coder->memlimit_stop = my_max(1, options->memlimit_stop); + if (coder->memlimit_threading > coder->memlimit_stop) + coder->memlimit_threading = coder->memlimit_stop; + + coder->tell_no_check = (options->flags & LZMA_TELL_NO_CHECK) != 0; + coder->tell_unsupported_check + = (options->flags & LZMA_TELL_UNSUPPORTED_CHECK) != 0; + coder->tell_any_check = (options->flags & LZMA_TELL_ANY_CHECK) != 0; + coder->ignore_check = (options->flags & LZMA_IGNORE_CHECK) != 0; + coder->concatenated = (options->flags & LZMA_CONCATENATED) != 0; + coder->fail_fast = (options->flags & LZMA_FAIL_FAST) != 0; + + coder->first_stream = true; + coder->out_was_filled = false; + coder->pos = 0; + + coder->threads_max = options->threads; + + return_if_error(lzma_outq_init(&coder->outq, allocator, + coder->threads_max)); + + return stream_decoder_reset(coder, allocator); +} + + +extern LZMA_API(lzma_ret) +lzma_stream_decoder_mt(lzma_stream *strm, const lzma_mt *options) +{ + lzma_next_strm_init(stream_decoder_mt_init, strm, options); + + strm->internal->supported_actions[LZMA_RUN] = true; + strm->internal->supported_actions[LZMA_FINISH] = true; + + return LZMA_OK; +} diff --git a/src/liblzma/common/stream_encoder.c b/src/liblzma/common/stream_encoder.c index b15229c39ec4..ee9204601856 100644 --- a/src/liblzma/common/stream_encoder.c +++ b/src/liblzma/common/stream_encoder.c @@ -219,8 +219,7 @@ stream_encoder_end(void *coder_ptr, const lzma_allocator *allocator) lzma_next_end(&coder->index_encoder, allocator); lzma_index_end(coder->index, allocator); - for (size_t i = 0; coder->filters[i].id != LZMA_VLI_UNKNOWN; ++i) - lzma_free(coder->filters[i].options, allocator); + lzma_filters_free(coder->filters, allocator); lzma_free(coder, allocator); return; @@ -271,22 +270,15 @@ stream_encoder_update(void *coder_ptr, const lzma_allocator *allocator, } // Free the options of the old chain. - for (size_t i = 0; coder->filters[i].id != LZMA_VLI_UNKNOWN; ++i) - lzma_free(coder->filters[i].options, allocator); + lzma_filters_free(coder->filters, allocator); // Copy the new filter chain in place. - size_t j = 0; - do { - coder->filters[j].id = temp[j].id; - coder->filters[j].options = temp[j].options; - } while (temp[j++].id != LZMA_VLI_UNKNOWN); + memcpy(coder->filters, temp, sizeof(temp)); return LZMA_OK; error: - for (size_t i = 0; temp[i].id != LZMA_VLI_UNKNOWN; ++i) - lzma_free(temp[i].options, allocator); - + lzma_filters_free(temp, allocator); return ret; } diff --git a/src/liblzma/common/stream_encoder_mt.c b/src/liblzma/common/stream_encoder_mt.c index 6dc10c3a095d..f4497c10b9b3 100644 --- a/src/liblzma/common/stream_encoder_mt.c +++ b/src/liblzma/common/stream_encoder_mt.c @@ -85,6 +85,11 @@ struct worker_thread_s { /// Compression options for this Block lzma_block block_options; + /// Filter chain for this thread. By copying the filters array + /// to each thread it is possible to change the filter chain + /// between Blocks using lzma_filters_update(). + lzma_filter filters[LZMA_FILTERS_MAX + 1]; + /// Next structure in the stack of free worker threads. worker_thread *next; @@ -109,9 +114,22 @@ struct lzma_stream_coder_s { /// LZMA_FULL_FLUSH or LZMA_FULL_BARRIER is used earlier. size_t block_size; - /// The filter chain currently in use + /// The filter chain to use for the next Block. + /// This can be updated using lzma_filters_update() + /// after LZMA_FULL_BARRIER or LZMA_FULL_FLUSH. lzma_filter filters[LZMA_FILTERS_MAX + 1]; + /// A copy of filters[] will be put here when attempting to get + /// a new worker thread. This will be copied to a worker thread + /// when a thread becomes free and then this cache is marked as + /// empty by setting [0].id = LZMA_VLI_UNKNOWN. Without this cache + /// the filter options from filters[] would get uselessly copied + /// multiple times (allocated and freed) when waiting for a new free + /// worker thread. + /// + /// This is freed if filters[] is updated via lzma_filters_update(). + lzma_filter filters_cache[LZMA_FILTERS_MAX + 1]; + /// Index to hold sizes of the Blocks lzma_index *index; @@ -133,6 +151,9 @@ struct lzma_stream_coder_s { /// Output buffer queue for compressed data lzma_outq outq; + /// How much memory to allocate for each lzma_outbuf.buf + size_t outbuf_alloc_size; + /// Maximum wait time if cannot use all the input and cannot /// fill the output buffer. This is in milliseconds. @@ -196,7 +217,7 @@ worker_error(worker_thread *thr, lzma_ret ret) static worker_state -worker_encode(worker_thread *thr, worker_state state) +worker_encode(worker_thread *thr, size_t *out_pos, worker_state state) { assert(thr->progress_in == 0); assert(thr->progress_out == 0); @@ -205,12 +226,9 @@ worker_encode(worker_thread *thr, worker_state state) thr->block_options = (lzma_block){ .version = 0, .check = thr->coder->stream_flags.check, - .compressed_size = thr->coder->outq.buf_size_max, + .compressed_size = thr->outbuf->allocated, .uncompressed_size = thr->coder->block_size, - - // TODO: To allow changing the filter chain, the filters - // array must be copied to each worker_thread. - .filters = thr->coder->filters, + .filters = thr->filters, }; // Calculate maximum size of the Block Header. This amount is @@ -234,12 +252,12 @@ worker_encode(worker_thread *thr, worker_state state) size_t in_pos = 0; size_t in_size = 0; - thr->outbuf->size = thr->block_options.header_size; - const size_t out_size = thr->coder->outq.buf_size_max; + *out_pos = thr->block_options.header_size; + const size_t out_size = thr->outbuf->allocated; do { mythread_sync(thr->mutex) { - // Store in_pos and out_pos into *thr so that + // Store in_pos and *out_pos into *thr so that // an application may read them via // lzma_get_progress() to get progress information. // @@ -247,7 +265,7 @@ worker_encode(worker_thread *thr, worker_state state) // finishes. Instead, the final values are taken // later from thr->outbuf. thr->progress_in = in_pos; - thr->progress_out = thr->outbuf->size; + thr->progress_out = *out_pos; while (in_size == thr->in_size && thr->state == THR_RUN) @@ -277,8 +295,8 @@ worker_encode(worker_thread *thr, worker_state state) ret = thr->block_encoder.code( thr->block_encoder.coder, thr->allocator, thr->in, &in_pos, in_limit, thr->outbuf->buf, - &thr->outbuf->size, out_size, action); - } while (ret == LZMA_OK && thr->outbuf->size < out_size); + out_pos, out_size, action); + } while (ret == LZMA_OK && *out_pos < out_size); switch (ret) { case LZMA_STREAM_END: @@ -313,10 +331,10 @@ worker_encode(worker_thread *thr, worker_state state) return state; // Do the encoding. This takes care of the Block Header too. - thr->outbuf->size = 0; + *out_pos = 0; ret = lzma_block_uncomp_encode(&thr->block_options, thr->in, in_size, thr->outbuf->buf, - &thr->outbuf->size, out_size); + out_pos, out_size); // It shouldn't fail. if (ret != LZMA_OK) { @@ -367,11 +385,13 @@ worker_start(void *thr_ptr) } } + size_t out_pos = 0; + assert(state != THR_IDLE); assert(state != THR_STOP); if (state <= THR_FINISH) - state = worker_encode(thr, state); + state = worker_encode(thr, &out_pos, state); if (state == THR_EXIT) break; @@ -387,14 +407,17 @@ worker_start(void *thr_ptr) } mythread_sync(thr->coder->mutex) { - // Mark the output buffer as finished if - // no errors occurred. - thr->outbuf->finished = state == THR_FINISH; + // If no errors occurred, make the encoded data + // available to be copied out. + if (state == THR_FINISH) { + thr->outbuf->pos = out_pos; + thr->outbuf->finished = true; + } // Update the main progress info. thr->coder->progress_in += thr->outbuf->uncompressed_size; - thr->coder->progress_out += thr->outbuf->size; + thr->coder->progress_out += out_pos; thr->progress_in = 0; thr->progress_out = 0; @@ -407,6 +430,8 @@ worker_start(void *thr_ptr) } // Exiting, free the resources. + lzma_filters_free(thr->filters, thr->allocator); + mythread_mutex_destroy(&thr->mutex); mythread_cond_destroy(&thr->cond); @@ -490,6 +515,7 @@ initialize_new_thread(lzma_stream_coder *coder, thr->progress_in = 0; thr->progress_out = 0; thr->block_encoder = LZMA_NEXT_CODER_INIT; + thr->filters[0].id = LZMA_VLI_UNKNOWN; if (mythread_create(&thr->thread_id, &worker_start, thr)) goto error_thread; @@ -519,6 +545,18 @@ get_thread(lzma_stream_coder *coder, const lzma_allocator *allocator) if (!lzma_outq_has_buf(&coder->outq)) return LZMA_OK; + // That's also true if we cannot allocate memory for the output + // buffer in the output queue. + return_if_error(lzma_outq_prealloc_buf(&coder->outq, allocator, + coder->outbuf_alloc_size)); + + // Make a thread-specific copy of the filter chain. Put it in + // the cache array first so that if we cannot get a new thread yet, + // the allocation is ready when we try again. + if (coder->filters_cache[0].id == LZMA_VLI_UNKNOWN) + return_if_error(lzma_filters_copy( + coder->filters, coder->filters_cache, allocator)); + // If there is a free structure on the stack, use it. mythread_sync(coder->mutex) { if (coder->threads_free != NULL) { @@ -541,7 +579,16 @@ get_thread(lzma_stream_coder *coder, const lzma_allocator *allocator) mythread_sync(coder->thr->mutex) { coder->thr->state = THR_RUN; coder->thr->in_size = 0; - coder->thr->outbuf = lzma_outq_get_buf(&coder->outq); + coder->thr->outbuf = lzma_outq_get_buf(&coder->outq, NULL); + + // Free the old thread-specific filter options and replace + // them with the already-allocated new options from + // coder->filters_cache[]. Then mark the cache as empty. + lzma_filters_free(coder->thr->filters, allocator); + memcpy(coder->thr->filters, coder->filters_cache, + sizeof(coder->filters_cache)); + coder->filters_cache[0].id = LZMA_VLI_UNKNOWN; + mythread_cond_signal(&coder->thr->cond); } @@ -627,9 +674,13 @@ wait_for_work(lzma_stream_coder *coder, mythread_condtime *wait_abs, // to true here and calculate the absolute time when // we must return if there's nothing to do. // - // The idea of *has_blocked is to avoid unneeded calls - // to mythread_condtime_set(), which may do a syscall - // depending on the operating system. + // This way if we block multiple times for short moments + // less than "timeout" milliseconds, we will return once + // "timeout" amount of time has passed since the *first* + // blocking occurred. If the absolute time was calculated + // again every time we block, "timeout" would effectively + // be meaningless if we never consecutively block longer + // than "timeout" ms. *has_blocked = true; mythread_condtime_set(wait_abs, &coder->cond, coder->timeout); } @@ -704,7 +755,7 @@ stream_encode_mt(void *coder_ptr, const lzma_allocator *allocator, } // Try to read compressed data to out[]. - ret = lzma_outq_read(&coder->outq, + ret = lzma_outq_read(&coder->outq, allocator, out, out_pos, out_size, &unpadded_size, &uncompressed_size); @@ -849,8 +900,8 @@ stream_encoder_mt_end(void *coder_ptr, const lzma_allocator *allocator) threads_end(coder, allocator); lzma_outq_end(&coder->outq, allocator); - for (size_t i = 0; coder->filters[i].id != LZMA_VLI_UNKNOWN; ++i) - lzma_free(coder->filters[i].options, allocator); + lzma_filters_free(coder->filters, allocator); + lzma_filters_free(coder->filters_cache, allocator); lzma_next_end(&coder->index_encoder, allocator); lzma_index_end(coder->index, allocator); @@ -863,6 +914,45 @@ stream_encoder_mt_end(void *coder_ptr, const lzma_allocator *allocator) } +static lzma_ret +stream_encoder_mt_update(void *coder_ptr, const lzma_allocator *allocator, + const lzma_filter *filters, + const lzma_filter *reversed_filters + lzma_attribute((__unused__))) +{ + lzma_stream_coder *coder = coder_ptr; + + // Applications shouldn't attempt to change the options when + // we are already encoding the Index or Stream Footer. + if (coder->sequence > SEQ_BLOCK) + return LZMA_PROG_ERROR; + + // For now the threaded encoder doesn't support changing + // the options in the middle of a Block. + if (coder->thr != NULL) + return LZMA_PROG_ERROR; + + // Check if the filter chain seems mostly valid. See the comment + // in stream_encoder_mt_init(). + if (lzma_raw_encoder_memusage(filters) == UINT64_MAX) + return LZMA_OPTIONS_ERROR; + + // Make a copy to a temporary buffer first. This way the encoder + // state stays unchanged if an error occurs in lzma_filters_copy(). + lzma_filter temp[LZMA_FILTERS_MAX + 1]; + return_if_error(lzma_filters_copy(filters, temp, allocator)); + + // Free the options of the old chain as well as the cache. + lzma_filters_free(coder->filters, allocator); + lzma_filters_free(coder->filters_cache, allocator); + + // Copy the new filter chain in place. + memcpy(coder->filters, temp, sizeof(temp)); + + return LZMA_OK; +} + + /// Options handling for lzma_stream_encoder_mt_init() and /// lzma_stream_encoder_mt_memusage() static lzma_ret @@ -954,14 +1044,16 @@ stream_encoder_mt_init(lzma_next_coder *next, const lzma_allocator *allocator, &block_size, &outbuf_size_max)); #if SIZE_MAX < UINT64_MAX - if (block_size > SIZE_MAX) + if (block_size > SIZE_MAX || outbuf_size_max > SIZE_MAX) return LZMA_MEM_ERROR; #endif // Validate the filter chain so that we can give an error in this // function instead of delaying it to the first call to lzma_code(). // The memory usage calculation verifies the filter chain as - // a side effect so we take advantage of that. + // a side effect so we take advantage of that. It's not a perfect + // check though as raw encoder allows LZMA1 too but such problems + // will be caught eventually with Block Header encoder. if (lzma_raw_encoder_memusage(filters) == UINT64_MAX) return LZMA_OPTIONS_ERROR; @@ -1001,9 +1093,10 @@ stream_encoder_mt_init(lzma_next_coder *next, const lzma_allocator *allocator, next->code = &stream_encode_mt; next->end = &stream_encoder_mt_end; next->get_progress = &get_progress; -// next->update = &stream_encoder_mt_update; + next->update = &stream_encoder_mt_update; coder->filters[0].id = LZMA_VLI_UNKNOWN; + coder->filters_cache[0].id = LZMA_VLI_UNKNOWN; coder->index_encoder = LZMA_NEXT_CODER_INIT; coder->index = NULL; memzero(&coder->outq, sizeof(coder->outq)); @@ -1015,6 +1108,7 @@ stream_encoder_mt_init(lzma_next_coder *next, const lzma_allocator *allocator, // Basic initializations coder->sequence = SEQ_STREAM_HEADER; coder->block_size = (size_t)(block_size); + coder->outbuf_alloc_size = (size_t)(outbuf_size_max); coder->thread_error = LZMA_OK; coder->thr = NULL; @@ -1044,19 +1138,16 @@ stream_encoder_mt_init(lzma_next_coder *next, const lzma_allocator *allocator, // Output queue return_if_error(lzma_outq_init(&coder->outq, allocator, - outbuf_size_max, options->threads)); + options->threads)); // Timeout coder->timeout = options->timeout; - // Free the old filter chain and copy the new one. - for (size_t i = 0; coder->filters[i].id != LZMA_VLI_UNKNOWN; ++i) - lzma_free(coder->filters[i].options, allocator); - - // Mark it as empty so that it is in a safe state in case - // lzma_filters_copy() fails. - coder->filters[0].id = LZMA_VLI_UNKNOWN; + // Free the old filter chain and the cache. + lzma_filters_free(coder->filters, allocator); + lzma_filters_free(coder->filters_cache, allocator); + // Copy the new filter chain. return_if_error(lzma_filters_copy( filters, coder->filters, allocator)); diff --git a/src/liblzma/common/stream_flags_decoder.c b/src/liblzma/common/stream_flags_decoder.c index 4e43e359e1ef..b8d263ba4429 100644 --- a/src/liblzma/common/stream_flags_decoder.c +++ b/src/liblzma/common/stream_flags_decoder.c @@ -39,8 +39,11 @@ lzma_stream_header_decode(lzma_stream_flags *options, const uint8_t *in) const uint32_t crc = lzma_crc32(in + sizeof(lzma_header_magic), LZMA_STREAM_FLAGS_SIZE, 0); if (crc != read32le(in + sizeof(lzma_header_magic) - + LZMA_STREAM_FLAGS_SIZE)) + + LZMA_STREAM_FLAGS_SIZE)) { +#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION return LZMA_DATA_ERROR; +#endif + } // Stream Flags if (stream_flags_decode(options, in + sizeof(lzma_header_magic))) @@ -67,8 +70,11 @@ lzma_stream_footer_decode(lzma_stream_flags *options, const uint8_t *in) // CRC32 const uint32_t crc = lzma_crc32(in + sizeof(uint32_t), sizeof(uint32_t) + LZMA_STREAM_FLAGS_SIZE, 0); - if (crc != read32le(in)) + if (crc != read32le(in)) { +#ifndef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION return LZMA_DATA_ERROR; +#endif + } // Stream Flags if (stream_flags_decode(options, in + sizeof(uint32_t) * 2)) diff --git a/src/liblzma/common/string_conversion.c b/src/liblzma/common/string_conversion.c new file mode 100644 index 000000000000..53fdff2a62ee --- /dev/null +++ b/src/liblzma/common/string_conversion.c @@ -0,0 +1,1317 @@ +/////////////////////////////////////////////////////////////////////////////// +// +/// \file string_conversion.c +/// \brief Conversion of strings to filter chain and vice versa +// +// Author: Lasse Collin +// +// This file has been put into the public domain. +// You can do whatever you want with this file. +// +/////////////////////////////////////////////////////////////////////////////// + +#include "filter_common.h" + + +///////////////////// +// String building // +///////////////////// + +/// How much memory to allocate for strings. For now, no realloc is used +/// so this needs to be big enough even though there of course is +/// an overflow check still. +/// +/// FIXME? Using a fixed size is wasteful if the application doesn't free +/// the string fairly quickly but this can be improved later if needed. +#define STR_ALLOC_SIZE 800 + + +typedef struct { + char *buf; + size_t pos; +} lzma_str; + + +static lzma_ret +str_init(lzma_str *str, const lzma_allocator *allocator) +{ + str->buf = lzma_alloc(STR_ALLOC_SIZE, allocator); + if (str->buf == NULL) + return LZMA_MEM_ERROR; + + str->pos = 0; + return LZMA_OK; +} + + +static void +str_free(lzma_str *str, const lzma_allocator *allocator) +{ + lzma_free(str->buf, allocator); + return; +} + + +static bool +str_is_full(const lzma_str *str) +{ + return str->pos == STR_ALLOC_SIZE - 1; +} + + +static lzma_ret +str_finish(char **dest, lzma_str *str, const lzma_allocator *allocator) +{ + if (str_is_full(str)) { + // The preallocated buffer was too small. + // This shouldn't happen as STR_ALLOC_SIZE should + // be adjusted if new filters are added. + lzma_free(str->buf, allocator); + *dest = NULL; + assert(0); + return LZMA_PROG_ERROR; + } + + str->buf[str->pos] = '\0'; + *dest = str->buf; + return LZMA_OK; +} + + +static void +str_append_str(lzma_str *str, const char *s) +{ + const size_t len = strlen(s); + const size_t limit = STR_ALLOC_SIZE - 1 - str->pos; + const size_t copy_size = my_min(len, limit); + + memcpy(str->buf + str->pos, s, copy_size); + str->pos += copy_size; + return; +} + + +static void +str_append_u32(lzma_str *str, uint32_t v, bool use_byte_suffix) +{ + if (v == 0) { + str_append_str(str, "0"); + } else { + // NOTE: Don't use plain "B" because xz and the parser in this + // file don't support it and at glance it may look like 8 + // (there cannot be a space before the suffix). + static const char suffixes[4][4] = { "", "KiB", "MiB", "GiB" }; + + size_t suf = 0; + if (use_byte_suffix) { + while ((v & 1023) == 0 + && suf < ARRAY_SIZE(suffixes) - 1) { + v >>= 10; + ++suf; + } + } + + // UINT32_MAX in base 10 would need 10 + 1 bytes. Remember + // that initializing to "" initializes all elements to + // zero so '\0'-termination gets handled by this. + char buf[16] = ""; + size_t pos = sizeof(buf) - 1; + + do { + buf[--pos] = '0' + (v % 10); + v /= 10; + } while (v != 0); + + str_append_str(str, buf + pos); + str_append_str(str, suffixes[suf]); + } + + return; +} + + +////////////////////////////////////////////// +// Parsing and stringification declarations // +////////////////////////////////////////////// + +/// Maximum length for filter and option names. +/// 11 chars + terminating '\0' + sizeof(uint32_t) = 16 bytes +#define NAME_LEN_MAX 11 + + +/// For option_map.flags: Use .u.map to do convert the input value +/// to an integer. Without this flag, .u.range.{min,max} are used +/// as the allowed range for the integer. +#define OPTMAP_USE_NAME_VALUE_MAP 0x01 + +/// For option_map.flags: Allow KiB/MiB/GiB in input string and use them in +/// the stringified output if the value is an exact multiple of these. +/// This is used e.g. for LZMA1/2 dictionary size. +#define OPTMAP_USE_BYTE_SUFFIX 0x02 + +/// For option_map.flags: If the integer value is zero then this option +/// won't be included in the stringified output. It's used e.g. for +/// BCJ filter start offset which usually is zero. +#define OPTMAP_NO_STRFY_ZERO 0x04 + +/// Possible values for option_map.type. Since OPTMAP_TYPE_UINT32 is 0, +/// it doesn't need to be specified in the initializers as it is +/// the implicit value. +enum { + OPTMAP_TYPE_UINT32, + OPTMAP_TYPE_LZMA_MODE, + OPTMAP_TYPE_LZMA_MATCH_FINDER, + OPTMAP_TYPE_LZMA_PRESET, +}; + + +/// This is for mapping string values in options to integers. +/// The last element of an array must have "" as the name. +/// It's used e.g. for match finder names in LZMA1/2. +typedef struct { + const char name[NAME_LEN_MAX + 1]; + const uint32_t value; +} name_value_map; + + +/// Each filter that has options needs an array of option_map structures. +/// The array doesn't need to be terminated as the functions take the +/// length of the array as an argument. +/// +/// When converting a string to filter options structure, option values +/// will be handled in a few different ways: +/// +/// (1) If .type equals OPTMAP_TYPE_LZMA_PRESET then LZMA1/2 preset string +/// is handled specially. +/// +/// (2) If .flags has OPTMAP_USE_NAME_VALUE_MAP set then the string is +/// converted to an integer using the name_value_map pointed by .u.map. +/// The last element in .u.map must have .name = "" as the terminator. +/// +/// (3) Otherwise the string is treated as a non-negative unsigned decimal +/// integer which must be in the range set in .u.range. If .flags has +/// OPTMAP_USE_BYTE_SUFFIX then KiB, MiB, and GiB suffixes are allowed. +/// +/// The integer value from (2) or (3) is then stored to filter_options +/// at the offset specified in .offset using the type specified in .type +/// (default is uint32_t). +/// +/// Stringifying a filter is done by processing a given number of options +/// in oder from the beginning of an option_map array. The integer is +/// read from filter_options at .offset using the type from .type. +/// +/// If the integer is zero and .flags has OPTMAP_NO_STRFY_ZERO then the +/// option is skipped. +/// +/// If .flags has OPTMAP_USE_NAME_VALUE_MAP set then .u.map will be used +/// to convert the option to a string. If the map doesn't contain a string +/// for the integer value then "UNKNOWN" is used. +/// +/// If .flags doesn't have OPTMAP_USE_NAME_VALUE_MAP set then the integer is +/// converted to a decimal value. If OPTMAP_USE_BYTE_SUFFIX is used then KiB, +/// MiB, or GiB suffix is used if the value is an exact multiple of these. +/// Plain "B" suffix is never used. +typedef struct { + char name[NAME_LEN_MAX + 1]; + uint8_t type; + uint8_t flags; + uint16_t offset; + + union { + struct { + uint32_t min; + uint32_t max; + } range; + + const name_value_map *map; + } u; +} option_map; + + +static const char *parse_options(const char **const str, const char *str_end, + void *filter_options, + const option_map *const optmap, const size_t optmap_size); + + +///////// +// BCJ // +///////// + +#if defined(HAVE_ENCODER_X86) \ + || defined(HAVE_DECODER_X86) \ + || defined(HAVE_ENCODER_ARM) \ + || defined(HAVE_DECODER_ARM) \ + || defined(HAVE_ENCODER_ARMTHUMB) \ + || defined(HAVE_DECODER_ARMTHUMB) \ + || defined(HAVE_ENCODER_ARM64) \ + || defined(HAVE_DECODER_ARM64) \ + || defined(HAVE_ENCODER_POWERPC) \ + || defined(HAVE_DECODER_POWERPC) \ + || defined(HAVE_ENCODER_IA64) \ + || defined(HAVE_DECODER_IA64) \ + || defined(HAVE_ENCODER_SPARC) \ + || defined(HAVE_DECODER_SPARC) +static const option_map bcj_optmap[] = { + { + .name = "start", + .flags = OPTMAP_NO_STRFY_ZERO | OPTMAP_USE_BYTE_SUFFIX, + .offset = offsetof(lzma_options_bcj, start_offset), + .u.range.min = 0, + .u.range.max = UINT32_MAX, + } +}; + + +static const char * +parse_bcj(const char **const str, const char *str_end, void *filter_options) +{ + // filter_options was zeroed on allocation and that is enough + // for the default value. + return parse_options(str, str_end, filter_options, + bcj_optmap, ARRAY_SIZE(bcj_optmap)); +} +#endif + + +/////////// +// Delta // +/////////// + +#if defined(HAVE_ENCODER_DELTA) || defined(HAVE_DECODER_DELTA) +static const option_map delta_optmap[] = { + { + .name = "dist", + .offset = offsetof(lzma_options_delta, dist), + .u.range.min = LZMA_DELTA_DIST_MIN, + .u.range.max = LZMA_DELTA_DIST_MAX, + } +}; + + +static const char * +parse_delta(const char **const str, const char *str_end, void *filter_options) +{ + lzma_options_delta *opts = filter_options; + opts->type = LZMA_DELTA_TYPE_BYTE; + opts->dist = LZMA_DELTA_DIST_MIN; + + return parse_options(str, str_end, filter_options, + delta_optmap, ARRAY_SIZE(delta_optmap)); +} +#endif + + +/////////////////// +// LZMA1 & LZMA2 // +/////////////////// + +/// Help string for presets +#define LZMA12_PRESET_STR "0-9[e]" + + +static const char * +parse_lzma12_preset(const char **const str, const char *str_end, + uint32_t *preset) +{ + assert(*str < str_end); + *preset = (uint32_t)(**str - '0'); + + // NOTE: Remember to update LZMA_PRESET_STR if this is modified! + while (++*str < str_end) { + switch (**str) { + case 'e': + *preset |= LZMA_PRESET_EXTREME; + break; + + default: + return "Unsupported preset flag"; + } + } + + return NULL; +} + + +static const char * +set_lzma12_preset(const char **const str, const char *str_end, + void *filter_options) +{ + uint32_t preset; + const char *errmsg = parse_lzma12_preset(str, str_end, &preset); + if (errmsg != NULL) + return errmsg; + + lzma_options_lzma *opts = filter_options; + if (lzma_lzma_preset(opts, preset)) + return "Unsupported preset"; + + return NULL; +} + + +static const name_value_map lzma12_mode_map[] = { + { "fast", LZMA_MODE_FAST }, + { "normal", LZMA_MODE_NORMAL }, + { "", 0 } +}; + + +static const name_value_map lzma12_mf_map[] = { + { "hc3", LZMA_MF_HC3 }, + { "hc4", LZMA_MF_HC4 }, + { "bt2", LZMA_MF_BT2 }, + { "bt3", LZMA_MF_BT3 }, + { "bt4", LZMA_MF_BT4 }, + { "", 0 } +}; + + +static const option_map lzma12_optmap[] = { + { + .name = "preset", + .type = OPTMAP_TYPE_LZMA_PRESET, + }, { + .name = "dict", + .flags = OPTMAP_USE_BYTE_SUFFIX, + .offset = offsetof(lzma_options_lzma, dict_size), + .u.range.min = LZMA_DICT_SIZE_MIN, + // FIXME? The max is really max for encoding but decoding + // would allow 4 GiB - 1 B. + .u.range.max = (UINT32_C(1) << 30) + (UINT32_C(1) << 29), + }, { + .name = "lc", + .offset = offsetof(lzma_options_lzma, lc), + .u.range.min = LZMA_LCLP_MIN, + .u.range.max = LZMA_LCLP_MAX, + }, { + .name = "lp", + .offset = offsetof(lzma_options_lzma, lp), + .u.range.min = LZMA_LCLP_MIN, + .u.range.max = LZMA_LCLP_MAX, + }, { + .name = "pb", + .offset = offsetof(lzma_options_lzma, pb), + .u.range.min = LZMA_PB_MIN, + .u.range.max = LZMA_PB_MAX, + }, { + .name = "mode", + .type = OPTMAP_TYPE_LZMA_MODE, + .flags = OPTMAP_USE_NAME_VALUE_MAP, + .offset = offsetof(lzma_options_lzma, mode), + .u.map = lzma12_mode_map, + }, { + .name = "nice", + .offset = offsetof(lzma_options_lzma, nice_len), + .u.range.min = 2, + .u.range.max = 273, + }, { + .name = "mf", + .type = OPTMAP_TYPE_LZMA_MATCH_FINDER, + .flags = OPTMAP_USE_NAME_VALUE_MAP, + .offset = offsetof(lzma_options_lzma, mf), + .u.map = lzma12_mf_map, + }, { + .name = "depth", + .offset = offsetof(lzma_options_lzma, depth), + .u.range.min = 0, + .u.range.max = UINT32_MAX, + } +}; + + +static const char * +parse_lzma12(const char **const str, const char *str_end, void *filter_options) +{ + lzma_options_lzma *opts = filter_options; + + // It cannot fail. + const bool preset_ret = lzma_lzma_preset(opts, LZMA_PRESET_DEFAULT); + assert(!preset_ret); + (void)preset_ret; + + const char *errmsg = parse_options(str, str_end, filter_options, + lzma12_optmap, ARRAY_SIZE(lzma12_optmap)); + if (errmsg != NULL) + return errmsg; + + if (opts->lc + opts->lp > LZMA_LCLP_MAX) + return "The sum of lc and lp must not exceed 4"; + + return NULL; +} + + +///////////////////////////////////////// +// Generic parsing and stringification // +///////////////////////////////////////// + +static const struct { + /// Name of the filter + char name[NAME_LEN_MAX + 1]; + + /// For lzma_str_to_filters: + /// Size of the filter-specific options structure. + uint32_t opts_size; + + /// Filter ID + lzma_vli id; + + /// For lzma_str_to_filters: + /// Function to parse the filter-specific options. The filter_options + /// will already have been allocated using lzma_alloc_zero(). + const char *(*parse)(const char **str, const char *str_end, + void *filter_options); + + /// For lzma_str_from_filters: + /// If the flag LZMA_STR_ENCODER is used then the first + /// strfy_encoder elements of optmap are stringified. + /// With LZMA_STR_DECODER strfy_decoder is used. + /// Currently encoders use all flags that decoders do but if + /// that changes then this needs to be changed too, for example, + /// add a new OPTMAP flag to skip printing some decoder-only flags. + const option_map *optmap; + uint8_t strfy_encoder; + uint8_t strfy_decoder; + + /// For lzma_str_from_filters: + /// If true, lzma_filter.options is allowed to be NULL. In that case, + /// only the filter name is printed without any options. + bool allow_null; + +} filter_name_map[] = { +#if defined (HAVE_ENCODER_LZMA1) || defined(HAVE_DECODER_LZMA1) + { "lzma1", sizeof(lzma_options_lzma), LZMA_FILTER_LZMA1, + &parse_lzma12, lzma12_optmap, 9, 5, false }, +#endif + +#if defined(HAVE_ENCODER_LZMA2) || defined(HAVE_DECODER_LZMA2) + { "lzma2", sizeof(lzma_options_lzma), LZMA_FILTER_LZMA2, + &parse_lzma12, lzma12_optmap, 9, 2, false }, +#endif + +#if defined(HAVE_ENCODER_X86) || defined(HAVE_DECODER_X86) + { "x86", sizeof(lzma_options_bcj), LZMA_FILTER_X86, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_ARM) || defined(HAVE_DECODER_ARM) + { "arm", sizeof(lzma_options_bcj), LZMA_FILTER_ARM, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_ARMTHUMB) || defined(HAVE_DECODER_ARMTHUMB) + { "armthumb", sizeof(lzma_options_bcj), LZMA_FILTER_ARMTHUMB, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_ARM64) || defined(HAVE_DECODER_ARM64) + { "arm64", sizeof(lzma_options_bcj), LZMA_FILTER_ARM64, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_POWERPC) || defined(HAVE_DECODER_POWERPC) + { "powerpc", sizeof(lzma_options_bcj), LZMA_FILTER_POWERPC, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_IA64) || defined(HAVE_DECODER_IA64) + { "ia64", sizeof(lzma_options_bcj), LZMA_FILTER_IA64, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_SPARC) || defined(HAVE_DECODER_SPARC) + { "sparc", sizeof(lzma_options_bcj), LZMA_FILTER_SPARC, + &parse_bcj, bcj_optmap, 1, 1, true }, +#endif + +#if defined(HAVE_ENCODER_DELTA) || defined(HAVE_DECODER_DELTA) + { "delta", sizeof(lzma_options_delta), LZMA_FILTER_DELTA, + &parse_delta, delta_optmap, 1, 1, false }, +#endif +}; + + +/// Decodes options from a string for one filter (name1=value1,name2=value2). +/// Caller must have allocated memory for filter_options already and set +/// the initial default values. This is called from the filter-specific +/// parse_* functions. +/// +/// The input string starts at *str and the address in str_end is the first +/// char that is not part of the string anymore. So no '\0' terminator is +/// used. *str is advanced everytime something has been decoded successfully. +static const char * +parse_options(const char **const str, const char *str_end, + void *filter_options, + const option_map *const optmap, const size_t optmap_size) +{ + while (*str < str_end && **str != '\0') { + // Each option is of the form name=value. + // Commas (',') separate options. Extra commas are ignored. + // Ignoring extra commas makes it simpler if an optional + // option stored in a shell variable which can be empty. + if (**str == ',') { + ++*str; + continue; + } + + // Find where the next name=value ends. + const size_t str_len = (size_t)(str_end - *str); + const char *name_eq_value_end = memchr(*str, ',', str_len); + if (name_eq_value_end == NULL) + name_eq_value_end = str_end; + + const char *equals_sign = memchr(*str, '=', + (size_t)(name_eq_value_end - *str)); + + // Fail if the '=' wasn't found or the option name is missing + // (the first char is '='). + if (equals_sign == NULL || **str == '=') + return "Options must be 'name=value' pairs separated " + "with commas"; + + // Reject a too long option name so that the memcmp() + // in the loop below won't read past the end of the + // string in optmap[i].name. + const size_t name_len = (size_t)(equals_sign - *str); + if (name_len > NAME_LEN_MAX) + return "Unknown option name"; + + // Find the option name from optmap[]. + size_t i = 0; + while (true) { + if (i == optmap_size) + return "Unknown option name"; + + if (memcmp(*str, optmap[i].name, name_len) == 0 + && optmap[i].name[name_len] == '\0') + break; + + ++i; + } + + // The input string is good at least until the start of + // the option value. + *str = equals_sign + 1; + + // The code assumes that the option value isn't an empty + // string so check it here. + const size_t value_len = (size_t)(name_eq_value_end - *str); + if (value_len == 0) + return "Option value cannot be empty"; + + // LZMA1/2 preset has its own parsing function. + if (optmap[i].type == OPTMAP_TYPE_LZMA_PRESET) { + const char *errmsg = set_lzma12_preset(str, + name_eq_value_end, filter_options); + if (errmsg != NULL) + return errmsg; + + continue; + } + + // It's an integer value. + uint32_t v; + if (optmap[i].flags & OPTMAP_USE_NAME_VALUE_MAP) { + // The integer is picked from a string-to-integer map. + // + // Reject a too long value string so that the memcmp() + // in the loop below won't read past the end of the + // string in optmap[i].u.map[j].name. + if (value_len > NAME_LEN_MAX) + return "Invalid option value"; + + const name_value_map *map = optmap[i].u.map; + size_t j = 0; + while (true) { + // The array is terminated with an empty name. + if (map[j].name[0] == '\0') + return "Invalid option value"; + + if (memcmp(*str, map[j].name, value_len) == 0 + && map[j].name[value_len] + == '\0') { + v = map[j].value; + break; + } + + ++j; + } + } else if (**str < '0' || **str > '9') { + // Note that "max" isn't supported while it is + // supported in xz. It's not useful here. + return "Value is not a non-negative decimal integer"; + } else { + // strtoul() has locale-specific behavior so it cannot + // be relied on to get reproducible results since we + // cannot change the locate in a thread-safe library. + // It also needs '\0'-termination. + // + // Use a temporary pointer so that *str will point + // to the beginning of the value string in case + // an error occurs. + const char *p = *str; + v = 0; + do { + if (v > UINT32_MAX / 10) + return "Value out of range"; + + v *= 10; + + const uint32_t add = (uint32_t)(*p - '0'); + if (UINT32_MAX - add < v) + return "Value out of range"; + + v += add; + ++p; + } while (p < name_eq_value_end + && *p >= '0' && *p <= '9'); + + if (p < name_eq_value_end) { + // Remember this position so that it an be + // used for error messages that are + // specifically about the suffix. (Out of + // range values are about the whole value + // and those error messages point to the + // beginning of the number part, + // not to the suffix.) + const char *multiplier_start = p; + + // If multiplier suffix shouldn't be used + // then don't allow them even if the value + // would stay within limits. This is a somewhat + // unnecessary check but it rejects silly + // things like lzma2:pb=0MiB which xz allows. + if ((optmap[i].flags & OPTMAP_USE_BYTE_SUFFIX) + == 0) { + *str = multiplier_start; + return "This option does not support " + "any integer suffixes"; + } + + uint32_t shift; + + switch (*p) { + case 'k': + case 'K': + shift = 10; + break; + + case 'm': + case 'M': + shift = 20; + break; + + case 'g': + case 'G': + shift = 30; + break; + + default: + *str = multiplier_start; + return "Invalid multiplier suffix " + "(KiB, MiB, or GiB)"; + } + + ++p; + + // Allow "M", "Mi", "MB", "MiB" and the same + // for the other five characters from the + // switch-statement above. All are handled + // as base-2 (perhaps a mistake, perhaps not). + // Note that 'i' and 'B' are case sensitive. + if (p < name_eq_value_end && *p == 'i') + ++p; + + if (p < name_eq_value_end && *p == 'B') + ++p; + + // Now we must have no chars remaining. + if (p < name_eq_value_end) { + *str = multiplier_start; + return "Invalid multiplier suffix " + "(KiB, MiB, or GiB)"; + } + + if (v > (UINT32_MAX >> shift)) + return "Value out of range"; + + v <<= shift; + } + + if (v < optmap[i].u.range.min + || v > optmap[i].u.range.max) + return "Value out of range"; + } + + // Set the value in filter_options. Enums are handled + // specially since the underlying type isn't the same + // as uint32_t on all systems. + void *ptr = (char *)filter_options + optmap[i].offset; + switch (optmap[i].type) { + case OPTMAP_TYPE_LZMA_MODE: + *(lzma_mode *)ptr = (lzma_mode)v; + break; + + case OPTMAP_TYPE_LZMA_MATCH_FINDER: + *(lzma_match_finder *)ptr = (lzma_match_finder)v; + break; + + default: + *(uint32_t *)ptr = v; + break; + } + + // This option has been successfully handled. + *str = name_eq_value_end; + } + + // No errors. + return NULL; +} + + +/// Finds the name of the filter at the beginning of the string and +/// calls filter_name_map[i].parse() to decode the filter-specific options. +/// The caller must have set str_end so that exactly one filter and its +/// options are present without any trailing characters. +static const char * +parse_filter(const char **const str, const char *str_end, lzma_filter *filter, + const lzma_allocator *allocator, bool only_xz) +{ + // Search for a colon or equals sign that would separate the filter + // name from filter options. If neither is found, then the input + // string only contains a filter name and there are no options. + // + // First assume that a colon or equals sign won't be found: + const char *name_end = str_end; + const char *opts_start = str_end; + + for (const char *p = *str; p < str_end; ++p) { + if (*p == ':' || *p == '=') { + name_end = p; + + // Filter options (name1=value1,name2=value2,...) + // begin after the colon or equals sign. + opts_start = p + 1; + break; + } + } + + // Reject a too long filter name so that the memcmp() + // in the loop below won't read past the end of the + // string in filter_name_map[i].name. + const size_t name_len = (size_t)(name_end - *str); + if (name_len > NAME_LEN_MAX) + return "Unknown filter name"; + + for (size_t i = 0; i < ARRAY_SIZE(filter_name_map); ++i) { + if (memcmp(*str, filter_name_map[i].name, name_len) == 0 + && filter_name_map[i].name[name_len] == '\0') { + if (only_xz && filter_name_map[i].id + >= LZMA_FILTER_RESERVED_START) + return "This filter cannot be used in " + "the .xz format"; + + // Allocate the filter-specific options and + // initialize the memory with zeros. + void *options = lzma_alloc_zero( + filter_name_map[i].opts_size, + allocator); + if (options == NULL) + return "Memory allocation failed"; + + // Filter name was found so the input string is good + // at least this far. + *str = opts_start; + + const char *errmsg = filter_name_map[i].parse( + str, str_end, options); + if (errmsg != NULL) { + lzma_free(options, allocator); + return errmsg; + } + + // *filter is modified only when parsing is successful. + filter->id = filter_name_map[i].id; + filter->options = options; + return NULL; + } + } + + return "Unknown filter name"; +} + + +/// Converts the string to a filter chain (array of lzma_filter structures). +/// +/// *str is advanced everytime something has been decoded successfully. +/// This way the caller knows where in the string a possible error occurred. +static const char * +str_to_filters(const char **const str, lzma_filter *filters, uint32_t flags, + const lzma_allocator *allocator) +{ + const char *errmsg; + + // Skip leading spaces. + while (**str == ' ') + ++*str; + + if (**str == '\0') + return "Empty string is not allowed, " + "try \"6\" if a default value is needed"; + + // Detect the type of the string. + // + // A string beginning with a digit or a string beginning with + // one dash and a digit are treated as presets. Trailing spaces + // will be ignored too (leading spaces were already ignored above). + // + // For example, "6", "7 ", "-9e", or " -3 " are treated as presets. + // Strings like "-" or "- " aren't preset. +#define MY_IS_DIGIT(c) ((c) >= '0' && (c) <= '9') + if (MY_IS_DIGIT(**str) || (**str == '-' && MY_IS_DIGIT((*str)[1]))) { + if (**str == '-') + ++*str; + + // Ignore trailing spaces. + const size_t str_len = strlen(*str); + const char *str_end = memchr(*str, ' ', str_len); + if (str_end != NULL) { + // There is at least one trailing space. Check that + // there are no chars other than spaces. + for (size_t i = 1; str_end[i] != '\0'; ++i) + if (str_end[i] != ' ') + return "Unsupported preset"; + } else { + // There are no trailing spaces. Use the whole string. + str_end = *str + str_len; + } + + uint32_t preset; + errmsg = parse_lzma12_preset(str, str_end, &preset); + if (errmsg != NULL) + return errmsg; + + lzma_options_lzma *opts = lzma_alloc(sizeof(*opts), allocator); + if (opts == NULL) + return "Memory allocation failed"; + + if (lzma_lzma_preset(opts, preset)) { + lzma_free(opts, allocator); + return "Unsupported preset"; + } + + filters[0].id = LZMA_FILTER_LZMA2; + filters[0].options = opts; + filters[1].id = LZMA_VLI_UNKNOWN; + filters[1].options = NULL; + + return NULL; + } + + // Not a preset so it must be a filter chain. + // + // If LZMA_STR_ALL_FILTERS isn't used we allow only filters that + // can be used in .xz. + const bool only_xz = (flags & LZMA_STR_ALL_FILTERS) == 0; + + // Use a temporary array so that we don't modify the caller-supplied + // one until we know that no errors occurred. + lzma_filter temp_filters[LZMA_FILTERS_MAX + 1]; + + size_t i = 0; + do { + if (i == LZMA_FILTERS_MAX) { + errmsg = "The maximum number of filters is four"; + goto error; + } + + // Skip "--" if present. + if ((*str)[0] == '-' && (*str)[1] == '-') + *str += 2; + + // Locate the end of "filter:name1=value1,name2=value2", + // stopping at the first "--" or a single space. + const char *filter_end = *str; + while (filter_end[0] != '\0') { + if ((filter_end[0] == '-' && filter_end[1] == '-') + || filter_end[0] == ' ') + break; + + ++filter_end; + } + + // Inputs that have "--" at the end or "-- " in the middle + // will result in an empty filter name. + if (filter_end == *str) { + errmsg = "Filter name is missing"; + goto error; + } + + errmsg = parse_filter(str, filter_end, &temp_filters[i], + allocator, only_xz); + if (errmsg != NULL) + goto error; + + // Skip trailing spaces. + while (**str == ' ') + ++*str; + + ++i; + } while (**str != '\0'); + + // Seems to be good, terminate the array so that + // basic validation can be done. + temp_filters[i].id = LZMA_VLI_UNKNOWN; + temp_filters[i].options = NULL; + + // Do basic validation if the application didn't prohibit it. + if ((flags & LZMA_STR_NO_VALIDATION) == 0) { + size_t dummy; + const lzma_ret ret = lzma_validate_chain(temp_filters, &dummy); + assert(ret == LZMA_OK || ret == LZMA_OPTIONS_ERROR); + if (ret != LZMA_OK) { + errmsg = "Invalid filter chain " + "('lzma2' missing at the end?)"; + goto error; + } + } + + // All good. Copy the filters to the application supplied array. + memcpy(filters, temp_filters, (i + 1) * sizeof(lzma_filter)); + return NULL; + +error: + // Free the filter options that were successfully decoded. + while (i-- > 0) + lzma_free(temp_filters[i].options, allocator); + + return errmsg; +} + + +extern LZMA_API(const char *) +lzma_str_to_filters(const char *str, int *error_pos, lzma_filter *filters, + uint32_t flags, const lzma_allocator *allocator) +{ + if (str == NULL || filters == NULL) + return "Unexpected NULL pointer argument(s) " + "to lzma_str_to_filters()"; + + // Validate the flags. + const uint32_t supported_flags + = LZMA_STR_ALL_FILTERS + | LZMA_STR_NO_VALIDATION; + + if (flags & ~supported_flags) + return "Unsupported flags to lzma_str_to_filters()"; + + const char *used = str; + const char *errmsg = str_to_filters(&used, filters, flags, allocator); + + if (error_pos != NULL) { + const size_t n = (size_t)(used - str); + *error_pos = n > INT_MAX ? INT_MAX : (int)n; + } + + return errmsg; +} + + +/// Converts options of one filter to a string. +/// +/// The caller must have already put the filter name in the destination +/// string. Since it is possible that no options will be needed, the caller +/// won't have put a delimiter character (':' or '=') in the string yet. +/// We will add it if at least one option will be added to the string. +static void +strfy_filter(lzma_str *dest, const char *delimiter, + const option_map *optmap, size_t optmap_count, + const void *filter_options) +{ + for (size_t i = 0; i < optmap_count; ++i) { + // No attempt is made to reverse LZMA1/2 preset. + if (optmap[i].type == OPTMAP_TYPE_LZMA_PRESET) + continue; + + // All options have integer values, some just are mapped + // to a string with a name_value_map. LZMA1/2 preset + // isn't reversed back to preset=PRESET form. + uint32_t v; + const void *ptr + = (const char *)filter_options + optmap[i].offset; + switch (optmap[i].type) { + case OPTMAP_TYPE_LZMA_MODE: + v = *(const lzma_mode *)ptr; + break; + + case OPTMAP_TYPE_LZMA_MATCH_FINDER: + v = *(const lzma_match_finder *)ptr; + break; + + default: + v = *(const uint32_t *)ptr; + break; + } + + // Skip this if this option should be omitted from + // the string when the value is zero. + if (v == 0 && (optmap[i].flags & OPTMAP_NO_STRFY_ZERO)) + continue; + + // Before the first option we add whatever delimiter + // the caller gave us. For later options a comma is used. + str_append_str(dest, delimiter); + delimiter = ","; + + // Add the option name and equals sign. + str_append_str(dest, optmap[i].name); + str_append_str(dest, "="); + + if (optmap[i].flags & OPTMAP_USE_NAME_VALUE_MAP) { + const name_value_map *map = optmap[i].u.map; + size_t j = 0; + while (true) { + if (map[j].name[0] == '\0') { + str_append_str(dest, "UNKNOWN"); + break; + } + + if (map[j].value == v) { + str_append_str(dest, map[j].name); + break; + } + + ++j; + } + } else { + str_append_u32(dest, v, + optmap[i].flags & OPTMAP_USE_BYTE_SUFFIX); + } + } + + return; +} + + +extern LZMA_API(lzma_ret) +lzma_str_from_filters(char **output_str, const lzma_filter *filters, + uint32_t flags, const lzma_allocator *allocator) +{ + // On error *output_str is always set to NULL. + // Do it as the very first step. + if (output_str == NULL) + return LZMA_PROG_ERROR; + + *output_str = NULL; + + if (filters == NULL) + return LZMA_PROG_ERROR; + + // Validate the flags. + const uint32_t supported_flags + = LZMA_STR_ENCODER + | LZMA_STR_DECODER + | LZMA_STR_GETOPT_LONG + | LZMA_STR_NO_SPACES; + + if (flags & ~supported_flags) + return LZMA_OPTIONS_ERROR; + + // There must be at least one filter. + if (filters[0].id == LZMA_VLI_UNKNOWN) + return LZMA_OPTIONS_ERROR; + + // Allocate memory for the output string. + lzma_str dest; + return_if_error(str_init(&dest, allocator)); + + const bool show_opts = (flags & (LZMA_STR_ENCODER | LZMA_STR_DECODER)); + + const char *opt_delim = (flags & LZMA_STR_GETOPT_LONG) ? "=" : ":"; + + for (size_t i = 0; filters[i].id != LZMA_VLI_UNKNOWN; ++i) { + // Don't add a space between filters if the caller + // doesn't want them. + if (i > 0 && !(flags & LZMA_STR_NO_SPACES)) + str_append_str(&dest, " "); + + // Use dashes for xz getopt_long() compatible syntax but also + // use dashes to separate filters when spaces weren't wanted. + if ((flags & LZMA_STR_GETOPT_LONG) + || (i > 0 && (flags & LZMA_STR_NO_SPACES))) + str_append_str(&dest, "--"); + + size_t j = 0; + while (true) { + if (j == ARRAY_SIZE(filter_name_map)) { + // Filter ID in filters[i].id isn't supported. + str_free(&dest, allocator); + return LZMA_OPTIONS_ERROR; + } + + if (filter_name_map[j].id == filters[i].id) { + // Add the filter name. + str_append_str(&dest, filter_name_map[j].name); + + // If only the filter names were wanted then + // skip to the next filter. In this case + // .options is ignored and may be NULL even + // when the filter doesn't allow NULL options. + if (!show_opts) + break; + + if (filters[i].options == NULL) { + if (!filter_name_map[j].allow_null) { + // Filter-specific options + // are missing but with + // this filter the options + // structure is mandatory. + str_free(&dest, allocator); + return LZMA_OPTIONS_ERROR; + } + + // .options is allowed to be NULL. + // There is no need to add any + // options to the string. + break; + } + + // Options structure is available. Add + // the filter options to the string. + const size_t optmap_count + = (flags & LZMA_STR_ENCODER) + ? filter_name_map[j].strfy_encoder + : filter_name_map[j].strfy_decoder; + strfy_filter(&dest, opt_delim, + filter_name_map[j].optmap, + optmap_count, + filters[i].options); + break; + } + + ++j; + } + } + + return str_finish(output_str, &dest, allocator); +} + + +extern LZMA_API(lzma_ret) +lzma_str_list_filters(char **output_str, lzma_vli filter_id, uint32_t flags, + const lzma_allocator *allocator) +{ + // On error *output_str is always set to NULL. + // Do it as the very first step. + if (output_str == NULL) + return LZMA_PROG_ERROR; + + *output_str = NULL; + + // Validate the flags. + const uint32_t supported_flags + = LZMA_STR_ALL_FILTERS + | LZMA_STR_ENCODER + | LZMA_STR_DECODER + | LZMA_STR_GETOPT_LONG; + + if (flags & ~supported_flags) + return LZMA_OPTIONS_ERROR; + + // Allocate memory for the output string. + lzma_str dest; + return_if_error(str_init(&dest, allocator)); + + // If only listing the filter names then separate them with spaces. + // Otherwise use newlines. + const bool show_opts = (flags & (LZMA_STR_ENCODER | LZMA_STR_DECODER)); + const char *filter_delim = show_opts ? "\n" : " "; + + const char *opt_delim = (flags & LZMA_STR_GETOPT_LONG) ? "=" : ":"; + bool first_filter_printed = false; + + for (size_t i = 0; i < ARRAY_SIZE(filter_name_map); ++i) { + // If we are printing only one filter then skip others. + if (filter_id != LZMA_VLI_UNKNOWN + && filter_id != filter_name_map[i].id) + continue; + + // If we are printing only .xz filters then skip the others. + if (filter_name_map[i].id >= LZMA_FILTER_RESERVED_START + && (flags & LZMA_STR_ALL_FILTERS) == 0 + && filter_id == LZMA_VLI_UNKNOWN) + continue; + + // Add a new line if this isn't the first filter being + // written to the string. + if (first_filter_printed) + str_append_str(&dest, filter_delim); + + first_filter_printed = true; + + if (flags & LZMA_STR_GETOPT_LONG) + str_append_str(&dest, "--"); + + str_append_str(&dest, filter_name_map[i].name); + + // If only the filter names were wanted then continue + // to the next filter. + if (!show_opts) + continue; + + const option_map *optmap = filter_name_map[i].optmap; + const char *d = opt_delim; + + const size_t end = (flags & LZMA_STR_ENCODER) + ? filter_name_map[i].strfy_encoder + : filter_name_map[i].strfy_decoder; + + for (size_t j = 0; j < end; ++j) { + // The first option is delimited from the filter + // name using "=" or ":" and the rest of the options + // are separated with ",". + str_append_str(&dest, d); + d = ","; + + // optname=<possible_values> + str_append_str(&dest, optmap[j].name); + str_append_str(&dest, "=<"); + + if (optmap[j].type == OPTMAP_TYPE_LZMA_PRESET) { + // LZMA1/2 preset has its custom help string. + str_append_str(&dest, LZMA12_PRESET_STR); + } else if (optmap[j].flags + & OPTMAP_USE_NAME_VALUE_MAP) { + // Separate the possible option values by "|". + const name_value_map *m = optmap[j].u.map; + for (size_t k = 0; m[k].name[0] != '\0'; ++k) { + if (k > 0) + str_append_str(&dest, "|"); + + str_append_str(&dest, m[k].name); + } + } else { + // Integer range is shown as min-max. + const bool use_byte_suffix = optmap[j].flags + & OPTMAP_USE_BYTE_SUFFIX; + str_append_u32(&dest, optmap[j].u.range.min, + use_byte_suffix); + str_append_str(&dest, "-"); + str_append_u32(&dest, optmap[j].u.range.max, + use_byte_suffix); + } + + str_append_str(&dest, ">"); + } + } + + // If no filters were added to the string then it must be because + // the caller provided an unsupported Filter ID. + if (!first_filter_printed) { + str_free(&dest, allocator); + return LZMA_OPTIONS_ERROR; + } + + return str_finish(output_str, &dest, allocator); +} diff --git a/src/liblzma/liblzma_generic.map b/src/liblzma/liblzma_generic.map index 8cca05bdeead..bb82167ed57a 100644 --- a/src/liblzma/liblzma_generic.map +++ b/src/liblzma/liblzma_generic.map @@ -106,3 +106,16 @@ global: lzma_stream_encoder_mt; lzma_stream_encoder_mt_memusage; } XZ_5.0; + +XZ_5.4 { +global: + lzma_file_info_decoder; + lzma_filters_free; + lzma_lzip_decoder; + lzma_microlzma_decoder; + lzma_microlzma_encoder; + lzma_stream_decoder_mt; + lzma_str_from_filters; + lzma_str_list_filters; + lzma_str_to_filters; +} XZ_5.2; diff --git a/src/liblzma/liblzma_linux.map b/src/liblzma/liblzma_linux.map index 4be882cc2914..449f5fd682db 100644 --- a/src/liblzma/liblzma_linux.map +++ b/src/liblzma/liblzma_linux.map @@ -121,3 +121,16 @@ global: lzma_stream_encoder_mt; lzma_stream_encoder_mt_memusage; } XZ_5.1.2alpha; + +XZ_5.4 { +global: + lzma_file_info_decoder; + lzma_filters_free; + lzma_lzip_decoder; + lzma_microlzma_decoder; + lzma_microlzma_encoder; + lzma_stream_decoder_mt; + lzma_str_from_filters; + lzma_str_list_filters; + lzma_str_to_filters; +} XZ_5.2; diff --git a/src/liblzma/lz/lz_decoder.c b/src/liblzma/lz/lz_decoder.c index ab6af0dd99b3..06c95c1137df 100644 --- a/src/liblzma/lz/lz_decoder.c +++ b/src/liblzma/lz/lz_decoder.c @@ -212,7 +212,8 @@ extern lzma_ret lzma_lz_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, const lzma_filter_info *filters, lzma_ret (*lz_init)(lzma_lz_decoder *lz, - const lzma_allocator *allocator, const void *options, + const lzma_allocator *allocator, + lzma_vli id, const void *options, lzma_lz_options *lz_options)) { // Allocate the base structure if it isn't already allocated. @@ -236,7 +237,7 @@ lzma_lz_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, // us the dictionary size. lzma_lz_options lz_options; return_if_error(lz_init(&coder->lz, allocator, - filters[0].options, &lz_options)); + filters[0].id, filters[0].options, &lz_options)); // If the dictionary size is very small, increase it to 4096 bytes. // This is to prevent constant wrapping of the dictionary, which @@ -301,17 +302,3 @@ lzma_lz_decoder_memusage(size_t dictionary_size) { return sizeof(lzma_coder) + (uint64_t)(dictionary_size); } - - -extern void -lzma_lz_decoder_uncompressed(void *coder_ptr, lzma_vli uncompressed_size, - bool allow_eopm) -{ - lzma_coder *coder = coder_ptr; - - if (uncompressed_size == LZMA_VLI_UNKNOWN) - allow_eopm = true; - - coder->lz.set_uncompressed(coder->lz.coder, uncompressed_size, - allow_eopm); -} diff --git a/src/liblzma/lz/lz_decoder.h b/src/liblzma/lz/lz_decoder.h index e6d7ab2a4107..ad80d4dd0d14 100644 --- a/src/liblzma/lz/lz_decoder.h +++ b/src/liblzma/lz/lz_decoder.h @@ -87,14 +87,12 @@ extern lzma_ret lzma_lz_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, const lzma_filter_info *filters, lzma_ret (*lz_init)(lzma_lz_decoder *lz, - const lzma_allocator *allocator, const void *options, + const lzma_allocator *allocator, + lzma_vli id, const void *options, lzma_lz_options *lz_options)); extern uint64_t lzma_lz_decoder_memusage(size_t dictionary_size); -extern void lzma_lz_decoder_uncompressed( - void *coder, lzma_vli uncompressed_size, bool allow_eopm); - ////////////////////// // Inline functions // diff --git a/src/liblzma/lz/lz_encoder.c b/src/liblzma/lz/lz_encoder.c index e6a49fb19e31..5489085a0860 100644 --- a/src/liblzma/lz/lz_encoder.c +++ b/src/liblzma/lz/lz_encoder.c @@ -293,11 +293,15 @@ lz_encoder_prepare(lzma_mf *mf, const lzma_allocator *allocator, return true; } - // Calculate the sizes of mf->hash and mf->son and check that - // nice_len is big enough for the selected match finder. - const uint32_t hash_bytes = lz_options->match_finder & 0x0F; - if (hash_bytes > mf->nice_len) - return true; + // Calculate the sizes of mf->hash and mf->son. + // + // NOTE: Since 5.3.5beta the LZMA encoder ensures that nice_len + // is big enough for the selected match finder. This makes it + // easier for applications as nice_len = 2 will always be accepted + // even though the effective value can be slightly bigger. + const uint32_t hash_bytes + = mf_get_hash_bytes(lz_options->match_finder); + assert(hash_bytes <= mf->nice_len); const bool is_bt = (lz_options->match_finder & 0x10) != 0; uint32_t hs; @@ -521,14 +525,30 @@ lz_encoder_update(void *coder_ptr, const lzma_allocator *allocator, } +static lzma_ret +lz_encoder_set_out_limit(void *coder_ptr, uint64_t *uncomp_size, + uint64_t out_limit) +{ + lzma_coder *coder = coder_ptr; + + // This is supported only if there are no other filters chained. + if (coder->next.code == NULL && coder->lz.set_out_limit != NULL) + return coder->lz.set_out_limit( + coder->lz.coder, uncomp_size, out_limit); + + return LZMA_OPTIONS_ERROR; +} + + extern lzma_ret lzma_lz_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, const lzma_filter_info *filters, lzma_ret (*lz_init)(lzma_lz_encoder *lz, - const lzma_allocator *allocator, const void *options, + const lzma_allocator *allocator, + lzma_vli id, const void *options, lzma_lz_options *lz_options)) { -#ifdef HAVE_SMALL +#if defined(HAVE_SMALL) && !defined(HAVE_FUNC_ATTRIBUTE_CONSTRUCTOR) // We need that the CRC32 table has been initialized. lzma_crc32_init(); #endif @@ -544,6 +564,7 @@ lzma_lz_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, next->code = &lz_encode; next->end = &lz_encoder_end; next->update = &lz_encoder_update; + next->set_out_limit = &lz_encoder_set_out_limit; coder->lz.coder = NULL; coder->lz.code = NULL; @@ -565,7 +586,7 @@ lzma_lz_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, // Initialize the LZ-based encoder. lzma_lz_options lz_options; return_if_error(lz_init(&coder->lz, allocator, - filters[0].options, &lz_options)); + filters[0].id, filters[0].options, &lz_options)); // Setup the size information into coder->mf and deallocate // old buffers if they have wrong size. diff --git a/src/liblzma/lz/lz_encoder.h b/src/liblzma/lz/lz_encoder.h index 426dcd8a3875..7950a2f4ef1b 100644 --- a/src/liblzma/lz/lz_encoder.h +++ b/src/liblzma/lz/lz_encoder.h @@ -204,6 +204,10 @@ typedef struct { /// Update the options in the middle of the encoding. lzma_ret (*options_update)(void *coder, const lzma_filter *filter); + /// Set maximum allowed output size + lzma_ret (*set_out_limit)(void *coder, uint64_t *uncomp_size, + uint64_t out_limit); + } lzma_lz_encoder; @@ -216,6 +220,15 @@ typedef struct { // are called `read ahead'. +/// Get how many bytes the match finder hashes in its initial step. +/// This is also the minimum nice_len value with the match finder. +static inline uint32_t +mf_get_hash_bytes(lzma_match_finder match_finder) +{ + return (uint32_t)match_finder & 0x0F; +} + + /// Get pointer to the first byte not ran through the match finder static inline const uint8_t * mf_ptr(const lzma_mf *mf) @@ -298,7 +311,8 @@ extern lzma_ret lzma_lz_encoder_init( lzma_next_coder *next, const lzma_allocator *allocator, const lzma_filter_info *filters, lzma_ret (*lz_init)(lzma_lz_encoder *lz, - const lzma_allocator *allocator, const void *options, + const lzma_allocator *allocator, + lzma_vli id, const void *options, lzma_lz_options *lz_options)); diff --git a/src/liblzma/lzma/lzma2_decoder.c b/src/liblzma/lzma/lzma2_decoder.c index 105a28dc97ed..567df490ca5b 100644 --- a/src/liblzma/lzma/lzma2_decoder.c +++ b/src/liblzma/lzma/lzma2_decoder.c @@ -226,7 +226,8 @@ lzma2_decoder_end(void *coder_ptr, const lzma_allocator *allocator) static lzma_ret lzma2_decoder_init(lzma_lz_decoder *lz, const lzma_allocator *allocator, - const void *opt, lzma_lz_options *lz_options) + lzma_vli id lzma_attribute((__unused__)), const void *opt, + lzma_lz_options *lz_options) { lzma_lzma2_coder *coder = lz->coder; if (coder == NULL) { diff --git a/src/liblzma/lzma/lzma2_encoder.c b/src/liblzma/lzma/lzma2_encoder.c index 6914f2793b26..4b6b23118d70 100644 --- a/src/liblzma/lzma/lzma2_encoder.c +++ b/src/liblzma/lzma/lzma2_encoder.c @@ -310,7 +310,8 @@ lzma2_encoder_options_update(void *coder_ptr, const lzma_filter *filter) static lzma_ret lzma2_encoder_init(lzma_lz_encoder *lz, const lzma_allocator *allocator, - const void *options, lzma_lz_options *lz_options) + lzma_vli id lzma_attribute((__unused__)), const void *options, + lzma_lz_options *lz_options) { if (options == NULL) return LZMA_PROG_ERROR; @@ -340,7 +341,7 @@ lzma2_encoder_init(lzma_lz_encoder *lz, const lzma_allocator *allocator, // Initialize LZMA encoder return_if_error(lzma_lzma_encoder_create(&coder->lzma, allocator, - &coder->opt_cur, lz_options)); + LZMA_FILTER_LZMA2, &coder->opt_cur, lz_options)); // Make sure that we will always have enough history available in // case we need to use uncompressed chunks. They are used when the diff --git a/src/liblzma/lzma/lzma_decoder.c b/src/liblzma/lzma/lzma_decoder.c index f56ccaf77fd3..26c148a95e25 100644 --- a/src/liblzma/lzma/lzma_decoder.c +++ b/src/liblzma/lzma/lzma_decoder.c @@ -986,7 +986,7 @@ lzma_decoder_reset(void *coder_ptr, const void *opt) extern lzma_ret lzma_lzma_decoder_create(lzma_lz_decoder *lz, const lzma_allocator *allocator, - const void *opt, lzma_lz_options *lz_options) + const lzma_options_lzma *options, lzma_lz_options *lz_options) { if (lz->coder == NULL) { lz->coder = lzma_alloc(sizeof(lzma_lzma1_decoder), allocator); @@ -1000,7 +1000,6 @@ lzma_lzma_decoder_create(lzma_lz_decoder *lz, const lzma_allocator *allocator, // All dictionary sizes are OK here. LZ decoder will take care of // the special cases. - const lzma_options_lzma *options = opt; lz_options->dict_size = options->dict_size; lz_options->preset_dict = options->preset_dict; lz_options->preset_dict_size = options->preset_dict_size; @@ -1014,16 +1013,40 @@ lzma_lzma_decoder_create(lzma_lz_decoder *lz, const lzma_allocator *allocator, /// the LZ initialization). static lzma_ret lzma_decoder_init(lzma_lz_decoder *lz, const lzma_allocator *allocator, - const void *options, lzma_lz_options *lz_options) + lzma_vli id, const void *options, lzma_lz_options *lz_options) { if (!is_lclppb_valid(options)) return LZMA_PROG_ERROR; + lzma_vli uncomp_size = LZMA_VLI_UNKNOWN; + bool allow_eopm = true; + + if (id == LZMA_FILTER_LZMA1EXT) { + const lzma_options_lzma *opt = options; + + // Only one flag is supported. + if (opt->ext_flags & ~LZMA_LZMA1EXT_ALLOW_EOPM) + return LZMA_OPTIONS_ERROR; + + // FIXME? Using lzma_vli instead of uint64_t is weird because + // this has nothing to do with .xz headers and variable-length + // integer encoding. On the other hand, using LZMA_VLI_UNKNOWN + // instead of UINT64_MAX is clearer when unknown size is + // meant. A problem with using lzma_vli is that now we + // allow > LZMA_VLI_MAX which is fine in this file but + // it's still confusing. Note that alone_decoder.c also + // allows > LZMA_VLI_MAX when setting uncompressed size. + uncomp_size = opt->ext_size_low + + ((uint64_t)(opt->ext_size_high) << 32); + allow_eopm = (opt->ext_flags & LZMA_LZMA1EXT_ALLOW_EOPM) != 0 + || uncomp_size == LZMA_VLI_UNKNOWN; + } + return_if_error(lzma_lzma_decoder_create( lz, allocator, options, lz_options)); lzma_decoder_reset(lz->coder, options); - lzma_decoder_uncompressed(lz->coder, LZMA_VLI_UNKNOWN, true); + lzma_decoder_uncompressed(lz->coder, uncomp_size, allow_eopm); return LZMA_OK; } diff --git a/src/liblzma/lzma/lzma_decoder.h b/src/liblzma/lzma/lzma_decoder.h index fa8ecb23e43c..1427bc2461f4 100644 --- a/src/liblzma/lzma/lzma_decoder.h +++ b/src/liblzma/lzma/lzma_decoder.h @@ -42,7 +42,7 @@ extern bool lzma_lzma_lclppb_decode( /// LZMA2 decoders. extern lzma_ret lzma_lzma_decoder_create( lzma_lz_decoder *lz, const lzma_allocator *allocator, - const void *opt, lzma_lz_options *lz_options); + const lzma_options_lzma *opt, lzma_lz_options *lz_options); /// Gets memory usage without validating lc/lp/pb. This is used by LZMA2 /// decoder, because raw LZMA2 decoding doesn't need lc/lp/pb. diff --git a/src/liblzma/lzma/lzma_encoder.c b/src/liblzma/lzma/lzma_encoder.c index bbbb5d2d7cdf..dc62f44f1bfb 100644 --- a/src/liblzma/lzma/lzma_encoder.c +++ b/src/liblzma/lzma/lzma_encoder.c @@ -268,6 +268,7 @@ static bool encode_init(lzma_lzma1_encoder *coder, lzma_mf *mf) { assert(mf_position(mf) == 0); + assert(coder->uncomp_size == 0); if (mf->read_pos == mf->read_limit) { if (mf->action == LZMA_RUN) @@ -283,6 +284,7 @@ encode_init(lzma_lzma1_encoder *coder, lzma_mf *mf) mf->read_ahead = 0; rc_bit(&coder->rc, &coder->is_match[0][0], 0); rc_bittree(&coder->rc, coder->literal[0], 8, mf->buffer[0]); + ++coder->uncomp_size; } // Initialization is done (except if empty file). @@ -317,21 +319,28 @@ lzma_lzma_encode(lzma_lzma1_encoder *restrict coder, lzma_mf *restrict mf, if (!coder->is_initialized && !encode_init(coder, mf)) return LZMA_OK; - // Get the lowest bits of the uncompressed offset from the LZ layer. - uint32_t position = mf_position(mf); + // Encode pending output bytes from the range encoder. + // At the start of the stream, encode_init() encodes one literal. + // Later there can be pending output only with LZMA1 because LZMA2 + // ensures that there is always enough output space. Thus when using + // LZMA2, rc_encode() calls in this function will always return false. + if (rc_encode(&coder->rc, out, out_pos, out_size)) { + // We don't get here with LZMA2. + assert(limit == UINT32_MAX); + return LZMA_OK; + } - while (true) { - // Encode pending bits, if any. Calling this before encoding - // the next symbol is needed only with plain LZMA, since - // LZMA2 always provides big enough buffer to flush - // everything out from the range encoder. For the same reason, - // rc_encode() never returns true when this function is used - // as part of LZMA2 encoder. - if (rc_encode(&coder->rc, out, out_pos, out_size)) { - assert(limit == UINT32_MAX); - return LZMA_OK; - } + // If the range encoder was flushed in an earlier call to this + // function but there wasn't enough output buffer space, those + // bytes would have now been encoded by the above rc_encode() call + // and the stream has now been finished. This can only happen with + // LZMA1 as LZMA2 always provides enough output buffer space. + if (coder->is_flushed) { + assert(limit == UINT32_MAX); + return LZMA_STREAM_END; + } + while (true) { // With LZMA2 we need to take care that compressed size of // a chunk doesn't get too big. // FIXME? Check if this could be improved. @@ -365,37 +374,64 @@ lzma_lzma_encode(lzma_lzma1_encoder *restrict coder, lzma_mf *restrict mf, if (coder->fast_mode) lzma_lzma_optimum_fast(coder, mf, &back, &len); else - lzma_lzma_optimum_normal( - coder, mf, &back, &len, position); + lzma_lzma_optimum_normal(coder, mf, &back, &len, + (uint32_t)(coder->uncomp_size)); + + encode_symbol(coder, mf, back, len, + (uint32_t)(coder->uncomp_size)); + + // If output size limiting is active (out_limit != 0), check + // if encoding this LZMA symbol would make the output size + // exceed the specified limit. + if (coder->out_limit != 0 && rc_encode_dummy( + &coder->rc, coder->out_limit)) { + // The most recent LZMA symbol would make the output + // too big. Throw it away. + rc_forget(&coder->rc); + + // FIXME: Tell the LZ layer to not read more input as + // it would be waste of time. This doesn't matter if + // output-size-limited encoding is done with a single + // call though. - encode_symbol(coder, mf, back, len, position); - - position += len; - } - - if (!coder->is_flushed) { - coder->is_flushed = true; - - // We don't support encoding plain LZMA streams without EOPM, - // and LZMA2 doesn't use EOPM at LZMA level. - if (limit == UINT32_MAX) - encode_eopm(coder, position); + break; + } - // Flush the remaining bytes from the range encoder. - rc_flush(&coder->rc); + // This symbol will be encoded so update the uncompressed size. + coder->uncomp_size += len; - // Copy the remaining bytes to the output buffer. If there - // isn't enough output space, we will copy out the remaining - // bytes on the next call to this function by using - // the rc_encode() call in the encoding loop above. + // Encode the LZMA symbol. if (rc_encode(&coder->rc, out, out_pos, out_size)) { + // Once again, this can only happen with LZMA1. assert(limit == UINT32_MAX); return LZMA_OK; } } - // Make it ready for the next LZMA2 chunk. - coder->is_flushed = false; + // Make the uncompressed size available to the application. + if (coder->uncomp_size_ptr != NULL) + *coder->uncomp_size_ptr = coder->uncomp_size; + + // LZMA2 doesn't use EOPM at LZMA level. + // + // Plain LZMA streams without EOPM aren't supported except when + // output size limiting is enabled. + if (coder->use_eopm) + encode_eopm(coder, (uint32_t)(coder->uncomp_size)); + + // Flush the remaining bytes from the range encoder. + rc_flush(&coder->rc); + + // Copy the remaining bytes to the output buffer. If there + // isn't enough output space, we will copy out the remaining + // bytes on the next call to this function. + if (rc_encode(&coder->rc, out, out_pos, out_size)) { + // This cannot happen with LZMA2. + assert(limit == UINT32_MAX); + + coder->is_flushed = true; + return LZMA_OK; + } return LZMA_STREAM_END; } @@ -414,6 +450,23 @@ lzma_encode(void *coder, lzma_mf *restrict mf, } +static lzma_ret +lzma_lzma_set_out_limit( + void *coder_ptr, uint64_t *uncomp_size, uint64_t out_limit) +{ + // Minimum output size is 5 bytes but that cannot hold any output + // so we use 6 bytes. + if (out_limit < 6) + return LZMA_BUF_ERROR; + + lzma_lzma1_encoder *coder = coder_ptr; + coder->out_limit = out_limit; + coder->uncomp_size_ptr = uncomp_size; + coder->use_eopm = false; + return LZMA_OK; +} + + //////////////////// // Initialization // //////////////////// @@ -440,7 +493,8 @@ set_lz_options(lzma_lz_options *lz_options, const lzma_options_lzma *options) lz_options->dict_size = options->dict_size; lz_options->after_size = LOOP_INPUT_MAX; lz_options->match_len_max = MATCH_LEN_MAX; - lz_options->nice_len = options->nice_len; + lz_options->nice_len = my_max(mf_get_hash_bytes(options->mf), + options->nice_len); lz_options->match_finder = options->mf; lz_options->depth = options->depth; lz_options->preset_dict = options->preset_dict; @@ -546,10 +600,13 @@ lzma_lzma_encoder_reset(lzma_lzma1_encoder *coder, extern lzma_ret -lzma_lzma_encoder_create(void **coder_ptr, - const lzma_allocator *allocator, - const lzma_options_lzma *options, lzma_lz_options *lz_options) +lzma_lzma_encoder_create(void **coder_ptr, const lzma_allocator *allocator, + lzma_vli id, const lzma_options_lzma *options, + lzma_lz_options *lz_options) { + assert(id == LZMA_FILTER_LZMA1 || id == LZMA_FILTER_LZMA1EXT + || id == LZMA_FILTER_LZMA2); + // Allocate lzma_lzma1_encoder if it wasn't already allocated. if (*coder_ptr == NULL) { *coder_ptr = lzma_alloc(sizeof(lzma_lzma1_encoder), allocator); @@ -591,10 +648,14 @@ lzma_lzma_encoder_create(void **coder_ptr, coder->dist_table_size = log_size * 2; // Length encoders' price table size + const uint32_t nice_len = my_max( + mf_get_hash_bytes(options->mf), + options->nice_len); + coder->match_len_encoder.table_size - = options->nice_len + 1 - MATCH_LEN_MIN; + = nice_len + 1 - MATCH_LEN_MIN; coder->rep_len_encoder.table_size - = options->nice_len + 1 - MATCH_LEN_MIN; + = nice_len + 1 - MATCH_LEN_MIN; break; } @@ -609,6 +670,37 @@ lzma_lzma_encoder_create(void **coder_ptr, coder->is_initialized = options->preset_dict != NULL && options->preset_dict_size > 0; coder->is_flushed = false; + coder->uncomp_size = 0; + coder->uncomp_size_ptr = NULL; + + // Output size limitting is disabled by default. + coder->out_limit = 0; + + // Determine if end marker is wanted: + // - It is never used with LZMA2. + // - It is always used with LZMA_FILTER_LZMA1 (unless + // lzma_lzma_set_out_limit() is called later). + // - LZMA_FILTER_LZMA1EXT has a flag for it in the options. + coder->use_eopm = (id == LZMA_FILTER_LZMA1); + if (id == LZMA_FILTER_LZMA1EXT) { + // Check if unsupported flags are present. + if (options->ext_flags & ~LZMA_LZMA1EXT_ALLOW_EOPM) + return LZMA_OPTIONS_ERROR; + + coder->use_eopm = (options->ext_flags + & LZMA_LZMA1EXT_ALLOW_EOPM) != 0; + + // TODO? As long as there are no filters that change the size + // of the data, it is enough to look at lzma_stream.total_in + // after encoding has been finished to know the uncompressed + // size of the LZMA1 stream. But in the future there could be + // filters that change the size of the data and then total_in + // doesn't work as the LZMA1 stream size might be different + // due to another filter in the chain. The problem is simple + // to solve: Add another flag to ext_flags and then set + // coder->uncomp_size_ptr to the address stored in + // lzma_options_lzma.reserved_ptr2 (or _ptr1). + } set_lz_options(lz_options, options); @@ -618,11 +710,12 @@ lzma_lzma_encoder_create(void **coder_ptr, static lzma_ret lzma_encoder_init(lzma_lz_encoder *lz, const lzma_allocator *allocator, - const void *options, lzma_lz_options *lz_options) + lzma_vli id, const void *options, lzma_lz_options *lz_options) { lz->code = &lzma_encode; + lz->set_out_limit = &lzma_lzma_set_out_limit; return lzma_lzma_encoder_create( - &lz->coder, allocator, options, lz_options); + &lz->coder, allocator, id, options, lz_options); } diff --git a/src/liblzma/lzma/lzma_encoder.h b/src/liblzma/lzma/lzma_encoder.h index 6cfdf228bf5c..84d8c9163f2d 100644 --- a/src/liblzma/lzma/lzma_encoder.h +++ b/src/liblzma/lzma/lzma_encoder.h @@ -40,7 +40,8 @@ extern bool lzma_lzma_lclppb_encode( /// Initializes raw LZMA encoder; this is used by LZMA2. extern lzma_ret lzma_lzma_encoder_create( void **coder_ptr, const lzma_allocator *allocator, - const lzma_options_lzma *options, lzma_lz_options *lz_options); + lzma_vli id, const lzma_options_lzma *options, + lzma_lz_options *lz_options); /// Resets an already initialized LZMA encoder; this is used by LZMA2. diff --git a/src/liblzma/lzma/lzma_encoder_private.h b/src/liblzma/lzma/lzma_encoder_private.h index 2e34aace16ef..b228c5776173 100644 --- a/src/liblzma/lzma/lzma_encoder_private.h +++ b/src/liblzma/lzma/lzma_encoder_private.h @@ -72,6 +72,18 @@ struct lzma_lzma1_encoder_s { /// Range encoder lzma_range_encoder rc; + /// Uncompressed size (doesn't include possible preset dictionary) + uint64_t uncomp_size; + + /// If non-zero, produce at most this much output. + /// Some input may then be missing from the output. + uint64_t out_limit; + + /// If the above out_limit is non-zero, *uncomp_size_ptr is set to + /// the amount of uncompressed data that we were able to fit + /// in the output buffer. + uint64_t *uncomp_size_ptr; + /// State lzma_lzma_state state; @@ -99,6 +111,9 @@ struct lzma_lzma1_encoder_s { /// have been written to the output buffer yet. bool is_flushed; + /// True if end of payload marker will be written. + bool use_eopm; + uint32_t pos_mask; ///< (1 << pos_bits) - 1 uint32_t literal_context_bits; uint32_t literal_pos_mask; diff --git a/src/liblzma/rangecoder/range_encoder.h b/src/liblzma/rangecoder/range_encoder.h index 1e1c36995b63..d794eabbccea 100644 --- a/src/liblzma/rangecoder/range_encoder.h +++ b/src/liblzma/rangecoder/range_encoder.h @@ -19,9 +19,9 @@ /// Maximum number of symbols that can be put pending into lzma_range_encoder -/// structure between calls to lzma_rc_encode(). For LZMA, 52+5 is enough +/// structure between calls to lzma_rc_encode(). For LZMA, 48+5 is enough /// (match with big distance and length followed by range encoder flush). -#define RC_SYMBOLS_MAX 58 +#define RC_SYMBOLS_MAX 53 typedef struct { @@ -30,6 +30,9 @@ typedef struct { uint32_t range; uint8_t cache; + /// Number of bytes written out by rc_encode() -> rc_shift_low() + uint64_t out_total; + /// Number of symbols in the tables size_t count; @@ -58,12 +61,22 @@ rc_reset(lzma_range_encoder *rc) rc->cache_size = 1; rc->range = UINT32_MAX; rc->cache = 0; + rc->out_total = 0; rc->count = 0; rc->pos = 0; } static inline void +rc_forget(lzma_range_encoder *rc) +{ + // This must not be called when rc_encode() is partially done. + assert(rc->pos == 0); + rc->count = 0; +} + + +static inline void rc_bit(lzma_range_encoder *rc, probability *prob, uint32_t bit) { rc->symbols[rc->count] = bit; @@ -132,6 +145,7 @@ rc_shift_low(lzma_range_encoder *rc, out[*out_pos] = rc->cache + (uint8_t)(rc->low >> 32); ++*out_pos; + ++rc->out_total; rc->cache = 0xFF; } while (--rc->cache_size != 0); @@ -146,6 +160,34 @@ rc_shift_low(lzma_range_encoder *rc, } +// NOTE: The last two arguments are uint64_t instead of size_t because in +// the dummy version these refer to the size of the whole range-encoded +// output stream, not just to the currently available output buffer space. +static inline bool +rc_shift_low_dummy(uint64_t *low, uint64_t *cache_size, uint8_t *cache, + uint64_t *out_pos, uint64_t out_size) +{ + if ((uint32_t)(*low) < (uint32_t)(0xFF000000) + || (uint32_t)(*low >> 32) != 0) { + do { + if (*out_pos == out_size) + return true; + + ++*out_pos; + *cache = 0xFF; + + } while (--*cache_size != 0); + + *cache = (*low >> 24) & 0xFF; + } + + ++*cache_size; + *low = (*low & 0x00FFFFFF) << RC_SHIFT_BITS; + + return false; +} + + static inline bool rc_encode(lzma_range_encoder *rc, uint8_t *out, size_t *out_pos, size_t out_size) @@ -222,6 +264,83 @@ rc_encode(lzma_range_encoder *rc, } +static inline bool +rc_encode_dummy(const lzma_range_encoder *rc, uint64_t out_limit) +{ + assert(rc->count <= RC_SYMBOLS_MAX); + + uint64_t low = rc->low; + uint64_t cache_size = rc->cache_size; + uint32_t range = rc->range; + uint8_t cache = rc->cache; + uint64_t out_pos = rc->out_total; + + size_t pos = rc->pos; + + while (true) { + // Normalize + if (range < RC_TOP_VALUE) { + if (rc_shift_low_dummy(&low, &cache_size, &cache, + &out_pos, out_limit)) + return true; + + range <<= RC_SHIFT_BITS; + } + + // This check is here because the normalization above must + // be done before flushing the last bytes. + if (pos == rc->count) + break; + + // Encode a bit + switch (rc->symbols[pos]) { + case RC_BIT_0: { + probability prob = *rc->probs[pos]; + range = (range >> RC_BIT_MODEL_TOTAL_BITS) + * prob; + break; + } + + case RC_BIT_1: { + probability prob = *rc->probs[pos]; + const uint32_t bound = prob * (range + >> RC_BIT_MODEL_TOTAL_BITS); + low += bound; + range -= bound; + break; + } + + case RC_DIRECT_0: + range >>= 1; + break; + + case RC_DIRECT_1: + range >>= 1; + low += range; + break; + + case RC_FLUSH: + default: + assert(0); + break; + } + + ++pos; + } + + // Flush the last bytes. This isn't in rc->symbols[] so we do + // it after the above loop to take into account the size of + // the flushing that will be done at the end of the stream. + for (pos = 0; pos < 5; ++pos) { + if (rc_shift_low_dummy(&low, &cache_size, + &cache, &out_pos, out_limit)) + return true; + } + + return false; +} + + static inline uint64_t rc_pending(const lzma_range_encoder *rc) { diff --git a/src/liblzma/simple/arm.c b/src/liblzma/simple/arm.c index ff5073ae58f3..6e53970d2f27 100644 --- a/src/liblzma/simple/arm.c +++ b/src/liblzma/simple/arm.c @@ -53,6 +53,7 @@ arm_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, } +#ifdef HAVE_ENCODER_ARM extern lzma_ret lzma_simple_arm_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -60,8 +61,10 @@ lzma_simple_arm_encoder_init(lzma_next_coder *next, { return arm_coder_init(next, allocator, filters, true); } +#endif +#ifdef HAVE_DECODER_ARM extern lzma_ret lzma_simple_arm_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -69,3 +72,4 @@ lzma_simple_arm_decoder_init(lzma_next_coder *next, { return arm_coder_init(next, allocator, filters, false); } +#endif diff --git a/src/liblzma/simple/arm64.c b/src/liblzma/simple/arm64.c new file mode 100644 index 000000000000..5e7f26562d5a --- /dev/null +++ b/src/liblzma/simple/arm64.c @@ -0,0 +1,136 @@ +/////////////////////////////////////////////////////////////////////////////// +// +/// \file arm64.c +/// \brief Filter for ARM64 binaries +/// +/// This converts ARM64 relative addresses in the BL and ADRP immediates +/// to absolute values to increase redundancy of ARM64 code. +/// +/// Converting B or ADR instructions was also tested but it's not useful. +/// A majority of the jumps for the B instruction are very small (+/- 0xFF). +/// These are typical for loops and if-statements. Encoding them to their +/// absolute address reduces redundancy since many of the small relative +/// jump values are repeated, but very few of the absolute addresses are. +// +// Authors: Lasse Collin +// Jia Tan +// +// This file has been put into the public domain. +// You can do whatever you want with this file. +// +/////////////////////////////////////////////////////////////////////////////// + +#include "simple_private.h" + + +static size_t +arm64_code(void *simple lzma_attribute((__unused__)), + uint32_t now_pos, bool is_encoder, + uint8_t *buffer, size_t size) +{ + size_t i; + + // Clang 14.0.6 on x86-64 makes this four times bigger and 40 % slower + // with auto-vectorization that is enabled by default with -O2. + // Such vectorization bloat happens with -O2 when targeting ARM64 too + // but performance hasn't been tested. +#ifdef __clang__ +# pragma clang loop vectorize(disable) +#endif + for (i = 0; i + 4 <= size; i += 4) { + uint32_t pc = (uint32_t)(now_pos + i); + uint32_t instr = read32le(buffer + i); + + if ((instr >> 26) == 0x25) { + // BL instruction: + // The full 26-bit immediate is converted. + // The range is +/-128 MiB. + // + // Using the full range is helps quite a lot with + // big executables. Smaller range would reduce false + // positives in non-code sections of the input though + // so this is a compromise that slightly favors big + // files. With the full range only six bits of the 32 + // need to match to trigger a conversion. + const uint32_t src = instr; + instr = 0x94000000; + + pc >>= 2; + if (!is_encoder) + pc = 0U - pc; + + instr |= (src + pc) & 0x03FFFFFF; + write32le(buffer + i, instr); + + } else if ((instr & 0x9F000000) == 0x90000000) { + // ADRP instruction: + // Only values in the range +/-512 MiB are converted. + // + // Using less than the full +/-4 GiB range reduces + // false positives on non-code sections of the input + // while being excellent for executables up to 512 MiB. + // The positive effect of ADRP conversion is smaller + // than that of BL but it also doesn't hurt so much in + // non-code sections of input because, with +/-512 MiB + // range, nine bits of 32 need to match to trigger a + // conversion (two 10-bit match choices = 9 bits). + const uint32_t src = ((instr >> 29) & 3) + | ((instr >> 3) & 0x001FFFFC); + + // With the addition only one branch is needed to + // check the +/- range. This is usually false when + // processing ARM64 code so branch prediction will + // handle it well in terms of performance. + // + //if ((src & 0x001E0000) != 0 + // && (src & 0x001E0000) != 0x001E0000) + if ((src + 0x00020000) & 0x001C0000) + continue; + + instr &= 0x9000001F; + + pc >>= 12; + if (!is_encoder) + pc = 0U - pc; + + const uint32_t dest = src + pc; + instr |= (dest & 3) << 29; + instr |= (dest & 0x0003FFFC) << 3; + instr |= (0U - (dest & 0x00020000)) & 0x00E00000; + write32le(buffer + i, instr); + } + } + + return i; +} + + +static lzma_ret +arm64_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, + const lzma_filter_info *filters, bool is_encoder) +{ + return lzma_simple_coder_init(next, allocator, filters, + &arm64_code, 0, 4, 4, is_encoder); +} + + +#ifdef HAVE_ENCODER_ARM64 +extern lzma_ret +lzma_simple_arm64_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return arm64_coder_init(next, allocator, filters, true); +} +#endif + + +#ifdef HAVE_DECODER_ARM64 +extern lzma_ret +lzma_simple_arm64_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters) +{ + return arm64_coder_init(next, allocator, filters, false); +} +#endif diff --git a/src/liblzma/simple/armthumb.c b/src/liblzma/simple/armthumb.c index a8da334a04f8..25d8dbd4f36e 100644 --- a/src/liblzma/simple/armthumb.c +++ b/src/liblzma/simple/armthumb.c @@ -58,6 +58,7 @@ armthumb_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, } +#ifdef HAVE_ENCODER_ARMTHUMB extern lzma_ret lzma_simple_armthumb_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -65,8 +66,10 @@ lzma_simple_armthumb_encoder_init(lzma_next_coder *next, { return armthumb_coder_init(next, allocator, filters, true); } +#endif +#ifdef HAVE_DECODER_ARMTHUMB extern lzma_ret lzma_simple_armthumb_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -74,3 +77,4 @@ lzma_simple_armthumb_decoder_init(lzma_next_coder *next, { return armthumb_coder_init(next, allocator, filters, false); } +#endif diff --git a/src/liblzma/simple/ia64.c b/src/liblzma/simple/ia64.c index 6492d0a38469..692b0a295ef2 100644 --- a/src/liblzma/simple/ia64.c +++ b/src/liblzma/simple/ia64.c @@ -94,6 +94,7 @@ ia64_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, } +#ifdef HAVE_ENCODER_IA64 extern lzma_ret lzma_simple_ia64_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -101,8 +102,10 @@ lzma_simple_ia64_encoder_init(lzma_next_coder *next, { return ia64_coder_init(next, allocator, filters, true); } +#endif +#ifdef HAVE_DECODER_IA64 extern lzma_ret lzma_simple_ia64_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -110,3 +113,4 @@ lzma_simple_ia64_decoder_init(lzma_next_coder *next, { return ia64_coder_init(next, allocator, filters, false); } +#endif diff --git a/src/liblzma/simple/powerpc.c b/src/liblzma/simple/powerpc.c index 0b60e9b3fe3c..3a340fd171a5 100644 --- a/src/liblzma/simple/powerpc.c +++ b/src/liblzma/simple/powerpc.c @@ -58,6 +58,7 @@ powerpc_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, } +#ifdef HAVE_ENCODER_POWERPC extern lzma_ret lzma_simple_powerpc_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -65,8 +66,10 @@ lzma_simple_powerpc_encoder_init(lzma_next_coder *next, { return powerpc_coder_init(next, allocator, filters, true); } +#endif +#ifdef HAVE_DECODER_POWERPC extern lzma_ret lzma_simple_powerpc_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -74,3 +77,4 @@ lzma_simple_powerpc_decoder_init(lzma_next_coder *next, { return powerpc_coder_init(next, allocator, filters, false); } +#endif diff --git a/src/liblzma/simple/simple_coder.h b/src/liblzma/simple/simple_coder.h index 19c2ee03affd..668a5092ad5e 100644 --- a/src/liblzma/simple/simple_coder.h +++ b/src/liblzma/simple/simple_coder.h @@ -61,6 +61,15 @@ extern lzma_ret lzma_simple_armthumb_decoder_init(lzma_next_coder *next, const lzma_filter_info *filters); +extern lzma_ret lzma_simple_arm64_encoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + +extern lzma_ret lzma_simple_arm64_decoder_init(lzma_next_coder *next, + const lzma_allocator *allocator, + const lzma_filter_info *filters); + + extern lzma_ret lzma_simple_sparc_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, const lzma_filter_info *filters); diff --git a/src/liblzma/simple/sparc.c b/src/liblzma/simple/sparc.c index 74b2655f36ea..bad8492ebc06 100644 --- a/src/liblzma/simple/sparc.c +++ b/src/liblzma/simple/sparc.c @@ -65,6 +65,7 @@ sparc_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, } +#ifdef HAVE_ENCODER_SPARC extern lzma_ret lzma_simple_sparc_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -72,8 +73,10 @@ lzma_simple_sparc_encoder_init(lzma_next_coder *next, { return sparc_coder_init(next, allocator, filters, true); } +#endif +#ifdef HAVE_DECODER_SPARC extern lzma_ret lzma_simple_sparc_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -81,3 +84,4 @@ lzma_simple_sparc_decoder_init(lzma_next_coder *next, { return sparc_coder_init(next, allocator, filters, false); } +#endif diff --git a/src/liblzma/simple/x86.c b/src/liblzma/simple/x86.c index 0e78909ccc0d..232b29542e66 100644 --- a/src/liblzma/simple/x86.c +++ b/src/liblzma/simple/x86.c @@ -141,6 +141,7 @@ x86_coder_init(lzma_next_coder *next, const lzma_allocator *allocator, } +#ifdef HAVE_ENCODER_X86 extern lzma_ret lzma_simple_x86_encoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -148,8 +149,10 @@ lzma_simple_x86_encoder_init(lzma_next_coder *next, { return x86_coder_init(next, allocator, filters, true); } +#endif +#ifdef HAVE_DECODER_X86 extern lzma_ret lzma_simple_x86_decoder_init(lzma_next_coder *next, const lzma_allocator *allocator, @@ -157,3 +160,4 @@ lzma_simple_x86_decoder_init(lzma_next_coder *next, { return x86_coder_init(next, allocator, filters, false); } +#endif diff --git a/src/xz/args.c b/src/xz/args.c index 9238fb32ec00..b0a1174a25bd 100644 --- a/src/xz/args.c +++ b/src/xz/args.c @@ -29,19 +29,29 @@ bool opt_ignore_check = false; const char stdin_filename[] = "(stdin)"; -/// Parse and set the memory usage limit for compression and/or decompression. +/// Parse and set the memory usage limit for compression, decompression, +/// and/or multithreaded decompression. static void -parse_memlimit(const char *name, const char *name_percentage, char *str, - bool set_compress, bool set_decompress) +parse_memlimit(const char *name, const char *name_percentage, const char *str, + bool set_compress, bool set_decompress, bool set_mtdec) { bool is_percentage = false; uint64_t value; const size_t len = strlen(str); if (len > 0 && str[len - 1] == '%') { - str[len - 1] = '\0'; + // Make a copy so that we can get rid of %. + // + // In the past str wasn't const and we modified it directly + // but that modified argv[] and thus affected what was visible + // in "ps auxf" or similar tools which was confusing. For + // example, --memlimit=50% would show up as --memlimit=50 + // since the percent sign was overwritten here. + char *s = xstrdup(str); + s[len - 1] = '\0'; is_percentage = true; - value = str_to_uint64(name_percentage, str, 1, 100); + value = str_to_uint64(name_percentage, s, 1, 100); + free(s); } else { // On 32-bit systems, SIZE_MAX would make more sense than // UINT64_MAX. But use UINT64_MAX still so that scripts @@ -49,15 +59,19 @@ parse_memlimit(const char *name, const char *name_percentage, char *str, value = str_to_uint64(name, str, 0, UINT64_MAX); } - hardware_memlimit_set( - value, set_compress, set_decompress, is_percentage); + hardware_memlimit_set(value, set_compress, set_decompress, set_mtdec, + is_percentage); return; } static void -parse_block_list(char *str) +parse_block_list(const char *str_const) { + // We need a modifiable string in the for-loop. + char *str_start = xstrdup(str_const); + char *str = str_start; + // It must be non-empty and not begin with a comma. if (str[0] == '\0' || str[0] == ',') message_fatal(_("%s: Invalid argument to --block-list"), str); @@ -112,6 +126,8 @@ parse_block_list(char *str) // Terminate the array. opt_block_list[count] = 0; + + free(str_start); return; } @@ -125,6 +141,7 @@ parse_real(args_info *args, int argc, char **argv) OPT_IA64, OPT_ARM, OPT_ARMTHUMB, + OPT_ARM64, OPT_SPARC, OPT_DELTA, OPT_LZMA1, @@ -138,6 +155,7 @@ parse_real(args_info *args, int argc, char **argv) OPT_BLOCK_LIST, OPT_MEM_COMPRESS, OPT_MEM_DECOMPRESS, + OPT_MEM_MT_DECOMPRESS, OPT_NO_ADJUST, OPT_INFO_MEMORY, OPT_ROBOT, @@ -176,6 +194,7 @@ parse_real(args_info *args, int argc, char **argv) { "block-list", required_argument, NULL, OPT_BLOCK_LIST }, { "memlimit-compress", required_argument, NULL, OPT_MEM_COMPRESS }, { "memlimit-decompress", required_argument, NULL, OPT_MEM_DECOMPRESS }, + { "memlimit-mt-decompress", required_argument, NULL, OPT_MEM_MT_DECOMPRESS }, { "memlimit", required_argument, NULL, 'M' }, { "memory", required_argument, NULL, 'M' }, // Old alias { "no-adjust", no_argument, NULL, OPT_NO_ADJUST }, @@ -194,6 +213,7 @@ parse_real(args_info *args, int argc, char **argv) { "ia64", optional_argument, NULL, OPT_IA64 }, { "arm", optional_argument, NULL, OPT_ARM }, { "armthumb", optional_argument, NULL, OPT_ARMTHUMB }, + { "arm64", optional_argument, NULL, OPT_ARM64 }, { "sparc", optional_argument, NULL, OPT_SPARC }, { "delta", optional_argument, NULL, OPT_DELTA }, @@ -225,20 +245,27 @@ parse_real(args_info *args, int argc, char **argv) case OPT_MEM_COMPRESS: parse_memlimit("memlimit-compress", "memlimit-compress%", optarg, - true, false); + true, false, false); break; // --memlimit-decompress case OPT_MEM_DECOMPRESS: parse_memlimit("memlimit-decompress", "memlimit-decompress%", optarg, - false, true); + false, true, false); + break; + + // --memlimit-mt-decompress + case OPT_MEM_MT_DECOMPRESS: + parse_memlimit("memlimit-mt-decompress", + "memlimit-mt-decompress%", optarg, + false, false, true); break; // --memlimit case 'M': parse_memlimit("memlimit", "memlimit%", optarg, - true, true); + true, true, true); break; // --suffix @@ -246,11 +273,23 @@ parse_real(args_info *args, int argc, char **argv) suffix_set(optarg); break; - case 'T': + case 'T': { + // Since xz 5.4.0: Ignore leading '+' first. + const char *s = optarg; + if (optarg[0] == '+') + ++s; + // The max is from src/liblzma/common/common.h. - hardware_threads_set(str_to_uint64("threads", - optarg, 0, 16384)); + uint32_t t = str_to_uint64("threads", s, 0, 16384); + + // If leading '+' was used then use multi-threaded + // mode even if exactly one thread was specified. + if (t == 1 && optarg[0] == '+') + t = UINT32_MAX; + + hardware_threads_set(t); break; + } // --version case 'V': @@ -360,6 +399,11 @@ parse_real(args_info *args, int argc, char **argv) options_bcj(optarg)); break; + case OPT_ARM64: + coder_add_filter(LZMA_FILTER_ARM64, + options_bcj(optarg)); + break; + case OPT_SPARC: coder_add_filter(LZMA_FILTER_SPARC, options_bcj(optarg)); @@ -395,8 +439,9 @@ parse_real(args_info *args, int argc, char **argv) { "xz", FORMAT_XZ }, { "lzma", FORMAT_LZMA }, { "alone", FORMAT_LZMA }, - // { "gzip", FORMAT_GZIP }, - // { "gz", FORMAT_GZIP }, +#ifdef HAVE_LZIP_DECODER + { "lzip", FORMAT_LZIP }, +#endif { "raw", FORMAT_RAW }, }; @@ -475,7 +520,7 @@ parse_real(args_info *args, int argc, char **argv) "or `--files0'.")); if (optarg == NULL) { - args->files_name = (char *)stdin_filename; + args->files_name = stdin_filename; args->files_file = stdin; } else { args->files_name = optarg; @@ -651,6 +696,12 @@ args_parse(args_info *args, int argc, char **argv) "at build time")); #endif +#ifdef HAVE_LZIP_DECODER + if (opt_mode == MODE_COMPRESS && opt_format == FORMAT_LZIP) + message_fatal(_("Compression of lzip files (.lz) " + "is not supported")); +#endif + // Never remove the source file when the destination is not on disk. // In test mode the data is written nowhere, but setting opt_stdout // will make the rest of the code behave well. diff --git a/src/xz/args.h b/src/xz/args.h index 46a8e8edfc97..a1a5930ae5b1 100644 --- a/src/xz/args.h +++ b/src/xz/args.h @@ -19,7 +19,7 @@ typedef struct { /// Name of the file from which to read filenames. This is NULL /// if --files or --files0 was not used. - char *files_name; + const char *files_name; /// File opened for reading from which filenames are read. This is /// non-NULL only if files_name is non-NULL. diff --git a/src/xz/coder.c b/src/xz/coder.c index 4f51e8d5c0e0..05f228882b73 100644 --- a/src/xz/coder.c +++ b/src/xz/coder.c @@ -51,7 +51,12 @@ static lzma_check check; /// This becomes false if the --check=CHECK option is used. static bool check_default = true; -#if defined(HAVE_ENCODERS) && defined(MYTHREAD_ENABLED) +/// Indicates if unconsumed input is allowed to remain after +/// decoding has successfully finished. This is set for each file +/// in coder_init(). +static bool allow_trailing_input; + +#ifdef MYTHREAD_ENABLED static lzma_mt mt_options = { .flags = 0, .timeout = 300, @@ -136,6 +141,11 @@ memlimit_too_small(uint64_t memory_usage) extern void coder_set_compression_settings(void) { +#ifdef HAVE_LZIP_DECODER + // .lz compression isn't supported. + assert(opt_format != FORMAT_LZIP); +#endif + // The default check type is CRC64, but fallback to CRC32 // if CRC64 isn't supported by the copy of liblzma we are // using. CRC32 is always supported. @@ -211,7 +221,7 @@ coder_set_compression_settings(void) } } - if (hardware_threads_get() > 1) { + if (hardware_threads_is_mt()) { message(V_WARNING, _("Switching to single-threaded " "mode due to --flush-timeout")); hardware_threads_set(1); @@ -220,12 +230,16 @@ coder_set_compression_settings(void) // Get the memory usage. Note that if --format=raw was used, // we can be decompressing. - const uint64_t memory_limit = hardware_memlimit_get(opt_mode); + // + // If multithreaded .xz compression is done, this value will be + // replaced. + uint64_t memory_limit = hardware_memlimit_get(opt_mode); uint64_t memory_usage = UINT64_MAX; if (opt_mode == MODE_COMPRESS) { #ifdef HAVE_ENCODERS # ifdef MYTHREAD_ENABLED - if (opt_format == FORMAT_XZ && hardware_threads_get() > 1) { + if (opt_format == FORMAT_XZ && hardware_threads_is_mt()) { + memory_limit = hardware_memlimit_mtenc_get(); mt_options.threads = hardware_threads_get(); mt_options.block_size = opt_block_size; mt_options.check = check; @@ -269,47 +283,90 @@ coder_set_compression_settings(void) if (memory_usage <= memory_limit) return; - // If --no-adjust was used or we didn't find LZMA1 or - // LZMA2 as the last filter, give an error immediately. - // --format=raw implies --no-adjust. - if (!opt_auto_adjust || opt_format == FORMAT_RAW) + // With --format=raw settings are never adjusted to meet + // the memory usage limit. + if (opt_format == FORMAT_RAW) memlimit_too_small(memory_usage); assert(opt_mode == MODE_COMPRESS); #ifdef HAVE_ENCODERS # ifdef MYTHREAD_ENABLED - if (opt_format == FORMAT_XZ && mt_options.threads > 1) { + if (opt_format == FORMAT_XZ && hardware_threads_is_mt()) { // Try to reduce the number of threads before // adjusting the compression settings down. - do { - // FIXME? The real single-threaded mode has - // lower memory usage, but it's not comparable - // because it doesn't write the size info - // into Block Headers. - if (--mt_options.threads == 0) - memlimit_too_small(memory_usage); - + while (mt_options.threads > 1) { + // Reduce the number of threads by one and check + // the memory usage. + --mt_options.threads; memory_usage = lzma_stream_encoder_mt_memusage( &mt_options); if (memory_usage == UINT64_MAX) message_bug(); - } while (memory_usage > memory_limit); + if (memory_usage <= memory_limit) { + // The memory usage is now low enough. + message(V_WARNING, _("Reduced the number of " + "threads from %s to %s to not exceed " + "the memory usage limit of %s MiB"), + uint64_to_str( + hardware_threads_get(), 0), + uint64_to_str(mt_options.threads, 1), + uint64_to_str(round_up_to_mib( + memory_limit), 2)); + return; + } + } - message(V_WARNING, _("Adjusted the number of threads " - "from %s to %s to not exceed " - "the memory usage limit of %s MiB"), - uint64_to_str(hardware_threads_get(), 0), - uint64_to_str(mt_options.threads, 1), - uint64_to_str(round_up_to_mib( - memory_limit), 2)); + // If the memory usage limit is only a soft limit (automatic + // number of threads and no --memlimit-compress), the limit + // is only used to reduce the number of threads and once at + // just one thread, the limit is completely ignored. This + // way -T0 won't use insane amount of memory but at the same + // time the soft limit will never make xz fail and never make + // xz change settings that would affect the compressed output. + if (hardware_memlimit_mtenc_is_default()) { + message(V_WARNING, _("Reduced the number of threads " + "from %s to one. The automatic memory usage " + "limit of %s MiB is still being exceeded. " + "%s MiB of memory is required. " + "Continuing anyway."), + uint64_to_str(hardware_threads_get(), 0), + uint64_to_str( + round_up_to_mib(memory_limit), 1), + uint64_to_str( + round_up_to_mib(memory_usage), 2)); + return; + } + + // If --no-adjust was used, we cannot drop to single-threaded + // mode since it produces different compressed output. + // + // NOTE: In xz 5.2.x, --no-adjust also prevented reducing + // the number of threads. This changed in 5.3.3alpha. + if (!opt_auto_adjust) + memlimit_too_small(memory_usage); + + // Switch to single-threaded mode. It uses + // less memory than using one thread in + // the multithreaded mode but the output + // is also different. + hardware_threads_set(1); + memory_usage = lzma_raw_encoder_memusage(filters); + message(V_WARNING, _("Switching to single-threaded mode " + "to not exceed the memory usage limit of %s MiB"), + uint64_to_str(round_up_to_mib(memory_limit), 0)); } # endif if (memory_usage <= memory_limit) return; + // Don't adjust LZMA2 or LZMA1 dictionary size if --no-adjust + // was specified as that would change the compressed output. + if (!opt_auto_adjust) + memlimit_too_small(memory_usage); + // Look for the last filter if it is LZMA2 or LZMA1, so we can make // it use less RAM. With other filters we don't know what to do. size_t i = 0; @@ -423,6 +480,18 @@ is_format_lzma(void) return true; } + + +#ifdef HAVE_LZIP_DECODER +/// Return true if the data in in_buf seems to be in the .lz format. +static bool +is_format_lzip(void) +{ + static const uint8_t magic[4] = { 0x4C, 0x5A, 0x49, 0x50 }; + return strm.avail_in >= sizeof(magic) + && memcmp(in_buf.u8, magic, sizeof(magic)) == 0; +} +#endif #endif @@ -436,6 +505,12 @@ coder_init(file_pair *pair) { lzma_ret ret = LZMA_PROG_ERROR; + // In most cases if there is input left when coding finishes, + // something has gone wrong. Exceptions are --single-stream + // and decoding .lz files which can contain trailing non-.lz data. + // These will be handled later in this function. + allow_trailing_input = false; + if (opt_mode == MODE_COMPRESS) { #ifdef HAVE_ENCODERS switch (opt_format) { @@ -446,7 +521,7 @@ coder_init(file_pair *pair) case FORMAT_XZ: # ifdef MYTHREAD_ENABLED - if (hardware_threads_get() > 1) + if (hardware_threads_is_mt()) ret = lzma_stream_encoder_mt( &strm, &mt_options); else @@ -459,6 +534,14 @@ coder_init(file_pair *pair) ret = lzma_alone_encoder(&strm, filters[0].options); break; +# ifdef HAVE_LZIP_DECODER + case FORMAT_LZIP: + // args.c should disallow this. + assert(0); + ret = LZMA_PROG_ERROR; + break; +# endif + case FORMAT_RAW: ret = lzma_raw_encoder(&strm, filters); break; @@ -475,7 +558,9 @@ coder_init(file_pair *pair) else flags |= LZMA_TELL_UNSUPPORTED_CHECK; - if (!opt_single_stream) + if (opt_single_stream) + allow_trailing_input = true; + else flags |= LZMA_CONCATENATED; // We abuse FORMAT_AUTO to indicate unknown file format, @@ -484,8 +569,14 @@ coder_init(file_pair *pair) switch (opt_format) { case FORMAT_AUTO: + // .lz is checked before .lzma since .lzma detection + // is more complicated (no magic bytes). if (is_format_xz()) init_format = FORMAT_XZ; +# ifdef HAVE_LZIP_DECODER + else if (is_format_lzip()) + init_format = FORMAT_LZIP; +# endif else if (is_format_lzma()) init_format = FORMAT_LZMA; break; @@ -500,6 +591,13 @@ coder_init(file_pair *pair) init_format = FORMAT_LZMA; break; +# ifdef HAVE_LZIP_DECODER + case FORMAT_LZIP: + if (is_format_lzip()) + init_format = FORMAT_LZIP; + break; +# endif + case FORMAT_RAW: init_format = FORMAT_RAW; break; @@ -524,9 +622,31 @@ coder_init(file_pair *pair) break; case FORMAT_XZ: +# ifdef MYTHREAD_ENABLED + mt_options.flags = flags; + + mt_options.threads = hardware_threads_get(); + mt_options.memlimit_stop + = hardware_memlimit_get(MODE_DECOMPRESS); + + // If single-threaded mode was requested, set the + // memlimit for threading to zero. This forces the + // decoder to use single-threaded mode which matches + // the behavior of lzma_stream_decoder(). + // + // Otherwise use the limit for threaded decompression + // which has a sane default (users are still free to + // make it insanely high though). + mt_options.memlimit_threading + = mt_options.threads == 1 + ? 0 : hardware_memlimit_mtdec_get(); + + ret = lzma_stream_decoder_mt(&strm, &mt_options); +# else ret = lzma_stream_decoder(&strm, hardware_memlimit_get( MODE_DECOMPRESS), flags); +# endif break; case FORMAT_LZMA: @@ -535,6 +655,15 @@ coder_init(file_pair *pair) MODE_DECOMPRESS)); break; +# ifdef HAVE_LZIP_DECODER + case FORMAT_LZIP: + allow_trailing_input = true; + ret = lzma_lzip_decoder(&strm, + hardware_memlimit_get( + MODE_DECOMPRESS), flags); + break; +# endif + case FORMAT_RAW: // Memory usage has already been checked in // coder_set_compression_settings(). @@ -598,7 +727,7 @@ split_block(uint64_t *block_remaining, { if (*next_block_remaining > 0) { // The Block at *list_pos has previously been split up. - assert(hardware_threads_get() == 1); + assert(!hardware_threads_is_mt()); assert(opt_block_size > 0); assert(opt_block_list != NULL); @@ -626,7 +755,7 @@ split_block(uint64_t *block_remaining, // If in single-threaded mode, split up the Block if needed. // This is not needed in multi-threaded mode because liblzma // will do this due to how threaded encoding works. - if (hardware_threads_get() == 1 && opt_block_size > 0 + if (!hardware_threads_is_mt() && opt_block_size > 0 && *block_remaining > opt_block_size) { *next_block_remaining = *block_remaining - opt_block_size; @@ -686,7 +815,7 @@ coder_normal(file_pair *pair) // --block-size doesn't do anything here in threaded mode, // because the threaded encoder will take care of splitting // to fixed-sized Blocks. - if (hardware_threads_get() == 1 && opt_block_size > 0) + if (!hardware_threads_is_mt() && opt_block_size > 0) block_remaining = opt_block_size; // If --block-list was used, start with the first size. @@ -700,7 +829,7 @@ coder_normal(file_pair *pair) // mode the size info isn't written into Block Headers. if (opt_block_list != NULL) { if (block_remaining < opt_block_list[list_pos]) { - assert(hardware_threads_get() == 1); + assert(!hardware_threads_is_mt()); next_block_remaining = opt_block_list[list_pos] - block_remaining; } else { @@ -764,7 +893,7 @@ coder_normal(file_pair *pair) } else { // Start a new Block after LZMA_FULL_BARRIER. if (opt_block_list == NULL) { - assert(hardware_threads_get() == 1); + assert(!hardware_threads_is_mt()); assert(opt_block_size > 0); block_remaining = opt_block_size; } else { @@ -795,7 +924,7 @@ coder_normal(file_pair *pair) } if (ret == LZMA_STREAM_END) { - if (opt_single_stream) { + if (allow_trailing_input) { io_fix_src_pos(pair, strm.avail_in); success = true; break; @@ -803,7 +932,9 @@ coder_normal(file_pair *pair) // Check that there is no trailing garbage. // This is needed for LZMA_Alone and raw - // streams. + // streams. This is *not* done with .lz files + // as that format specifically requires + // allowing trailing garbage. if (strm.avail_in == 0 && !pair->src_eof) { // Try reading one more byte. // Hopefully we don't get any more diff --git a/src/xz/coder.h b/src/xz/coder.h index 583da8f68d50..9e69e5765ccb 100644 --- a/src/xz/coder.h +++ b/src/xz/coder.h @@ -23,7 +23,9 @@ enum format_type { FORMAT_AUTO, FORMAT_XZ, FORMAT_LZMA, - // HEADER_GZIP, +#ifdef HAVE_LZIP_DECODER + FORMAT_LZIP, +#endif FORMAT_RAW, }; diff --git a/src/xz/file_io.c b/src/xz/file_io.c index 41e4c2d893f8..a5841b370302 100644 --- a/src/xz/file_io.c +++ b/src/xz/file_io.c @@ -212,6 +212,17 @@ io_sandbox_enter(int src_fd) if (cap_enter()) goto error; +#elif defined(HAVE_PLEDGE) + // pledge() was introduced in OpenBSD 5.9. + // + // main() unconditionally calls pledge() with fairly relaxed + // promises which work in all situations. Here we make the + // sandbox more strict. + if (pledge("stdio", "")) + goto error; + + (void)src_fd; + #else # error ENABLE_SANDBOX is defined but no sandboxing method was found. #endif @@ -221,7 +232,7 @@ io_sandbox_enter(int src_fd) return; error: - message(V_DEBUG, _("Failed to enable the sandbox")); + message_fatal(_("Failed to enable the sandbox")); } #endif // ENABLE_SANDBOX @@ -748,8 +759,10 @@ error: extern file_pair * io_open_src(const char *src_name) { - if (is_empty_filename(src_name)) + if (src_name[0] == '\0') { + message_error(_("Empty filename, skipping")); return NULL; + } // Since we have only one file open at a time, we can use // a statically allocated structure. @@ -1195,16 +1208,36 @@ io_read(file_pair *pair, io_buf *buf, size_t size) extern bool -io_pread(file_pair *pair, io_buf *buf, size_t size, off_t pos) +io_seek_src(file_pair *pair, uint64_t pos) { - // Using lseek() and read() is more portable than pread() and - // for us it is as good as real pread(). - if (lseek(pair->src_fd, pos, SEEK_SET) != pos) { + // Caller must not attempt to seek past the end of the input file + // (seeking to 100 in a 100-byte file is seeking to the end of + // the file, not past the end of the file, and thus that is allowed). + // + // This also validates that pos can be safely cast to off_t. + if (pos > (uint64_t)(pair->src_st.st_size)) + message_bug(); + + if (lseek(pair->src_fd, (off_t)(pos), SEEK_SET) == -1) { message_error(_("%s: Error seeking the file: %s"), pair->src_name, strerror(errno)); return true; } + pair->src_eof = false; + + return false; +} + + +extern bool +io_pread(file_pair *pair, io_buf *buf, size_t size, uint64_t pos) +{ + // Using lseek() and read() is more portable than pread() and + // for us it is as good as real pread(). + if (io_seek_src(pair, pos)) + return true; + const size_t amount = io_read(pair, buf, size); if (amount == SIZE_MAX) return true; diff --git a/src/xz/file_io.h b/src/xz/file_io.h index c533d641badb..8a9e33630db3 100644 --- a/src/xz/file_io.h +++ b/src/xz/file_io.h @@ -139,6 +139,19 @@ extern size_t io_read(file_pair *pair, io_buf *buf, size_t size); extern void io_fix_src_pos(file_pair *pair, size_t rewind_size); +/// \brief Seek to the given absolute position in the source file +/// +/// This calls lseek() and also clears pair->src_eof. +/// +/// \param pair Seekable source file +/// \param pos Offset relative to the beginning of the file, +/// from which the data should be read. +/// +/// \return On success, false is returned. On error, error message +/// is printed and true is returned. +extern bool io_seek_src(file_pair *pair, uint64_t pos); + + /// \brief Read from source file from given offset to a buffer /// /// This is remotely similar to standard pread(). This uses lseek() though, @@ -152,7 +165,7 @@ extern void io_fix_src_pos(file_pair *pair, size_t rewind_size); /// /// \return On success, false is returned. On error, error message /// is printed and true is returned. -extern bool io_pread(file_pair *pair, io_buf *buf, size_t size, off_t pos); +extern bool io_pread(file_pair *pair, io_buf *buf, size_t size, uint64_t pos); /// \brief Writes a buffer to the destination file diff --git a/src/xz/hardware.c b/src/xz/hardware.c index da3ed3f1cd11..ccdc3b9eb5c9 100644 --- a/src/xz/hardware.c +++ b/src/xz/hardware.c @@ -17,11 +17,42 @@ /// the --threads=NUM command line option. static uint32_t threads_max = 1; +/// True when the number of threads is automatically determined based +/// on the available hardware threads. +static bool threads_are_automatic = false; + +/// If true, then try to use multi-threaded mode (if memlimit allows) +/// even if only one thread was requested explicitly (-T+1). +static bool use_mt_mode_with_one_thread = false; + /// Memory usage limit for compression -static uint64_t memlimit_compress; +static uint64_t memlimit_compress = 0; /// Memory usage limit for decompression -static uint64_t memlimit_decompress; +static uint64_t memlimit_decompress = 0; + +/// Default memory usage for multithreaded modes: +/// +/// - Default value for --memlimit-compress when automatic number of threads +/// is used. However, if the limit wouldn't allow even one thread then +/// the limit is ignored in coder.c and one thread will be used anyway. +/// This mess is a compromise: we wish to prevent -T0 from using too +/// many threads but we also don't want xz to give an error due to +/// a memlimit that the user didn't explicitly set. +/// +/// - Default value for --memlimit-mt-decompress +/// +/// This value is caluclated in hardware_init() and cannot be changed later. +static uint64_t memlimit_mt_default; + +/// Memory usage limit for multithreaded decompression. This is a soft limit: +/// if reducing the number of threads to one isn't enough to keep memory +/// usage below this limit, then one thread is used and this limit is ignored. +/// memlimit_decompress is still obeyed. +/// +/// This can be set with --memlimit-mt-decompress. The default value for +/// this is memlimit_mt_default. +static uint64_t memlimit_mtdec; /// Total amount of physical RAM static uint64_t total_ram; @@ -30,8 +61,17 @@ static uint64_t total_ram; extern void hardware_threads_set(uint32_t n) { + // Reset these to false first and set them to true when appropriate. + threads_are_automatic = false; + use_mt_mode_with_one_thread = false; + if (n == 0) { // Automatic number of threads was requested. + // If there is only one hardware thread, multi-threaded + // mode will still be used if memory limit allows. + threads_are_automatic = true; + use_mt_mode_with_one_thread = true; + // If threading support was enabled at build time, // use the number of available CPU cores. Otherwise // use one thread since disabling threading support @@ -43,6 +83,9 @@ hardware_threads_set(uint32_t n) #else threads_max = 1; #endif + } else if (n == UINT32_MAX) { + use_mt_mode_with_one_thread = true; + threads_max = 1; } else { threads_max = n; } @@ -58,9 +101,21 @@ hardware_threads_get(void) } +extern bool +hardware_threads_is_mt(void) +{ +#ifdef MYTHREAD_ENABLED + return threads_max > 1 || use_mt_mode_with_one_thread; +#else + return false; +#endif +} + + extern void hardware_memlimit_set(uint64_t new_memlimit, - bool set_compress, bool set_decompress, bool is_percentage) + bool set_compress, bool set_decompress, bool set_mtdec, + bool is_percentage) { if (is_percentage) { assert(new_memlimit > 0); @@ -110,6 +165,9 @@ hardware_memlimit_set(uint64_t new_memlimit, if (set_decompress) memlimit_decompress = new_memlimit; + if (set_mtdec) + memlimit_mtdec = new_memlimit; + return; } @@ -117,32 +175,69 @@ hardware_memlimit_set(uint64_t new_memlimit, extern uint64_t hardware_memlimit_get(enum operation_mode mode) { - // Zero is a special value that indicates the default. Currently - // the default simply disables the limit. Once there is threading - // support, this might be a little more complex, because there will - // probably be a special case where a user asks for "optimal" number - // of threads instead of a specific number (this might even become - // the default mode). Each thread may use a significant amount of - // memory. When there are no memory usage limits set, we need some - // default soft limit for calculating the "optimal" number of - // threads. + // 0 is a special value that indicates the default. + // It disables the limit in single-threaded mode. + // + // NOTE: For multithreaded decompression, this is the hard limit + // (memlimit_stop). hardware_memlimit_mtdec_get() gives the + // soft limit (memlimit_threaded). const uint64_t memlimit = mode == MODE_COMPRESS ? memlimit_compress : memlimit_decompress; return memlimit != 0 ? memlimit : UINT64_MAX; } +extern uint64_t +hardware_memlimit_mtenc_get(void) +{ + return hardware_memlimit_mtenc_is_default() + ? memlimit_mt_default + : hardware_memlimit_get(MODE_COMPRESS); +} + + +extern bool +hardware_memlimit_mtenc_is_default(void) +{ + return memlimit_compress == 0 && threads_are_automatic; +} + + +extern uint64_t +hardware_memlimit_mtdec_get(void) +{ + uint64_t m = memlimit_mtdec != 0 + ? memlimit_mtdec + : memlimit_mt_default; + + // Cap the value to memlimit_decompress if it has been specified. + // This is nice for --info-memory. It wouldn't be needed for liblzma + // since it does this anyway. + if (memlimit_decompress != 0 && m > memlimit_decompress) + m = memlimit_decompress; + + return m; +} + + /// Helper for hardware_memlimit_show() to print one human-readable info line. static void -memlimit_show(const char *str, uint64_t value) +memlimit_show(const char *str, size_t str_columns, uint64_t value) { + // Calculate the field width so that str will be padded to take + // str_columns on the terminal. + // + // NOTE: If the string is invalid, this will be -1. Using -1 as + // the field width is fine here so it's not handled specially. + const int fw = tuklib_mbstr_fw(str, (int)(str_columns)); + // The memory usage limit is considered to be disabled if value // is 0 or UINT64_MAX. This might get a bit more complex once there // is threading support. See the comment in hardware_memlimit_get(). if (value == 0 || value == UINT64_MAX) - printf("%s %s\n", str, _("Disabled")); + printf(" %-*s %s\n", fw, str, _("Disabled")); else - printf("%s %s MiB (%s B)\n", str, + printf(" %-*s %s MiB (%s B)\n", fw, str, uint64_to_str(round_up_to_mib(value), 0), uint64_to_str(value, 1)); @@ -153,18 +248,60 @@ memlimit_show(const char *str, uint64_t value) extern void hardware_memlimit_show(void) { + uint32_t cputhreads = 1; +#ifdef MYTHREAD_ENABLED + cputhreads = lzma_cputhreads(); + if (cputhreads == 0) + cputhreads = 1; +#endif + if (opt_robot) { - printf("%" PRIu64 "\t%" PRIu64 "\t%" PRIu64 "\n", total_ram, - memlimit_compress, memlimit_decompress); + printf("%" PRIu64 "\t%" PRIu64 "\t%" PRIu64 "\t%" PRIu64 + "\t%" PRIu64 "\t%" PRIu32 "\n", + total_ram, + memlimit_compress, + memlimit_decompress, + hardware_memlimit_mtdec_get(), + memlimit_mt_default, + cputhreads); } else { - // TRANSLATORS: Test with "xz --info-memory" to see if - // the alignment looks nice. - memlimit_show(_("Total amount of physical memory (RAM): "), - total_ram); - memlimit_show(_("Memory usage limit for compression: "), - memlimit_compress); - memlimit_show(_("Memory usage limit for decompression: "), - memlimit_decompress); + const char *msgs[] = { + _("Amount of physical memory (RAM):"), + _("Number of processor threads:"), + _("Compression:"), + _("Decompression:"), + _("Multi-threaded decompression:"), + _("Default for -T0:"), + }; + + size_t width_max = 1; + for (unsigned i = 0; i < ARRAY_SIZE(msgs); ++i) { + size_t w = tuklib_mbstr_width(msgs[i], NULL); + + // When debugging, catch invalid strings with + // an assertion. Otherwise fallback to 1 so + // that the columns just won't be aligned. + assert(w != (size_t)-1); + if (w == (size_t)-1) + w = 1; + + if (width_max < w) + width_max = w; + } + + puts(_("Hardware information:")); + memlimit_show(msgs[0], width_max, total_ram); + printf(" %-*s %" PRIu32 "\n", + tuklib_mbstr_fw(msgs[1], (int)(width_max)), + msgs[1], cputhreads); + + putchar('\n'); + puts(_("Memory usage limits:")); + memlimit_show(msgs[2], width_max, memlimit_compress); + memlimit_show(msgs[3], width_max, memlimit_decompress); + memlimit_show(msgs[4], width_max, + hardware_memlimit_mtdec_get()); + memlimit_show(msgs[5], width_max, memlimit_mt_default); } tuklib_exit(E_SUCCESS, E_ERROR, message_verbosity_get() != V_SILENT); @@ -180,7 +317,22 @@ hardware_init(void) if (total_ram == 0) total_ram = (uint64_t)(ASSUME_RAM) * 1024 * 1024; - // Set the defaults. - hardware_memlimit_set(0, true, true, false); + // FIXME? There may be better methods to determine the default value. + // One Linux-specific suggestion is to use MemAvailable from + // /proc/meminfo as the starting point. + memlimit_mt_default = total_ram / 4; + +#if SIZE_MAX == UINT32_MAX + // A too high value may cause 32-bit xz to run out of address space. + // Use a conservative maximum value here. A few typical address space + // sizes with Linux: + // - x86-64 with 32-bit xz: 4 GiB + // - x86: 3 GiB + // - MIPS32: 2 GiB + const size_t mem_ceiling = 1400U << 20; + if (memlimit_mt_default > mem_ceiling) + memlimit_mt_default = mem_ceiling; +#endif + return; } diff --git a/src/xz/hardware.h b/src/xz/hardware.h index 4fae61815656..2bb3d7ba9493 100644 --- a/src/xz/hardware.h +++ b/src/xz/hardware.h @@ -16,22 +16,59 @@ extern void hardware_init(void); /// Set the maximum number of worker threads. +/// A special value of UINT32_MAX sets one thread in multi-threaded mode. extern void hardware_threads_set(uint32_t threadlimit); /// Get the maximum number of worker threads. extern uint32_t hardware_threads_get(void); +/// Returns true if multithreaded mode should be used for .xz compression. +/// This can be true even if the number of threads is one. +extern bool hardware_threads_is_mt(void); -/// Set the memory usage limit. There are separate limits for compression -/// and decompression (the latter includes also --list), one or both can -/// be set with a single call to this function. Zero indicates resetting -/// the limit back to the defaults. The limit can also be set as a percentage -/// of installed RAM; the percentage must be in the range [1, 100]. + +/// Set the memory usage limit. There are separate limits for compression, +/// decompression (also includes --list), and multithreaded decompression. +/// Any combination of these can be set with a single call to this function. +/// Zero indicates resetting the limit back to the defaults. +/// The limit can also be set as a percentage of installed RAM; the +/// percentage must be in the range [1, 100]. extern void hardware_memlimit_set(uint64_t new_memlimit, - bool set_compress, bool set_decompress, bool is_percentage); + bool set_compress, bool set_decompress, bool set_mtdec, + bool is_percentage); /// Get the current memory usage limit for compression or decompression. +/// This is a hard limit that will not be exceeded. This is obeyed in +/// both single-threaded and multithreaded modes. extern uint64_t hardware_memlimit_get(enum operation_mode mode); +/// This returns a system-specific default value if all of the following +/// conditions are true: +/// +/// - An automatic number of threads was requested (--threads=0). +/// +/// - --memlimit-compress wasn't used or it was reset to the default +/// value by setting it to 0. +/// +/// Otherwise this is identical to hardware_memlimit_get(MODE_COMPRESS). +/// +/// The idea is to keep automatic thread count reasonable so that too +/// high memory usage is avoided and, with 32-bit xz, running out of +/// address space is avoided. +extern uint64_t hardware_memlimit_mtenc_get(void); + +/// Returns true if the value returned by hardware_memlimit_mtenc_get() is +/// a system-specific default value. coder.c uses this to ignore the default +/// memlimit in case it's too small even for a single thread in multithreaded +/// mode. This way the default limit will never make xz fail or affect the +/// compressed output; it will only make xz reduce the number of threads. +extern bool hardware_memlimit_mtenc_is_default(void); + +/// Get the current memory usage limit for multithreaded decompression. +/// This is only used to reduce the number of threads. This limit can be +/// exceeded if the number of threads are reduce to one. Then the value +/// from hardware_memlimit_get() will be honored like in single-threaded mode. +extern uint64_t hardware_memlimit_mtdec_get(void); + /// Display the amount of RAM and memory usage limits and exit. extern void hardware_memlimit_show(void) lzma_attribute((__noreturn__)); diff --git a/src/xz/list.c b/src/xz/list.c index 18fdca21763c..ee61aeb93bf4 100644 --- a/src/xz/list.c +++ b/src/xz/list.c @@ -52,23 +52,126 @@ typedef struct { uint64_t memusage; /// The filter chain of this Block in human-readable form - char filter_chain[FILTERS_STR_SIZE]; + char *filter_chain; } block_header_info; +#define BLOCK_HEADER_INFO_INIT { .filter_chain = NULL } +#define block_header_info_end(bhi) free((bhi)->filter_chain) + + +/// Strings ending in a colon. These are used for lines like +/// " Foo: 123 MiB". These are grouped because translated strings +/// may have different maximum string length, and we want to pad all +/// strings so that the values are aligned nicely. +static const char *colon_strs[] = { + N_("Streams:"), + N_("Blocks:"), + N_("Compressed size:"), + N_("Uncompressed size:"), + N_("Ratio:"), + N_("Check:"), + N_("Stream Padding:"), + N_("Memory needed:"), + N_("Sizes in headers:"), + // This won't be aligned because it's so long: + //N_("Minimum XZ Utils version:"), + N_("Number of files:"), +}; + +/// Enum matching the above strings. +enum { + COLON_STR_STREAMS, + COLON_STR_BLOCKS, + COLON_STR_COMPRESSED_SIZE, + COLON_STR_UNCOMPRESSED_SIZE, + COLON_STR_RATIO, + COLON_STR_CHECK, + COLON_STR_STREAM_PADDING, + COLON_STR_MEMORY_NEEDED, + COLON_STR_SIZES_IN_HEADERS, + //COLON_STR_MINIMUM_XZ_VERSION, + COLON_STR_NUMBER_OF_FILES, +}; + +/// Field widths to use with printf to pad the strings to use the same number +/// of columns on a terminal. +static int colon_strs_fw[ARRAY_SIZE(colon_strs)]; + +/// Convenience macro to get the translated string and its field width +/// using a COLON_STR_foo enum. +#define COLON_STR(num) colon_strs_fw[num], _(colon_strs[num]) + + +/// Column headings +static struct { + /// Table column heading string + const char *str; + + /// Number of terminal-columns to use for this table-column. + /// If a translated string is longer than the initial value, + /// this value will be increased in init_headings(). + int columns; + + /// Field width to use for printf() to pad "str" to use "columns" + /// number of columns on a terminal. This is calculated in + /// init_headings(). + int fw; + +} headings[] = { + { N_("Stream"), 6, 0 }, + { N_("Block"), 9, 0 }, + { N_("Blocks"), 9, 0 }, + { N_("CompOffset"), 15, 0 }, + { N_("UncompOffset"), 15, 0 }, + { N_("CompSize"), 15, 0 }, + { N_("UncompSize"), 15, 0 }, + { N_("TotalSize"), 15, 0 }, + { N_("Ratio"), 5, 0 }, + { N_("Check"), 10, 0 }, + { N_("CheckVal"), 1, 0 }, + { N_("Padding"), 7, 0 }, + { N_("Header"), 5, 0 }, + { N_("Flags"), 2, 0 }, + { N_("MemUsage"), 7 + 4, 0 }, // +4 is for " MiB" + { N_("Filters"), 1, 0 }, +}; + +/// Enum matching the above strings. +enum { + HEADING_STREAM, + HEADING_BLOCK, + HEADING_BLOCKS, + HEADING_COMPOFFSET, + HEADING_UNCOMPOFFSET, + HEADING_COMPSIZE, + HEADING_UNCOMPSIZE, + HEADING_TOTALSIZE, + HEADING_RATIO, + HEADING_CHECK, + HEADING_CHECKVAL, + HEADING_PADDING, + HEADING_HEADERSIZE, + HEADING_HEADERFLAGS, + HEADING_MEMUSAGE, + HEADING_FILTERS, +}; + +#define HEADING_STR(num) headings[num].fw, _(headings[num].str) + /// Check ID to string mapping static const char check_names[LZMA_CHECK_ID_MAX + 1][12] = { // TRANSLATORS: Indicates that there is no integrity check. - // This string is used in tables, so the width must not - // exceed ten columns with a fixed-width font. + // This string is used in tables. In older xz version this + // string was limited to ten columns in a fixed-width font, but + // nowadays there is no strict length restriction anymore. N_("None"), "CRC32", // TRANSLATORS: Indicates that integrity check name is not known, - // but the Check ID is known (here 2). This and other "Unknown-N" - // strings are used in tables, so the width must not exceed ten - // columns with a fixed-width font. It's OK to omit the dash if - // you need space for one extra letter, but don't use spaces. + // but the Check ID is known (here 2). In older xz version these + // strings were limited to ten columns in a fixed-width font, but + // nowadays there is no strict length restriction anymore. N_("Unknown-2"), N_("Unknown-3"), "CRC64", @@ -112,6 +215,104 @@ static struct { } totals = { 0, 0, 0, 0, 0, 0, 0, 0, 50000002, true }; +/// Initialize colon_strs_fw[]. +static void +init_colon_strs(void) +{ + // Lengths of translated strings as bytes. + size_t lens[ARRAY_SIZE(colon_strs)]; + + // Lengths of translated strings as columns. + size_t widths[ARRAY_SIZE(colon_strs)]; + + // Maximum number of columns needed by a translated string. + size_t width_max = 0; + + for (unsigned i = 0; i < ARRAY_SIZE(colon_strs); ++i) { + widths[i] = tuklib_mbstr_width(_(colon_strs[i]), &lens[i]); + + // If debugging is enabled, catch invalid strings with + // an assertion. However, when not debugging, use the + // byte count as the fallback width. This shouldn't + // ever happen unless there is a bad string in the + // translations, but in such case I guess it's better + // to try to print something useful instead of failing + // completely. + assert(widths[i] != (size_t)-1); + if (widths[i] == (size_t)-1) + widths[i] = lens[i]; + + if (widths[i] > width_max) + width_max = widths[i]; + } + + // Calculate the field width for printf("%*s") so that the strings + // will use width_max columns on a terminal. + for (unsigned i = 0; i < ARRAY_SIZE(colon_strs); ++i) + colon_strs_fw[i] = (int)(lens[i] + width_max - widths[i]); + + return; +} + + +/// Initialize headings[]. +static void +init_headings(void) +{ + // Before going through the heading strings themselves, treat + // the Check heading specially: Look at the widths of the various + // check names and increase the width of the Check column if needed. + // The width of the heading name "Check" will then be handled normally + // with other heading names in the second loop in this function. + for (unsigned i = 0; i < ARRAY_SIZE(check_names); ++i) { + size_t len; + size_t w = tuklib_mbstr_width(_(check_names[i]), &len); + + // Error handling like in init_colon_strs(). + assert(w != (size_t)-1); + if (w == (size_t)-1) + w = len; + + // If the translated string is wider than the minimum width + // set at compile time, increase the width. + if ((size_t)(headings[HEADING_CHECK].columns) < w) + headings[HEADING_CHECK].columns = w; + } + + for (unsigned i = 0; i < ARRAY_SIZE(headings); ++i) { + size_t len; + size_t w = tuklib_mbstr_width(_(headings[i].str), &len); + + // Error handling like in init_colon_strs(). + assert(w != (size_t)-1); + if (w == (size_t)-1) + w = len; + + // If the translated string is wider than the minimum width + // set at compile time, increase the width. + if ((size_t)(headings[i].columns) < w) + headings[i].columns = w; + + // Calculate the field width for printf("%*s") so that + // the string uses .columns number of columns on a terminal. + headings[i].fw = (int)(len + (size_t)headings[i].columns - w); + } + + return; +} + + +/// Initialize the printf field widths that are needed to get nicely aligned +/// output with translated strings. +static void +init_field_widths(void) +{ + init_colon_strs(); + init_headings(); + return; +} + + /// Convert XZ Utils version number to a string. static const char * xz_ver_to_str(uint32_t ver) @@ -143,9 +344,6 @@ xz_ver_to_str(uint32_t ver) /// /// \return On success, false is returned. On error, true is returned. /// -// TODO: This function is pretty big. liblzma should have a function that -// takes a callback function to parse the Index(es) from a .xz file to make -// it easy for applications. static bool parse_indexes(xz_file_info *xfi, file_pair *pair) { @@ -161,238 +359,74 @@ parse_indexes(xz_file_info *xfi, file_pair *pair) } io_buf buf; - lzma_stream_flags header_flags; - lzma_stream_flags footer_flags; - lzma_ret ret; - - // lzma_stream for the Index decoder lzma_stream strm = LZMA_STREAM_INIT; + lzma_index *idx = NULL; - // All Indexes decoded so far - lzma_index *combined_index = NULL; - - // The Index currently being decoded - lzma_index *this_index = NULL; - - // Current position in the file. We parse the file backwards so - // initialize it to point to the end of the file. - off_t pos = pair->src_st.st_size; - - // Each loop iteration decodes one Index. - do { - // Check that there is enough data left to contain at least - // the Stream Header and Stream Footer. This check cannot - // fail in the first pass of this loop. - if (pos < 2 * LZMA_STREAM_HEADER_SIZE) { - message_error("%s: %s", pair->src_name, - message_strm(LZMA_DATA_ERROR)); - goto error; - } - - pos -= LZMA_STREAM_HEADER_SIZE; - lzma_vli stream_padding = 0; - - // Locate the Stream Footer. There may be Stream Padding which - // we must skip when reading backwards. - while (true) { - if (pos < LZMA_STREAM_HEADER_SIZE) { - message_error("%s: %s", pair->src_name, - message_strm( - LZMA_DATA_ERROR)); - goto error; - } + lzma_ret ret = lzma_file_info_decoder(&strm, &idx, + hardware_memlimit_get(MODE_LIST), + (uint64_t)(pair->src_st.st_size)); + if (ret != LZMA_OK) { + message_error("%s: %s", pair->src_name, message_strm(ret)); + return true; + } - if (io_pread(pair, &buf, - LZMA_STREAM_HEADER_SIZE, pos)) + while (true) { + if (strm.avail_in == 0) { + strm.next_in = buf.u8; + strm.avail_in = io_read(pair, &buf, IO_BUFFER_SIZE); + if (strm.avail_in == SIZE_MAX) goto error; - - // Stream Padding is always a multiple of four bytes. - int i = 2; - if (buf.u32[i] != 0) - break; - - // To avoid calling io_pread() for every four bytes - // of Stream Padding, take advantage that we read - // 12 bytes (LZMA_STREAM_HEADER_SIZE) already and - // check them too before calling io_pread() again. - do { - stream_padding += 4; - pos -= 4; - --i; - } while (i >= 0 && buf.u32[i] == 0); } - // Decode the Stream Footer. - ret = lzma_stream_footer_decode(&footer_flags, buf.u8); - if (ret != LZMA_OK) { - message_error("%s: %s", pair->src_name, - message_strm(ret)); - goto error; - } + ret = lzma_code(&strm, LZMA_RUN); - // Check that the Stream Footer doesn't specify something - // that we don't support. This can only happen if the xz - // version is older than liblzma and liblzma supports - // something new. - // - // It is enough to check Stream Footer. Stream Header must - // match when it is compared against Stream Footer with - // lzma_stream_flags_compare(). - if (footer_flags.version != 0) { - message_error("%s: %s", pair->src_name, - message_strm(LZMA_OPTIONS_ERROR)); - goto error; - } + switch (ret) { + case LZMA_OK: + break; - // Check that the size of the Index field looks sane. - lzma_vli index_size = footer_flags.backward_size; - if ((lzma_vli)(pos) < index_size + LZMA_STREAM_HEADER_SIZE) { - message_error("%s: %s", pair->src_name, - message_strm(LZMA_DATA_ERROR)); - goto error; - } + case LZMA_SEEK_NEEDED: + // liblzma won't ask us to seek past the known size + // of the input file. + assert(strm.seek_pos + <= (uint64_t)(pair->src_st.st_size)); + if (io_seek_src(pair, strm.seek_pos)) + goto error; - // Set pos to the beginning of the Index. - pos -= index_size; + // avail_in must be zero so that we will read new + // input. + strm.avail_in = 0; + break; - // See how much memory we can use for decoding this Index. - uint64_t memlimit = hardware_memlimit_get(MODE_LIST); - uint64_t memused = 0; - if (combined_index != NULL) { - memused = lzma_index_memused(combined_index); - if (memused > memlimit) - message_bug(); + case LZMA_STREAM_END: { + lzma_end(&strm); + xfi->idx = idx; - memlimit -= memused; - } + // Calculate xfi->stream_padding. + lzma_index_iter iter; + lzma_index_iter_init(&iter, xfi->idx); + while (!lzma_index_iter_next(&iter, + LZMA_INDEX_ITER_STREAM)) + xfi->stream_padding += iter.stream.padding; - // Decode the Index. - ret = lzma_index_decoder(&strm, &this_index, memlimit); - if (ret != LZMA_OK) { - message_error("%s: %s", pair->src_name, - message_strm(ret)); - goto error; + return false; } - do { - // Don't give the decoder more input than the - // Index size. - strm.avail_in = my_min(IO_BUFFER_SIZE, index_size); - if (io_pread(pair, &buf, strm.avail_in, pos)) - goto error; - - pos += strm.avail_in; - index_size -= strm.avail_in; - - strm.next_in = buf.u8; - ret = lzma_code(&strm, LZMA_RUN); - - } while (ret == LZMA_OK); - - // If the decoding seems to be successful, check also that - // the Index decoder consumed as much input as indicated - // by the Backward Size field. - if (ret == LZMA_STREAM_END) - if (index_size != 0 || strm.avail_in != 0) - ret = LZMA_DATA_ERROR; - - if (ret != LZMA_STREAM_END) { - // LZMA_BUFFER_ERROR means that the Index decoder - // would have liked more input than what the Index - // size should be according to Stream Footer. - // The message for LZMA_DATA_ERROR makes more - // sense in that case. - if (ret == LZMA_BUF_ERROR) - ret = LZMA_DATA_ERROR; - + default: message_error("%s: %s", pair->src_name, message_strm(ret)); // If the error was too low memory usage limit, // show also how much memory would have been needed. - if (ret == LZMA_MEMLIMIT_ERROR) { - uint64_t needed = lzma_memusage(&strm); - if (UINT64_MAX - needed < memused) - needed = UINT64_MAX; - else - needed += memused; - - message_mem_needed(V_ERROR, needed); - } - - goto error; - } - - // Decode the Stream Header and check that its Stream Flags - // match the Stream Footer. - pos -= footer_flags.backward_size + LZMA_STREAM_HEADER_SIZE; - if ((lzma_vli)(pos) < lzma_index_total_size(this_index)) { - message_error("%s: %s", pair->src_name, - message_strm(LZMA_DATA_ERROR)); - goto error; - } - - pos -= lzma_index_total_size(this_index); - if (io_pread(pair, &buf, LZMA_STREAM_HEADER_SIZE, pos)) - goto error; + if (ret == LZMA_MEMLIMIT_ERROR) + message_mem_needed(V_ERROR, + lzma_memusage(&strm)); - ret = lzma_stream_header_decode(&header_flags, buf.u8); - if (ret != LZMA_OK) { - message_error("%s: %s", pair->src_name, - message_strm(ret)); goto error; } - - ret = lzma_stream_flags_compare(&header_flags, &footer_flags); - if (ret != LZMA_OK) { - message_error("%s: %s", pair->src_name, - message_strm(ret)); - goto error; - } - - // Store the decoded Stream Flags into this_index. This is - // needed so that we can print which Check is used in each - // Stream. - ret = lzma_index_stream_flags(this_index, &footer_flags); - if (ret != LZMA_OK) - message_bug(); - - // Store also the size of the Stream Padding field. It is - // needed to show the offsets of the Streams correctly. - ret = lzma_index_stream_padding(this_index, stream_padding); - if (ret != LZMA_OK) - message_bug(); - - if (combined_index != NULL) { - // Append the earlier decoded Indexes - // after this_index. - ret = lzma_index_cat( - this_index, combined_index, NULL); - if (ret != LZMA_OK) { - message_error("%s: %s", pair->src_name, - message_strm(ret)); - goto error; - } - } - - combined_index = this_index; - this_index = NULL; - - xfi->stream_padding += stream_padding; - - } while (pos > 0); - - lzma_end(&strm); - - // All OK. Make combined_index available to the caller. - xfi->idx = combined_index; - return false; + } error: - // Something went wrong, free the allocated memory. lzma_end(&strm); - lzma_index_end(combined_index, NULL); - lzma_index_end(this_index, NULL); return true; } @@ -454,6 +488,10 @@ parse_block_header(file_pair *pair, const lzma_index_iter *iter, // Check the Block Flags. These must be done before calling // lzma_block_compressed_size(), because it overwrites // block.compressed_size. + // + // NOTE: If you add new characters here, update the minimum number of + // columns in headings[HEADING_HEADERFLAGS] to match the number of + // characters used here. bhi->flags[0] = block.compressed_size != LZMA_VLI_UNKNOWN ? 'c' : '-'; bhi->flags[1] = block.uncompressed_size != LZMA_VLI_UNKNOWN @@ -488,9 +526,7 @@ parse_block_header(file_pair *pair, const lzma_index_iter *iter, case LZMA_DATA_ERROR: // Free the memory allocated by lzma_block_header_decode(). - for (size_t i = 0; filters[i].id != LZMA_VLI_UNKNOWN; ++i) - free(filters[i].options); - + lzma_filters_free(filters, NULL); goto data_error; default: @@ -509,25 +545,42 @@ parse_block_header(file_pair *pair, const lzma_index_iter *iter, // Determine the minimum XZ Utils version that supports this Block. // - // Currently the only thing that 5.0.0 doesn't support is empty - // LZMA2 Block. This decoder bug was fixed in 5.0.2. - { + // - ARM64 filter needs 5.4.0. + // + // - 5.0.0 doesn't support empty LZMA2 streams and thus empty + // Blocks that use LZMA2. This decoder bug was fixed in 5.0.2. + if (xfi->min_version < 50040002U) { + for (size_t i = 0; filters[i].id != LZMA_VLI_UNKNOWN; ++i) { + if (filters[i].id == LZMA_FILTER_ARM64) { + xfi->min_version = 50040002U; + break; + } + } + } + + if (xfi->min_version < 50000022U) { size_t i = 0; while (filters[i + 1].id != LZMA_VLI_UNKNOWN) ++i; if (filters[i].id == LZMA_FILTER_LZMA2 - && iter->block.uncompressed_size == 0 - && xfi->min_version < 50000022U) + && iter->block.uncompressed_size == 0) xfi->min_version = 50000022U; } // Convert the filter chain to human readable form. - message_filters_to_str(bhi->filter_chain, filters, false); + const lzma_ret str_ret = lzma_str_from_filters( + &bhi->filter_chain, filters, + LZMA_STR_DECODER | LZMA_STR_GETOPT_LONG, NULL); // Free the memory allocated by lzma_block_header_decode(). - for (size_t i = 0; filters[i].id != LZMA_VLI_UNKNOWN; ++i) - free(filters[i].options); + lzma_filters_free(filters, NULL); + + // Check if the stringification succeeded. + if (str_ret != LZMA_OK) { + message_error("%s: %s", pair->src_name, message_strm(str_ret)); + return true; + } return false; @@ -553,7 +606,7 @@ parse_check_value(file_pair *pair, const lzma_index_iter *iter) // Locate and read the Check field. const uint32_t size = lzma_check_size(iter->stream.flags->check); - const off_t offset = iter->block.compressed_file_offset + const uint64_t offset = iter->block.compressed_file_offset + iter->block.total_size - size; io_buf buf; if (io_pread(pair, &buf, size, offset)) @@ -714,20 +767,20 @@ print_adv_helper(uint64_t stream_count, uint64_t block_count, char checks_str[CHECKS_STR_SIZE]; get_check_names(checks_str, checks, true); - printf(_(" Streams: %s\n"), + printf(" %-*s %s\n", COLON_STR(COLON_STR_STREAMS), uint64_to_str(stream_count, 0)); - printf(_(" Blocks: %s\n"), + printf(" %-*s %s\n", COLON_STR(COLON_STR_BLOCKS), uint64_to_str(block_count, 0)); - printf(_(" Compressed size: %s\n"), + printf(" %-*s %s\n", COLON_STR(COLON_STR_COMPRESSED_SIZE), uint64_to_nicestr(compressed_size, NICESTR_B, NICESTR_TIB, true, 0)); - printf(_(" Uncompressed size: %s\n"), + printf(" %-*s %s\n", COLON_STR(COLON_STR_UNCOMPRESSED_SIZE), uint64_to_nicestr(uncompressed_size, NICESTR_B, NICESTR_TIB, true, 0)); - printf(_(" Ratio: %s\n"), + printf(" %-*s %s\n", COLON_STR(COLON_STR_RATIO), get_ratio(compressed_size, uncompressed_size)); - printf(_(" Check: %s\n"), checks_str); - printf(_(" Stream padding: %s\n"), + printf(" %-*s %s\n", COLON_STR(COLON_STR_CHECK), checks_str); + printf(" %-*s %s\n", COLON_STR(COLON_STR_STREAM_PADDING), uint64_to_nicestr(stream_padding, NICESTR_B, NICESTR_TIB, true, 0)); return; @@ -752,13 +805,19 @@ print_info_adv(xz_file_info *xfi, file_pair *pair) // Print information about the Streams. // - // TRANSLATORS: The second line is column headings. All except - // Check are right aligned; Check is left aligned. Test with - // "xz -lv foo.xz". - puts(_(" Streams:\n Stream Blocks" - " CompOffset UncompOffset" - " CompSize UncompSize Ratio" - " Check Padding")); + // All except Check are right aligned; Check is left aligned. + // Test with "xz -lv foo.xz". + printf(" %s\n %*s %*s %*s %*s %*s %*s %*s %-*s %*s\n", + _(colon_strs[COLON_STR_STREAMS]), + HEADING_STR(HEADING_STREAM), + HEADING_STR(HEADING_BLOCKS), + HEADING_STR(HEADING_COMPOFFSET), + HEADING_STR(HEADING_UNCOMPOFFSET), + HEADING_STR(HEADING_COMPSIZE), + HEADING_STR(HEADING_UNCOMPSIZE), + HEADING_STR(HEADING_RATIO), + HEADING_STR(HEADING_CHECK), + HEADING_STR(HEADING_PADDING)); lzma_index_iter iter; lzma_index_iter_init(&iter, xfi->idx); @@ -771,10 +830,18 @@ print_info_adv(xz_file_info *xfi, file_pair *pair) uint64_to_str(iter.stream.uncompressed_offset, 3), }; printf(" %*s %*s %*s %*s ", - tuklib_mbstr_fw(cols1[0], 6), cols1[0], - tuklib_mbstr_fw(cols1[1], 9), cols1[1], - tuklib_mbstr_fw(cols1[2], 15), cols1[2], - tuklib_mbstr_fw(cols1[3], 15), cols1[3]); + tuklib_mbstr_fw(cols1[0], + headings[HEADING_STREAM].columns), + cols1[0], + tuklib_mbstr_fw(cols1[1], + headings[HEADING_BLOCKS].columns), + cols1[1], + tuklib_mbstr_fw(cols1[2], + headings[HEADING_COMPOFFSET].columns), + cols1[2], + tuklib_mbstr_fw(cols1[3], + headings[HEADING_UNCOMPOFFSET].columns), + cols1[3]); const char *cols2[5] = { uint64_to_str(iter.stream.compressed_size, 0), @@ -785,11 +852,21 @@ print_info_adv(xz_file_info *xfi, file_pair *pair) uint64_to_str(iter.stream.padding, 2), }; printf("%*s %*s %*s %-*s %*s\n", - tuklib_mbstr_fw(cols2[0], 15), cols2[0], - tuklib_mbstr_fw(cols2[1], 15), cols2[1], - tuklib_mbstr_fw(cols2[2], 5), cols2[2], - tuklib_mbstr_fw(cols2[3], 10), cols2[3], - tuklib_mbstr_fw(cols2[4], 7), cols2[4]); + tuklib_mbstr_fw(cols2[0], + headings[HEADING_COMPSIZE].columns), + cols2[0], + tuklib_mbstr_fw(cols2[1], + headings[HEADING_UNCOMPSIZE].columns), + cols2[1], + tuklib_mbstr_fw(cols2[2], + headings[HEADING_RATIO].columns), + cols2[2], + tuklib_mbstr_fw(cols2[3], + headings[HEADING_CHECK].columns), + cols2[3], + tuklib_mbstr_fw(cols2[4], + headings[HEADING_PADDING].columns), + cols2[4]); // Update the maximum Check size. if (lzma_check_size(iter.stream.flags->check) > check_max) @@ -799,32 +876,47 @@ print_info_adv(xz_file_info *xfi, file_pair *pair) // Cache the verbosity level to a local variable. const bool detailed = message_verbosity_get() >= V_DEBUG; - // Information collected from Block Headers - block_header_info bhi; - // Print information about the Blocks but only if there is // at least one Block. if (lzma_index_block_count(xfi->idx) > 0) { - // Calculate the width of the CheckVal field. - const int checkval_width = my_max(8, 2 * check_max); - - // TRANSLATORS: The second line is column headings. All - // except Check are right aligned; Check is left aligned. - printf(_(" Blocks:\n Stream Block" - " CompOffset UncompOffset" - " TotalSize UncompSize Ratio Check")); + // Calculate the width of the CheckVal column. This can be + // used as is as the field width for printf() when printing + // the actual check value as it is hexadecimal. However, to + // print the column heading, further calculation is needed + // to handle a translated string (it's done a few lines later). + assert(check_max <= LZMA_CHECK_SIZE_MAX); + const int checkval_width = my_max( + headings[HEADING_CHECKVAL].columns, + (int)(2 * check_max)); + + // All except Check are right aligned; Check is left aligned. + printf(" %s\n %*s %*s %*s %*s %*s %*s %*s %-*s", + _(colon_strs[COLON_STR_BLOCKS]), + HEADING_STR(HEADING_STREAM), + HEADING_STR(HEADING_BLOCK), + HEADING_STR(HEADING_COMPOFFSET), + HEADING_STR(HEADING_UNCOMPOFFSET), + HEADING_STR(HEADING_TOTALSIZE), + HEADING_STR(HEADING_UNCOMPSIZE), + HEADING_STR(HEADING_RATIO), + detailed ? headings[HEADING_CHECK].fw : 1, + _(headings[HEADING_CHECK].str)); if (detailed) { - // TRANSLATORS: These are additional column headings - // for the most verbose listing mode. CheckVal - // (Check value), Flags, and Filters are left aligned. - // Header (Block Header Size), CompSize, and MemUsage - // are right aligned. %*s is replaced with 0-120 - // spaces to make the CheckVal column wide enough. - // Test with "xz -lvv foo.xz". - printf(_(" CheckVal %*s Header Flags " - "CompSize MemUsage Filters"), - checkval_width - 8, ""); + // CheckVal (Check value), Flags, and Filters are + // left aligned. Block Header Size, CompSize, and + // MemUsage are right aligned. Test with + // "xz -lvv foo.xz". + printf(" %-*s %*s %-*s %*s %*s %s", + headings[HEADING_CHECKVAL].fw + + checkval_width + - headings[HEADING_CHECKVAL].columns, + _(headings[HEADING_CHECKVAL].str), + HEADING_STR(HEADING_HEADERSIZE), + HEADING_STR(HEADING_HEADERFLAGS), + HEADING_STR(HEADING_COMPSIZE), + HEADING_STR(HEADING_MEMUSAGE), + _(headings[HEADING_FILTERS].str)); } putchar('\n'); @@ -833,8 +925,11 @@ print_info_adv(xz_file_info *xfi, file_pair *pair) // Iterate over the Blocks. while (!lzma_index_iter_next(&iter, LZMA_INDEX_ITER_BLOCK)) { + // If in detailed mode, collect the information from + // Block Header before starting to print the next line. + block_header_info bhi = BLOCK_HEADER_INFO_INIT; if (detailed && parse_details(pair, &iter, &bhi, xfi)) - return true; + return true; const char *cols1[4] = { uint64_to_str(iter.stream.number, 0), @@ -846,10 +941,18 @@ print_info_adv(xz_file_info *xfi, file_pair *pair) iter.block.uncompressed_file_offset, 3) }; printf(" %*s %*s %*s %*s ", - tuklib_mbstr_fw(cols1[0], 6), cols1[0], - tuklib_mbstr_fw(cols1[1], 9), cols1[1], - tuklib_mbstr_fw(cols1[2], 15), cols1[2], - tuklib_mbstr_fw(cols1[3], 15), cols1[3]); + tuklib_mbstr_fw(cols1[0], + headings[HEADING_STREAM].columns), + cols1[0], + tuklib_mbstr_fw(cols1[1], + headings[HEADING_BLOCK].columns), + cols1[1], + tuklib_mbstr_fw(cols1[2], + headings[HEADING_COMPOFFSET].columns), + cols1[2], + tuklib_mbstr_fw(cols1[3], headings[ + HEADING_UNCOMPOFFSET].columns), + cols1[3]); const char *cols2[4] = { uint64_to_str(iter.block.total_size, 0), @@ -860,11 +963,18 @@ print_info_adv(xz_file_info *xfi, file_pair *pair) _(check_names[iter.stream.flags->check]) }; printf("%*s %*s %*s %-*s", - tuklib_mbstr_fw(cols2[0], 15), cols2[0], - tuklib_mbstr_fw(cols2[1], 15), cols2[1], - tuklib_mbstr_fw(cols2[2], 5), cols2[2], - tuklib_mbstr_fw(cols2[3], detailed ? 11 : 1), - cols2[3]); + tuklib_mbstr_fw(cols2[0], + headings[HEADING_TOTALSIZE].columns), + cols2[0], + tuklib_mbstr_fw(cols2[1], + headings[HEADING_UNCOMPSIZE].columns), + cols2[1], + tuklib_mbstr_fw(cols2[2], + headings[HEADING_RATIO].columns), + cols2[2], + tuklib_mbstr_fw(cols2[3], detailed + ? headings[HEADING_CHECK].columns : 1), + cols2[3]); if (detailed) { const lzma_vli compressed_size @@ -885,25 +995,35 @@ print_info_adv(xz_file_info *xfi, file_pair *pair) }; // Show MiB for memory usage, because it // is the only size which is not in bytes. - printf("%-*s %*s %-5s %*s %*s MiB %s", + printf(" %-*s %*s %-*s %*s %*s MiB %s", checkval_width, cols3[0], - tuklib_mbstr_fw(cols3[1], 6), cols3[1], + tuklib_mbstr_fw(cols3[1], headings[ + HEADING_HEADERSIZE].columns), + cols3[1], + tuklib_mbstr_fw(cols3[2], headings[ + HEADING_HEADERFLAGS].columns), cols3[2], - tuklib_mbstr_fw(cols3[3], 15), - cols3[3], - tuklib_mbstr_fw(cols3[4], 7), cols3[4], + tuklib_mbstr_fw(cols3[3], headings[ + HEADING_COMPSIZE].columns), + cols3[3], + tuklib_mbstr_fw(cols3[4], headings[ + HEADING_MEMUSAGE].columns - 4), + cols3[4], cols3[5]); } putchar('\n'); + block_header_info_end(&bhi); } } if (detailed) { - printf(_(" Memory needed: %s MiB\n"), uint64_to_str( + printf(" %-*s %s MiB\n", COLON_STR(COLON_STR_MEMORY_NEEDED), + uint64_to_str( round_up_to_mib(xfi->memusage_max), 0)); - printf(_(" Sizes in headers: %s\n"), + printf(" %-*s %s\n", COLON_STR(COLON_STR_SIZES_IN_HEADERS), xfi->all_have_sizes ? _("Yes") : _("No")); + //printf(" %-*s %s\n", COLON_STR(COLON_STR_MINIMUM_XZ_VERSION), printf(_(" Minimum XZ Utils version: %s\n"), xz_ver_to_str(xfi->min_version)); } @@ -951,9 +1071,9 @@ print_info_robot(xz_file_info *xfi, file_pair *pair) iter.stream.padding); lzma_index_iter_rewind(&iter); - block_header_info bhi; while (!lzma_index_iter_next(&iter, LZMA_INDEX_ITER_BLOCK)) { + block_header_info bhi = BLOCK_HEADER_INFO_INIT; if (message_verbosity_get() >= V_DEBUG && parse_details( pair, &iter, &bhi, xfi)) @@ -984,6 +1104,7 @@ print_info_robot(xz_file_info *xfi, file_pair *pair) bhi.filter_chain); putchar('\n'); + block_header_info_end(&bhi); } } @@ -1068,17 +1189,19 @@ print_totals_adv(void) { putchar('\n'); puts(_("Totals:")); - printf(_(" Number of files: %s\n"), + printf(" %-*s %s\n", COLON_STR(COLON_STR_NUMBER_OF_FILES), uint64_to_str(totals.files, 0)); print_adv_helper(totals.streams, totals.blocks, totals.compressed_size, totals.uncompressed_size, totals.checks, totals.stream_padding); if (message_verbosity_get() >= V_DEBUG) { - printf(_(" Memory needed: %s MiB\n"), uint64_to_str( + printf(" %-*s %s MiB\n", COLON_STR(COLON_STR_MEMORY_NEEDED), + uint64_to_str( round_up_to_mib(totals.memusage_max), 0)); - printf(_(" Sizes in headers: %s\n"), + printf(" %-*s %s\n", COLON_STR(COLON_STR_SIZES_IN_HEADERS), totals.all_have_sizes ? _("Yes") : _("No")); + //printf(" %-*s %s\n", COLON_STR(COLON_STR_MINIMUM_XZ_VERSION), printf(_(" Minimum XZ Utils version: %s\n"), xz_ver_to_str(totals.min_version)); } @@ -1154,6 +1277,8 @@ list_file(const char *filename) return; } + init_field_widths(); + // Unset opt_stdout so that io_open_src() won't accept special files. // Set opt_force so that io_open_src() will follow symlinks. opt_stdout = false; diff --git a/src/xz/main.c b/src/xz/main.c index ca8a4680b688..c9c3deca2bf8 100644 --- a/src/xz/main.c +++ b/src/xz/main.c @@ -142,6 +142,20 @@ read_name(const args_info *args) int main(int argc, char **argv) { +#ifdef HAVE_PLEDGE + // OpenBSD's pledge(2) sandbox + // + // Unconditionally enable sandboxing with fairly relaxed promises. + // This is still way better than having no sandbox at all. :-) + // More strict promises will be made later in file_io.c if possible. + if (pledge("stdio rpath wpath cpath fattr", "")) { + // Don't translate the string or use message_fatal() as + // those haven't been initialized yet. + fprintf(stderr, "%s: Failed to enable the sandbox\n", argv[0]); + return E_ERROR; + } +#endif + #if defined(_WIN32) && !defined(__CYGWIN__) InitializeCriticalSection(&exit_status_cs); #endif diff --git a/src/xz/message.c b/src/xz/message.c index da90420fb897..4b5776568dd1 100644 --- a/src/xz/message.c +++ b/src/xz/message.c @@ -829,6 +829,15 @@ message_strm(lzma_ret code) case LZMA_STREAM_END: case LZMA_GET_CHECK: case LZMA_PROG_ERROR: + case LZMA_SEEK_NEEDED: + case LZMA_RET_INTERNAL1: + case LZMA_RET_INTERNAL2: + case LZMA_RET_INTERNAL3: + case LZMA_RET_INTERNAL4: + case LZMA_RET_INTERNAL5: + case LZMA_RET_INTERNAL6: + case LZMA_RET_INTERNAL7: + case LZMA_RET_INTERNAL8: // Without "default", compiler will warn if new constants // are added to lzma_ret, it is not too easy to forget to // add the new constants to this function. @@ -891,167 +900,20 @@ message_mem_needed(enum message_verbosity v, uint64_t memusage) } -/// \brief Convert uint32_t to a nice string for --lzma[12]=dict=SIZE -/// -/// The idea is to use KiB or MiB suffix when possible. -static const char * -uint32_to_optstr(uint32_t num) -{ - static char buf[16]; - - if ((num & ((UINT32_C(1) << 20) - 1)) == 0) - snprintf(buf, sizeof(buf), "%" PRIu32 "MiB", num >> 20); - else if ((num & ((UINT32_C(1) << 10) - 1)) == 0) - snprintf(buf, sizeof(buf), "%" PRIu32 "KiB", num >> 10); - else - snprintf(buf, sizeof(buf), "%" PRIu32, num); - - return buf; -} - - -extern void -message_filters_to_str(char buf[FILTERS_STR_SIZE], - const lzma_filter *filters, bool all_known) -{ - char *pos = buf; - size_t left = FILTERS_STR_SIZE; - - for (size_t i = 0; filters[i].id != LZMA_VLI_UNKNOWN; ++i) { - // Add the dashes for the filter option. A space is - // needed after the first and later filters. - my_snprintf(&pos, &left, "%s", i == 0 ? "--" : " --"); - - switch (filters[i].id) { - case LZMA_FILTER_LZMA1: - case LZMA_FILTER_LZMA2: { - const lzma_options_lzma *opt = filters[i].options; - const char *mode = NULL; - const char *mf = NULL; - - if (all_known) { - switch (opt->mode) { - case LZMA_MODE_FAST: - mode = "fast"; - break; - - case LZMA_MODE_NORMAL: - mode = "normal"; - break; - - default: - mode = "UNKNOWN"; - break; - } - - switch (opt->mf) { - case LZMA_MF_HC3: - mf = "hc3"; - break; - - case LZMA_MF_HC4: - mf = "hc4"; - break; - - case LZMA_MF_BT2: - mf = "bt2"; - break; - - case LZMA_MF_BT3: - mf = "bt3"; - break; - - case LZMA_MF_BT4: - mf = "bt4"; - break; - - default: - mf = "UNKNOWN"; - break; - } - } - - // Add the filter name and dictionary size, which - // is always known. - my_snprintf(&pos, &left, "lzma%c=dict=%s", - filters[i].id == LZMA_FILTER_LZMA2 - ? '2' : '1', - uint32_to_optstr(opt->dict_size)); - - // With LZMA1 also lc/lp/pb are known when - // decompressing, but this function is never - // used to print information about .lzma headers. - assert(filters[i].id == LZMA_FILTER_LZMA2 - || all_known); - - // Print the rest of the options, which are known - // only when compressing. - if (all_known) - my_snprintf(&pos, &left, - ",lc=%" PRIu32 ",lp=%" PRIu32 - ",pb=%" PRIu32 - ",mode=%s,nice=%" PRIu32 ",mf=%s" - ",depth=%" PRIu32, - opt->lc, opt->lp, opt->pb, - mode, opt->nice_len, mf, opt->depth); - break; - } - - case LZMA_FILTER_X86: - case LZMA_FILTER_POWERPC: - case LZMA_FILTER_IA64: - case LZMA_FILTER_ARM: - case LZMA_FILTER_ARMTHUMB: - case LZMA_FILTER_SPARC: { - static const char bcj_names[][9] = { - "x86", - "powerpc", - "ia64", - "arm", - "armthumb", - "sparc", - }; - - const lzma_options_bcj *opt = filters[i].options; - my_snprintf(&pos, &left, "%s", bcj_names[filters[i].id - - LZMA_FILTER_X86]); - - // Show the start offset only when really needed. - if (opt != NULL && opt->start_offset != 0) - my_snprintf(&pos, &left, "=start=%" PRIu32, - opt->start_offset); - - break; - } - - case LZMA_FILTER_DELTA: { - const lzma_options_delta *opt = filters[i].options; - my_snprintf(&pos, &left, "delta=dist=%" PRIu32, - opt->dist); - break; - } - - default: - // This should be possible only if liblzma is - // newer than the xz tool. - my_snprintf(&pos, &left, "UNKNOWN"); - break; - } - } - - return; -} - - extern void message_filters_show(enum message_verbosity v, const lzma_filter *filters) { if (v > verbosity) return; - char buf[FILTERS_STR_SIZE]; - message_filters_to_str(buf, filters, true); + char *buf; + const lzma_ret ret = lzma_str_from_filters(&buf, filters, + LZMA_STR_ENCODER | LZMA_STR_GETOPT_LONG, NULL); + if (ret != LZMA_OK) + message_fatal("%s", message_strm(ret)); + fprintf(stderr, _("%s: Filter chain: %s\n"), progname, buf); + free(buf); return; } @@ -1134,7 +996,7 @@ message_help(bool long_help) puts(_("\n Basic file format and compression options:\n")); puts(_( " -F, --format=FMT file format to encode or decode; possible values are\n" -" `auto' (default), `xz', `lzma', and `raw'\n" +" `auto' (default), `xz', `lzma', `lzip', and `raw'\n" " -C, --check=CHECK integrity check type: `none' (use with caution),\n" " `crc32', `crc64' (default), or `sha256'")); puts(_( @@ -1171,9 +1033,11 @@ message_help(bool long_help) puts(_( // xgettext:no-c-format " --memlimit-compress=LIMIT\n" " --memlimit-decompress=LIMIT\n" +" --memlimit-mt-decompress=LIMIT\n" " -M, --memlimit=LIMIT\n" " set memory usage limit for compression, decompression,\n" -" or both; LIMIT is in bytes, % of RAM, or 0 for defaults")); +" threaded decompression, or all of these; LIMIT is in\n" +" bytes, % of RAM, or 0 for defaults")); puts(_( " --no-adjust if compression settings exceed the memory usage limit,\n" @@ -1208,10 +1072,11 @@ message_help(bool long_help) puts(_( "\n" " --x86[=OPTS] x86 BCJ filter (32-bit and 64-bit)\n" +" --arm[=OPTS] ARM BCJ filter\n" +" --armthumb[=OPTS] ARM-Thumb BCJ filter\n" +" --arm64[=OPTS] ARM64 BCJ filter\n" " --powerpc[=OPTS] PowerPC BCJ filter (big endian only)\n" " --ia64[=OPTS] IA-64 (Itanium) BCJ filter\n" -" --arm[=OPTS] ARM BCJ filter (little endian only)\n" -" --armthumb[=OPTS] ARM-Thumb BCJ filter (little endian only)\n" " --sparc[=OPTS] SPARC BCJ filter\n" " Valid OPTS for all BCJ filters:\n" " start=NUM start offset for conversions (default=0)")); diff --git a/src/xz/message.h b/src/xz/message.h index 894ac7835f71..b264f82153a3 100644 --- a/src/xz/message.h +++ b/src/xz/message.h @@ -90,22 +90,6 @@ extern const char *message_strm(lzma_ret code); extern void message_mem_needed(enum message_verbosity v, uint64_t memusage); -/// Buffer size for message_filters_to_str() -#define FILTERS_STR_SIZE 512 - - -/// \brief Get the filter chain as a string -/// -/// \param buf Pointer to caller allocated buffer to hold -/// the filter chain string -/// \param filters Pointer to the filter chain -/// \param all_known If true, all filter options are printed. -/// If false, only the options that get stored -/// into .xz headers are printed. -extern void message_filters_to_str(char buf[FILTERS_STR_SIZE], - const lzma_filter *filters, bool all_known); - - /// Print the filter chain. extern void message_filters_show( enum message_verbosity v, const lzma_filter *filters); diff --git a/src/xz/options.c b/src/xz/options.c index 0c1ee221b22e..b434b0cf00ba 100644 --- a/src/xz/options.c +++ b/src/xz/options.c @@ -354,10 +354,5 @@ options_lzma(const char *str) if (options->lc + options->lp > LZMA_LCLP_MAX) message_fatal(_("The sum of lc and lp must not exceed 4")); - const uint32_t nice_len_min = options->mf & 0x0F; - if (options->nice_len < nice_len_min) - message_fatal(_("The selected match finder requires at " - "least nice=%" PRIu32), nice_len_min); - return options; } diff --git a/src/xz/private.h b/src/xz/private.h index d97c22cc6674..6414bdb5081c 100644 --- a/src/xz/private.h +++ b/src/xz/private.h @@ -45,7 +45,7 @@ # define STDERR_FILENO (fileno(stderr)) #endif -#ifdef HAVE_CAPSICUM +#if defined(HAVE_CAPSICUM) || defined(HAVE_PLEDGE) # define ENABLE_SANDBOX 1 #endif diff --git a/src/xz/suffix.c b/src/xz/suffix.c index 9d4fcd139b8f..6bb35d4229f6 100644 --- a/src/xz/suffix.c +++ b/src/xz/suffix.c @@ -119,9 +119,10 @@ uncompressed_name(const char *src_name, const size_t src_len) #ifdef __DJGPP__ { ".lzm", "" }, #endif - { ".tlz", ".tar" }, - // { ".gz", "" }, - // { ".tgz", ".tar" }, + { ".tlz", ".tar" }, // Both .tar.lzma and .tar.lz +#ifdef HAVE_LZIP_DECODER + { ".lz", "" }, +#endif }; const char *new_suffix = ""; @@ -208,12 +209,15 @@ compressed_name(const char *src_name, size_t src_len) #endif ".tlz", NULL -/* +#ifdef HAVE_LZIP_DECODER + // This is needed to keep the table indexing in sync with + // enum format_type from coder.h. }, { - ".gz", - ".tgz", - NULL +/* + ".lz", */ + NULL +#endif }, { // --format=raw requires specifying the suffix // manually or using stdout. @@ -221,8 +225,11 @@ compressed_name(const char *src_name, size_t src_len) } }; - // args.c ensures this. + // args.c ensures these. assert(opt_format != FORMAT_AUTO); +#ifdef HAVE_LZIP_DECODER + assert(opt_format != FORMAT_LZIP); +#endif const size_t format = opt_format - 1; const char *const *suffixes = all_suffixes[format]; @@ -299,9 +306,11 @@ compressed_name(const char *src_name, size_t src_len) // xz foo.tar -> foo.txz // xz -F lzma foo.tar -> foo.tlz static const char *const tar_suffixes[] = { - ".txz", - ".tlz", - // ".tgz", + ".txz", // .tar.xz + ".tlz", // .tar.lzma +/* + ".tlz", // .tar.lz +*/ }; suffix = tar_suffixes[format]; suffix_len = 4; diff --git a/src/xz/util.c b/src/xz/util.c index a1339f4fdf3c..9f9a8fb024e4 100644 --- a/src/xz/util.c +++ b/src/xz/util.c @@ -261,18 +261,6 @@ my_snprintf(char **pos, size_t *left, const char *fmt, ...) extern bool -is_empty_filename(const char *filename) -{ - if (filename[0] == '\0') { - message_error(_("Empty filename, skipping")); - return true; - } - - return false; -} - - -extern bool is_tty_stdin(void) { const bool ret = isatty(STDIN_FILENO); diff --git a/src/xz/util.h b/src/xz/util.h index a2516bf96803..4a536f52ea4e 100644 --- a/src/xz/util.h +++ b/src/xz/util.h @@ -105,10 +105,6 @@ extern void my_snprintf(char **pos, size_t *left, const char *fmt, ...) lzma_attribute((__format__(__printf__, 3, 4))); -/// \brief Check if filename is empty and print an error message -extern bool is_empty_filename(const char *filename); - - /// \brief Test if stdin is a terminal /// /// If stdin is a terminal, an error message is printed and exit status set diff --git a/src/xz/xz.1 b/src/xz/xz.1 index 066b13a213eb..caa5a068173e 100644 --- a/src/xz/xz.1 +++ b/src/xz/xz.1 @@ -5,7 +5,7 @@ .\" This file has been put into the public domain. .\" You can do whatever you want with this file. .\" -.TH XZ 1 "2022-10-25" "Tukaani" "XZ Utils" +.TH XZ 1 "2022-12-01" "Tukaani" "XZ Utils" . .SH NAME xz, unxz, xzcat, lzma, unlzma, lzcat \- Compress or decompress .xz and .lzma files @@ -62,6 +62,11 @@ format, but the legacy format used by LZMA Utils and raw compressed streams with no container format headers are also supported. +In addition, decompression of the +.B .lz +format used by +.B lzip +is supported. .PP .B xz compresses or decompresses each @@ -102,9 +107,10 @@ or is appended to the source filename to get the target filename. .IP \(bu 3 When decompressing, the -.B .xz +.BR .xz , +.BR .lzma , or -.B .lzma +.B .lz suffix is removed from the filename to get the target filename. .B xz also recognizes the suffixes @@ -158,8 +164,9 @@ doesn't have a suffix of any of the supported file formats .RB ( .xz , .BR .txz , .BR .lzma , +.BR .tlz , or -.BR .tlz ). +.BR .lz ). .PP After successfully compressing or decompressing the .IR file , @@ -507,8 +514,9 @@ in addition to files with the .BR .xz , .BR .txz , .BR .lzma , +.BR .tlz , or -.B .tlz +.B .lz suffix. If the source file has the suffix .IR .suf , @@ -575,6 +583,34 @@ The alternative name .B alone is provided for backwards compatibility with LZMA Utils. .TP +.B lzip +Accept only +.B .lz +files when decompressing. +Compression is not supported. +.IP "" +The +.B .lz +format version 0 and the unextended version 1 are supported. +Version 0 files were produced by +.B lzip +1.3 and older. +Such files aren't common but may be found from file archives +as a few source packages were released in this format. +People might have old personal files in this format too. +Decompression support for the format version 0 was removed in +.B lzip +1.18. +.IP "" +.B lzip +1.4 and later create files in the format version 1. +The sync flush marker extension to the format version 1 was added in +.B lzip +1.6. +This extension is rarely used and isn't supported by +.B xz +(diagnosed as corrupt input). +.TP .B raw Compress or uncompress a raw stream (no headers). This is meant for advanced users only. @@ -965,15 +1001,28 @@ the last one takes effect. If the compression settings exceed the .IR limit , .B xz -will adjust the settings downwards so that +will attempt to adjust the settings downwards so that the limit is no longer exceeded and display a notice that automatic adjustment was done. -Such adjustments are not made when compressing with +The adjustments are done in this order: +reducing the number of threads, +switching to single-threaded mode +if even one thread in multi-threaded mode exceeds the +.IR limit , +and finally reducing the LZMA2 dictionary size. +.IP "" +When compressing with .B \-\-format=raw or if .B \-\-no\-adjust -has been specified. -In those cases, an error is displayed and +has been specified, +only the number of threads may be reduced +since it can be done without affecting the compressed output. +.IP "" +If the +.I limit +cannot be met even with the adjustments described above, +an error is displayed and .B xz will exit with exit status 1. .IP "" @@ -1012,16 +1061,6 @@ This is currently equivalent to setting the to .B max (no memory usage limit). -Once multithreading support has been implemented, -there may be a difference between -.B 0 -and -.B max -for the multithreaded case, so it is recommended to use -.B 0 -instead of -.B max -until the details have been decided. .RE .IP "" For 32-bit @@ -1064,16 +1103,80 @@ See for possible ways to specify the .IR limit . .TP +.BI \-\-memlimit\-mt\-decompress= limit +Set a memory usage limit for multi-threaded decompression. +This can only affect the number of threads; +this will never make +.B xz +refuse to decompress a file. +If +.I limit +is too low to allow any multi-threading, the +.I limit +is ignored and +.B xz +will continue in single-threaded mode. +Note that if also +.B \-\-memlimit\-decompress +is used, +it will always apply to both single-threaded and multi-threaded modes, +and so the effective +.I limit +for multi-threading will never be higher than the limit set with +.BR \-\-memlimit\-decompress . +.IP "" +In contrast to the other memory usage limit options, +.BI \-\-memlimit\-mt\-decompress= limit +has a system-specific default +.IR limit . +.B "xz \-\-info\-memory" +can be used to see the current value. +.IP "" +This option and its default value exist +because without any limit the threaded decompressor +could end up allocating an insane amount of memory with some input files. +If the default +.I limit +is too low on your system, +feel free to increase the +.I limit +but never set it to a value larger than the amount of usable RAM +as with appropriate input files +.B xz +will attempt to use that amount of memory +even with a low number of threads. +Running out of memory or swapping +will not improve decompression performance. +.IP "" +See +.BI \-\-memlimit\-compress= limit +for possible ways to specify the +.IR limit . +Setting +.I limit +to +.B 0 +resets the +.I limit +to the default system-specific value. +.IP "" +.TP \fB\-M\fR \fIlimit\fR, \fB\-\-memlimit=\fIlimit\fR, \fB\-\-memory=\fIlimit This is equivalent to specifying .BI \-\-memlimit\-compress= limit -\fB\-\-memlimit\-decompress=\fIlimit\fR. +.BI \-\-memlimit-decompress= limit +\fB\-\-memlimit\-mt\-decompress=\fIlimit\fR. .TP .B \-\-no\-adjust -Display an error and exit if the compression settings exceed -the memory usage limit. -The default is to adjust the settings downwards so -that the memory usage limit is not exceeded. +Display an error and exit if the memory usage limit cannot be +met without adjusting settings that affect the compressed output. +That is, this prevents +.B xz +from switching the encoder from multi-threaded mode to single-threaded mode +and from reducing the LZMA2 dictionary size. +Even when this option is used the number of threads may be reduced +to meet the memory usage limit as that won't affect the compressed output. +.IP "" Automatic adjusting is always disabled when creating raw streams .RB ( \-\-format=raw ). .TP @@ -1085,13 +1188,66 @@ to a special value .B 0 makes .B xz -use as many threads as there are CPU cores on the system. -The actual number of threads can be less than +use up to as many threads as the processor(s) on the system support. +The actual number of threads can be fewer than .I threads if the input file is not big enough for threading with the given settings or if using more threads would exceed the memory usage limit. .IP "" +The single-threaded and multi-threaded compressors produce different output. +Single-threaded compressor will give the smallest file size but +only the output from the multi-threaded compressor can be decompressed +using multiple threads. +Setting +.I threads +to +.B 1 +will use the single-threaded mode. +Setting +.I threads +to any other value, including +.BR 0 , +will use the multi-threaded compressor +even if the system supports only one hardware thread. +.RB ( xz +5.2.x +used single-threaded mode in this situation.) +.IP "" +To use multi-threaded mode with only one thread, set +.I threads +to +.BR +1 . +The +.B + +prefix has no effect with values other than +.BR 1 . +A memory usage limit can still make +.B xz +switch to single-threaded mode unless +.B \-\-no\-adjust +is used. +Support for the +.B + +prefix was added in +.B xz +5.4.0. +.IP "" +If an automatic number of threads has been requested and +no memory usage limit has been specified, +then a system-specific default soft limit will be used to possibly +limit the number of threads. +It is a soft limit in sense that it is ignored +if the number of threads becomes one, +thus a soft limit will never stop +.B xz +from compressing or decompressing. +This default soft limit will not make +.B xz +switch from multi-threaded mode to single-threaded mode. +The active limits can be seen with +.BR "xz \-\-info\-memory" . +.IP "" Currently the only threading method is to split the input into blocks and compress them independently from each other. The default block size depends on the compression level and @@ -1099,13 +1255,13 @@ can be overridden with the .BI \-\-block\-size= size option. .IP "" -Threaded decompression hasn't been implemented yet. -It will only work on files that contain multiple blocks -with size information in block headers. -All files compressed in multi-threaded mode meet this condition, +Threaded decompression only works on files that contain +multiple blocks with size information in block headers. +All large enough files compressed in multi-threaded mode +meet this condition, but files compressed in single-threaded mode don't even if .BI \-\-block\-size= size -is used. +has been used. . .SS "Custom compressor filter chains" A custom filter chain allows specifying @@ -1537,14 +1693,16 @@ and \fB\-\-x86\fR[\fB=\fIoptions\fR] .PD 0 .TP -\fB\-\-powerpc\fR[\fB=\fIoptions\fR] -.TP -\fB\-\-ia64\fR[\fB=\fIoptions\fR] -.TP \fB\-\-arm\fR[\fB=\fIoptions\fR] .TP \fB\-\-armthumb\fR[\fB=\fIoptions\fR] .TP +\fB\-\-arm64\fR[\fB=\fIoptions\fR] +.TP +\fB\-\-powerpc\fR[\fB=\fIoptions\fR] +.TP +\fB\-\-ia64\fR[\fB=\fIoptions\fR] +.TP \fB\-\-sparc\fR[\fB=\fIoptions\fR] .PD Add a branch/call/jump (BCJ) filter to the filter chain. @@ -1553,7 +1711,7 @@ in the filter chain. .IP "" A BCJ filter converts relative addresses in the machine code to their absolute counterparts. -This doesn't change the size of the data, +This doesn't change the size of the data but it increases redundancy, which can help LZMA2 to produce 0\(en15\ % smaller .B .xz @@ -1562,21 +1720,8 @@ The BCJ filters are always reversible, so using a BCJ filter for wrong type of data doesn't cause any data loss, although it may make the compression ratio slightly worse. -.IP "" -It is fine to apply a BCJ filter on a whole executable; -there's no need to apply it only on the executable section. -Applying a BCJ filter on an archive that contains both executable -and non-executable files may or may not give good results, -so it generally isn't good to blindly apply a BCJ filter when -compressing binary packages for distribution. -.IP "" -These BCJ filters are very fast and -use insignificant amount of memory. -If a BCJ filter improves compression ratio of a file, -it can improve decompression speed at the same time. -This is because, on the same hardware, -the decompression speed of LZMA2 is roughly -a fixed number of bytes of compressed data per second. +The BCJ filters are very fast and +use an insignificant amount of memory. .IP "" These BCJ filters have known problems related to the compression ratio: @@ -1588,21 +1733,20 @@ have the addresses in the instructions filled with filler values. These BCJ filters will still do the address conversion, which will make the compression worse with these files. .IP \(bu 3 -Applying a BCJ filter on an archive containing multiple similar -executables can make the compression ratio worse than not using -a BCJ filter. -This is because the BCJ filter doesn't detect the boundaries -of the executable files, and doesn't reset -the address conversion counter for each executable. +If a BCJ filter is applied on an archive, +it is possible that it makes the compression ratio +worse than not using a BCJ filter. +For example, if there are similar or even identical executables +then filtering will likely make the files less similar +and thus compression is worse. +The contents of non-executable files in the same archive can matter too. +In practice one has to try with and without a BCJ filter to see +which is better in each situation. .RE .IP "" -Both of the above problems will be fixed -in the future in a new filter. -The old BCJ filters will still be useful in embedded systems, -because the decoder of the new filter will be bigger -and use more memory. -.IP "" Different instruction sets have different alignment: +the executable file must be aligned to a multiple of +this value in the input data to make the filter work. .RS .RS .PP @@ -1612,11 +1756,12 @@ l n l l n l. Filter;Alignment;Notes x86;1;32-bit or 64-bit x86 +ARM;4; +ARM-Thumb;2; +ARM64;4;4096-byte alignment is best PowerPC;4;Big endian only -ARM;4;Little endian only -ARM-Thumb;2;Little endian only -IA-64;16;Big or little endian -SPARC;4;Big or little endian +IA-64;16;Itanium +SPARC;4; .TE .RE .RE @@ -1627,6 +1772,8 @@ the LZMA2 options are set to match the alignment of the selected BCJ filter. For example, with the IA-64 filter, it's good to set .B pb=4 +or even +.B pb=4,lp=4,lc=0 with LZMA2 (2^4=16). The x86 filter is an exception; it's usually good to stick to LZMA2's default @@ -1774,6 +1921,7 @@ for details. .TP .B \-\-info\-memory Display, in human-readable format, how much physical memory (RAM) +and how many processor threads .B xz thinks the system has and the memory usage limits for compression and decompression, and exit successfully. @@ -1858,15 +2006,50 @@ and .B "xz \-\-robot \-\-info\-memory" prints a single line with three tab-separated columns: .IP 1. 4 -Total amount of physical memory (RAM) in bytes +Total amount of physical memory (RAM) in bytes. .IP 2. 4 -Memory usage limit for compression in bytes. -A special value of zero indicates the default setting, +Memory usage limit for compression in bytes +.RB ( \-\-memlimit\-compress ). +A special value of +.B 0 +indicates the default setting which for single-threaded mode is the same as no limit. .IP 3. 4 -Memory usage limit for decompression in bytes. -A special value of zero indicates the default setting, +Memory usage limit for decompression in bytes +.RB ( \-\-memlimit\-decompress ). +A special value of +.B 0 +indicates the default setting which for single-threaded mode is the same as no limit. +.IP 4. 4 +Since +.B xz +5.3.4alpha: +Memory usage for multi-threaded decompression in bytes +.RB ( \-\-memlimit\-mt\-decompress ). +This is never zero because a system-specific default value +shown in the column 5 +is used if no limit has been specified explicitly. +This is also never greater than the value in the column 3 +even if a larger value has been specified with +.BR \-\-memlimit\-mt\-decompress . +.IP 5. 4 +Since +.B xz +5.3.4alpha: +A system-specific default memory usage limit +that is used to limit the number of threads +when compressing with an automatic number of threads +.RB ( \-\-threads=0 ) +and no memory usage limit has been specified +.RB ( \-\-memlimit\-compress ). +This is also used as the default value for +.BR \-\-memlimit\-mt\-decompress . +.IP 6. 4 +Since +.B xz +5.3.4alpha: +Number of available processor threads. .PP In the future, the output of .B "xz \-\-robot \-\-info\-memory" |