New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Zstandard support (native or via zlibwrapper) [CORE6213] #6457
Comments
Modified by: Basil A. Sidorov (basid)Attachment: http://README-zlibWrapper.md [ 13410 ] |
Commented by: @dyemanov For record-level (de)compression I'd rather consider LZ4 instead. |
Modified by: @AlexPeshkoffassignee: Alexander Peshkov [ alexpeshkoff ] |
Commented by: @AlexPeshkoff WireCompression makes sense for slow links (like modem) in WAN case. For LAN with gigabit link it does not make sense. May be in WAN case with fast links zstandard will be efficient. BTW, does format of compressed stream differ for zlib/zstd? |
Commented by: Basil A. Sidorov (basid) Customers may use LAN-WAN combination: EDS between branch office and, typically WireCompression are globally enabled in firebird.conf |
Commented by: Sascha Michel (datiscum1) I've been watching the zlib-ng project for a while and did some speed tests today. Of course, it's not a leap like zstd, but it's clearly faster and above all 100% compatible with the old zlib. https://github.com/zlib-ng/zlib-ng Please note that zlib-ng in zlib-compat mode is API-compatible but not ABI-compatible, meaning that you cannot simply replace the zlib library/dll files and expect the application to work. The application will need to be recompiled against the zlib-ng headers and libs to ensure full compatability. zpipe_libzOrg = static linked with standard zlib. zpipe_ipp "Building a faster ZLIB with Intel Integrated Performance Primitives" Database does not contain packed data. test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe_libzOrg | mbuffer -o /dev/null test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe | mbuffer -o /dev/null test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe_SystemOptimized | mbuffer -o /dev/null --- test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zstd | mbuffer -o /dev/null test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | lz4 -3 | mbuffer -o /dev/null test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe_ipp | mbuffer -o /dev/null Database contains 99% blobs packed with zlib. test@HillSrv:/DB$ time mbuffer -i MATWDE_20180815.fbk | zpipe_libzOrg | mbuffer -o /dev/null test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zpipe | mbuffer -o /dev/null test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zpipe_SystemOptimized | mbuffer -o /dev/null --- test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zstd | mbuffer -o /dev/null test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | lz4 | mbuffer -o /dev/null test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zpipe_ipp | mbuffer -o /dev/null |
Submitted by: Basil A. Sidorov (basid)
Attachments:
README-zlibWrapper.md
Votes: 1
http://zstandard.net - stable and mature project, offered fast (de)compressor with many features and dual (BSD/GPL) licensing.
Firebird 3.0 may use WireCompression, but zlib are slow for modern gigabit link.
Fast (de)compression may be profitable for record chains in database file, for database crypt and so on.
For quick transition from zlib to zstd exist zlibwrapper as part of zstd source tree.
The text was updated successfully, but these errors were encountered: