Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Zstandard support (native or via zlibwrapper) [CORE6213] #6457

Open
firebird-automations opened this issue Dec 27, 2019 · 6 comments
Open

Comments

@firebird-automations
Copy link
Collaborator

Submitted by: Basil A. Sidorov (basid)

Attachments:
README-zlibWrapper.md

Votes: 1

http://zstandard.net - stable and mature project, offered fast (de)compressor with many features and dual (BSD/GPL) licensing.
Firebird 3.0 may use WireCompression, but zlib are slow for modern gigabit link.
Fast (de)compression may be profitable for record chains in database file, for database crypt and so on.

For quick transition from zlib to zstd exist zlibwrapper as part of zstd source tree.

@firebird-automations
Copy link
Collaborator Author

Modified by: Basil A. Sidorov (basid)

Attachment: http://README-zlibWrapper.md [ 13410 ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @dyemanov

For record-level (de)compression I'd rather consider LZ4 instead.

@firebird-automations
Copy link
Collaborator Author

Modified by: @AlexPeshkoff

assignee: Alexander Peshkov [ alexpeshkoff ]

@firebird-automations
Copy link
Collaborator Author

Commented by: @AlexPeshkoff

WireCompression makes sense for slow links (like modem) in WAN case. For LAN with gigabit link it does not make sense. May be in WAN case with fast links zstandard will be efficient. BTW, does format of compressed stream differ for zlib/zstd?

@firebird-automations
Copy link
Collaborator Author

Commented by: Basil A. Sidorov (basid)

Customers may use LAN-WAN combination: EDS between branch office and, typically WireCompression are globally enabled in firebird.conf
zstd packet format, of course, are different from deflate packet format.

@firebird-automations
Copy link
Collaborator Author

Commented by: Sascha Michel (datiscum1)

I've been watching the zlib-ng project for a while and did some speed tests today. Of course, it's not a leap like zstd, but it's clearly faster and above all 100% compatible with the old zlib.
The Intel library "zpipe_ipp" already caused me compatibility problems when I wanted to unpack data with a standard zlib.
Maybe someone will think about the zlib-ng for Firebird.

https://github.com/zlib-ng/zlib-ng
Zlib-ng can be compiled in zlib-compat mode, suitable for zlib-replacement in a single application or system-wide.

Please note that zlib-ng in zlib-compat mode is API-compatible but not ABI-compatible, meaning that you cannot simply replace the zlib library/dll files and expect the application to work. The application will need to be recompiled against the zlib-ng headers and libs to ensure full compatability.

zpipe_libzOrg = static linked with standard zlib.
zpipe = static linked with zlib-ng compat NOT optimized for system cpu
zpipe_SystemOptimized = static linked with zlib-ng compat optimized for system cpu

zpipe_ipp "Building a faster ZLIB with Intel Integrated Performance Primitives"

Database does not contain packed data.

test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe_libzOrg | mbuffer -o /dev/null
summary: 428 MiByte in 5.1sec - average of 84.3 MiB/s
summary: 93.1 MiByte

test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe | mbuffer -o /dev/null
summary: 428 MiByte in 2.9sec - average of 146 MiB/s
summary: 90.6 MiByte

test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe_SystemOptimized | mbuffer -o /dev/null
summary: 428 MiByte in 2.7sec - average of 161 MiB/s
summary: 90.6 MiByte

---

test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zstd | mbuffer -o /dev/null
summary: 428 MiByte in 1.3sec - average of 339 MiB/s
summary: 71.2 MiByte

test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | lz4 -3 | mbuffer -o /dev/null
summary: 428 MiByte in 3.6sec - average of 119 MiB/s
summary: 88.9 MiByte

test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe_ipp | mbuffer -o /dev/null
summary: 428 MiByte in 2.4sec - average of 182 MiB/s
summary: 94.6 MiByte

Database contains 99% blobs packed with zlib.

test@HillSrv:/DB$ time mbuffer -i MATWDE_20180815.fbk | zpipe_libzOrg | mbuffer -o /dev/null
summary: 1187 MiByte in 27.7sec - average of 42.8 MiB/s
summary: 1177 MiByte

test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zpipe | mbuffer -o /dev/null
summary: 1187 MiByte in 20.0sec - average of 59.4 MiB/s
summary: 1177 MiByte

test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zpipe_SystemOptimized | mbuffer -o /dev/null
summary: 1187 MiByte in 19.5sec - average of 60.7 MiB/s
summary: 1177 MiByte

---

test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zstd | mbuffer -o /dev/null
summary: 1187 MiByte in 1.4sec - average of 843 MiB/s
summary: 1180 MiByte

test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | lz4 | mbuffer -o /dev/null
summary: 1187 MiByte in 2.0sec - average of 583 MiB/s
summary: 1183 MiByte

test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zpipe_ipp | mbuffer -o /dev/null
summary: 1187 MiByte in 21.7sec - average of 54.6 MiB/s
summary: 1177 MiByte

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants