Native zlib libraries (original) (raw)

Xueming Shen xueming.shen at oracle.com
Tue Jul 31 04:32:23 UTC 2012

totalbytes_Read/Written are being tested though not the size > 4G. That test mainly tests the > 64K entries and total size > 4G (lots of > 32-bit offsets...), but the total bytes read/ written are all < 4G (for each entry). An easy manual test I always do after run that test is to jar the resulting > 6g file and unjar it. That will actually check the total bytes read/written

4G case.

If you keep using the same JRE which uses the system zlib, you might still be able to jar and unjar a single > 4G file. The total bytes read/written are used as one of the validation in zip format, ZipOutputStream writes these two number as size/csize fields of the loc and cen table, the ZipInputStream then verifies these numbers when inflating. This round-trip happens to work if the zos and zis both use the same JRE, in which both throw away the high bits of the total numbers (but their low bits are still match) But a simple jar tvf tells you the size number is incorrect. Also , if you jar some other files together with this > 4G file, you probably will get error when trying to extract individual entry file(s) out, as the position info of those files probably will get messed up.

Btw, the code to support > 4G total_in/out at java level is ready here, need some more tests and the bugid for this one and yours.

http://cr.openjdk.java.net/~sherman/undozip/webrev/

-Sherman

On 7/30/12 3:20 AM, Andrew Hughes wrote:

----- Original Message -----

On 07/27/2012 10:37 AM, Andrew Hughes wrote:

----- Original Message -----

The zip64 support (totalin/out) part probably can be done at Java level (ignore the totalin/out in ztreams). Need to remove this dependency. Will take a look later.

Yes, it seems they still mention the size of totalin/out on the website on the zlib site, and that they shouldn't be relied on: http://www.zlib.net/zlibfaq.html#faq32 "Note however that the strm.totalin and strmtotalout counters may be limited to 4 GB. These counters are provided as a convenience and are not used internally by inflate() or deflate(). The application can easily set up its own counters updated after each call of inflate() or deflate() to count beyond 4 GB" "The word "may" appears several times above since there is a 4 GB limit only if the compiler's long type is 32 bits. If the compiler's long type is 64 bits, then the limit is 16 exabytes." I notice a test went in with the 64-bit support, but I assume it can't test these counters as the Deflater for a ZipStream is protected. At least, they aren't failing on our builds with system zlib. That test is not configured to be run for "auto testing", it just takes too long to zip/unzip a 4G+ file. I use it manually to test the ZIP64 support. Yes, sorry, I noticed this after posting. I've since run it manually and it passes with a system zlib, both on the zip it creates and one I created containing just a single 4.2gb file (a RHEL ISO image zipped). But I'm not sure if it's testing total bytes read for overflow. I will give it a try to remove this dependency next week. Thanks. It would be helpful if you can help "migrate" the icetea patch. How does the icetea patch work now? Always use the system zlib, if it presents? any configurable option to switch on and off? It adds a switch and also support for setting the CFLAGS/LIBS: _USESYSTEMZLIB=true _ _ZLIBLIBS="$(pkg-config --libs zlib)" _ _ZLIBCFLAGS="$(pkg-config --cflags zlib)" _ This is the same nomenclature we use for the other libraries too (lcms, jpeg, png, gif, cups, etc.) I'll try and post the patch later today. -Sherman Are you actively working on this now or shall I take a look? -Sherman

On 7/11/2012 12:47 AM, Alan Bateman wrote: On 05/07/2012 17:11, Andrew Hughes wrote: ----- Original Message ----- Is there a way to get the native zlib libraries to get picked up instead of the hardcoded version within the JVM?

-- Azeem Jiva @javawithjiva We have this in IcedTea (USESYSTEMZLIB=true) and intend to get it upstream. However, I don't see how this is related to HotSpot, as the zlib usage is in the jdk tree. I think we need to (re)start the discussion on core-libs-dev with a view to eliminating the patches that the JDK has to zlib, see: http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/native/java/util/zip/zlib-1.2.5/patches/ChangeLogjava

One of these changes relates to the zip64 support and I believe there are corner cases when inflating or deflating>2GB that won't work if using the system zlib. Sherman will likely recall the details. Given that the new build already supports using the system zlib (at least on Linux) then it would be good to sort this out so that it just works. -Alan

More information about the core-libs-dev mailing list