sigsegv on porter stemmer (Lucene, but also otherwise) (original) (raw)

Vladimir Kozlov vladimir.kozlov at oracle.com
Tue Jul 26 12:44:22 PDT 2011

Thank you, Uwe

I will send the patch for reviews shortly.

About java 7 release. We are late to do any bugs fixes in GA which should happen soon. All loop optimization fixes will go definitely into jdk7 update 2. We will try to push them into update 1 (which is targeted only for security fixes) but we can't promise.

There is going discussion about using current Hotspot VM in future jdk6 updates but there is no decision yet. Note: current Hotspot VM sources are targeted for JDK8 and jdk7 updates only.

Regards, Vladimir

Uwe Schindler wrote:

Hi,

With the help from Vladimir we patched our test platform and verified that the bug is fixed with Vladimirs latest patches (including already committed patches not available in the latest Java 7 developer preview), so thanks for that. I think you may post your patches for review here, we are happy! I was able to apply the patches to our Jenkins server's Java 7 preview installation, which runs on FreeBSD. I prepared a combined patch to be placed in the ports directory and recompiled the openjdk7 package. After restarting our Jenkins jobs, all was fine, even the other random test failures without SIGSEGV seem to be resolved (e.g. Lucene Faceting module caused broken index on disk). More information can also be found in Lucene's JIRA: https://issues.apache.org/jira/browse/LUCENE-3335 You can find the working builds here: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/, starting with build #27 (some builds still failed because of another bug in IBM's ICU when the default locale, which is randomly set by our test framework, contained one of the new Java 7 ones - we have now a workaround) We now hope, that Java 7 will not be released with that bugs, because quite a lot loops got miscompiled without the fixes, which will break lots of applications. We are afraid of people using Lucene/Solr with Oracle JDK 1.7.00 on July 28th and will corrupt their indexes. Is there a chance to get the fixes in or delay the release? Also it would be interesting to know how this affects JDK 1.6.0, because 26 did not seem to contain the broken original fix that caused the porter failure and 1.6.0 is still broken with our readVInt method. We hope the complete fix will be in 27. Finally, we have a list of all issues related to Java7, if you are interested: http://s.apache.org/Java7Failures Uwe

-----Original Message----- From: hotspot-compiler-dev-bounces at openjdk.java.net [mailto:hotspot-_ _compiler-dev-bounces at openjdk.java.net] On Behalf Of Uwe Schindler Sent: Monday, July 25, 2011 9:52 PM To: vladimir.kozlov at oracle.com; hotspot-compiler-dev at openjdk.java.net Subject: Re: sigsegv on porter stemmer (Lucene, but also otherwise)

Hi thanks for taking care! Thanks for the workaround. We already found another workaround to get this running in our 2-hourly Lucene builds at: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/ https://builds.apache.org/job/Lucene-Solr-tests-only-3.x-java7/ It would be nice if you could look into the console logs of the failed builds on Saturday - you can see the bug in the earlier builds only (with always different stack traces). We drilled it down to one method (not sure if this information clipped out of the bug report. We then disabled compilation for only this affected method, its PorterStemmer.ends(...): - XX:CompileCommand=exclude,org/apache/lucene/analysis/en/PorterStem mer,ends - XX:CompileCommand=exclude,org/apache/lucene/analysis/PorterStemmer, ends (we have different class names in stable 3.x branch and trunk). We also see other random test failures not happening with Java 5 and Java 6, it would be nice, if you could review, too. One big bug in loops affected also Java 1.6.018 (still not fixed): our DataInput.readVInt method was incorrectly compiled in the case that MappedByteBuffer.get()/lucene.DataInput.readByte() was inlined, leading to simply wrong results (the method returned a decoded integer that was different than expected results). See the unwinded loops in https://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/src/java/org/ap ache /lucene/store/DataInput.java I hope this all helps you in finding more bad loop optimization bugs, all of those issues seem to be related to this special optimization in loops. The latest lucene builds also contains a failure in a test case only happening with Java 7 (not on every test run, so unreproducible). So it might be good for you to watch our Lucene builds also for other bugs. Some of the other developers already say, we should not trust any loops in java anymore and recommend not to use Java 7 with Apache Lucene/Solr, and that's bad news :( Thanks for the help, Uwe On Mon, Jul 25, 2011 at 7:49 PM, Vladimir Kozlov <vladimir.kozlov at_ _oracle.com> wrote: Thank you very much, Dawid, for providing the test case. Bug was filed in wrong category so we did not know about it. I will work on it since it could be my changes in loop optimizations. Use next flag as workaround:

-XX:-UseLoopPredicate Thanks, Vladimir Dawid Weiss wrote: Hello everyone,

I am an Apache Lucene developer, we've been running tests with Java 1.7 and this came up: http://bugs.sun.com/bugdatabase/viewbug.do?bugid=7070134 Porter stemmer is pretty widely used for shallow NLP, not only in Lucene. It'd be interesting to hear from jit gurus what's causing this (the problem does not occur in 1.6). Thanks in advance, Dawid ----- Uwe Schindler uschindler at apache.org Apache Lucene PMC Member / Committer Bremen, Germany http://lucene.apache.org/

More information about the hotspot-compiler-dev mailing list