sigsegv on porter stemmer (Lucene, but also otherwise) (original) (raw)

Uwe Schindler uschindler at apache.org
Mon Jul 25 12:51:33 PDT 2011

Hi thanks for taking care!

Thanks for the workaround. We already found another workaround to get this running in our 2-hourly Lucene builds at:

https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/ https://builds.apache.org/job/Lucene-Solr-tests-only-3.x-java7/

It would be nice if you could look into the console logs of the failed builds on Saturday - you can see the bug in the earlier builds only (with always different stack traces). We drilled it down to one method (not sure if this information clipped out of the bug report. We then disabled compilation for only this affected method, its PorterStemmer.ends(...):

-XX:CompileCommand=exclude,org/apache/lucene/analysis/en/PorterStemmer,ends -XX:CompileCommand=exclude,org/apache/lucene/analysis/PorterStemmer,ends

(we have different class names in stable 3.x branch and trunk).

We also see other random test failures not happening with Java 5 and Java 6, it would be nice, if you could review, too.

One big bug in loops affected also Java 1.6.0_18 (still not fixed): our DataInput.readVInt method was incorrectly compiled in the case that MappedByteBuffer.get()/lucene.DataInput.readByte() was inlined, leading to simply wrong results (the method returned a decoded integer that was different than expected results). See the unwinded loops in https://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/src/java/org/apache /lucene/store/DataInput.java

I hope this all helps you in finding more bad loop optimization bugs, all of those issues seem to be related to this special optimization in loops. The latest lucene builds also contains a failure in a test case only happening with Java 7 (not on every test run, so unreproducible). So it might be good for you to watch our Lucene builds also for other bugs.

Some of the other developers already say, we should not trust any loops in java anymore and recommend not to use Java 7 with Apache Lucene/Solr, and that's bad news :(

Thanks for the help, Uwe

On Mon, Jul 25, 2011 at 7:49 PM, Vladimir Kozlov <vladimir.kozlov at oracle.com> wrote:

Thank you very much, Dawid, for providing the test case. Bug was filed in wrong category so we did not know about it. I will work on it since it could be my changes in loop optimizations. Use next flag as workaround:

-XX:-UseLoopPredicate Thanks, Vladimir Dawid Weiss wrote:

Hello everyone, I am an Apache Lucene developer, we've been running tests with Java 1.7 and this came up: http://bugs.sun.com/bugdatabase/viewbug.do?bugid=7070134 Porter stemmer is pretty widely used for shallow NLP, not only in Lucene. It'd be interesting to hear from jit gurus what's causing this (the problem does not occur in 1.6). Thanks in advance, Dawid

Uwe Schindler uschindler at apache.org Apache Lucene PMC Member / Committer Bremen, Germany http://lucene.apache.org/

More information about the hotspot-compiler-dev mailing list