Issue 28700: test_dbm failure: KeyError: b'0' (intermittent in 3.5, reliable in 3.6) (original) (raw)

test_dbm.py fails reliably for me in 3.6, but in 3.5 it passes ~80% of the time. The failure in both cases is KeyError: b'0', which has come up previously in http://bugs.python.org/issue20094 and http://bugs.python.org/issue14120.

But since we've switched from 20% failure rate to 100% failure rate, I figured something must have changed. I used "hg bisect" to track it down to a recent commit:

changeset: 103360:0bd618fe0639 user: Victor Stinner <victor.stinner@gmail.com> date: Wed Sep 07 17:40:12 2016 -0700 summary: Implement compact dict

Here is how it fails:

$ ./python -m test -v test_dbm == CPython 3.6.0a4+ (default:0bd618fe0639, Nov 15 2016, 14:07:07) [GCC 5.4.0 20160609] == Linux-4.4.0-47-generic-x86_64-with-debian-stretch-sid little-endian == hash algorithm: siphash24 64bit == /home/data/src/cpython/3.6/build/test_python_10093 Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0) Run tests sequentially 0:00:00 [1/1] test_dbm test_keys (test.test_dbm.WhichDBTestCase) ... ok test_whichdb (test.test_dbm.WhichDBTestCase) ... ok test_whichdb_ndbm (test.test_dbm.WhichDBTestCase) ... BDB0004 fop_read_meta: @test_10093_tmp_ndbm.db: unexpected file type or format ok test_anydbm_access (test.test_dbm.TestCase-dbm.ndbm) ... ok test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm) ... ERROR BDB3028 @test_10093_tmp.db: unable to flush: No such file or directory test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.ndbm) ... ok test_anydbm_keys (test.test_dbm.TestCase-dbm.ndbm) ... ok test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm) ... ERROR BDB3028 @test_10093_tmp.db: unable to flush: No such file or directory test_anydbm_not_existing (test.test_dbm.TestCase-dbm.ndbm) ... ok test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm) ... ERROR test_error (test.test_dbm.TestCase-dbm.ndbm) ... ok test_anydbm_access (test.test_dbm.TestCase-dbm.dumb) ... ok test_anydbm_creation (test.test_dbm.TestCase-dbm.dumb) ... ok test_anydbm_creation_n_file_exists_with_invalid_contents (test.test_dbm.TestCase-dbm.dumb) ... ok test_anydbm_keys (test.test_dbm.TestCase-dbm.dumb) ... ok test_anydbm_modification (test.test_dbm.TestCase-dbm.dumb) ... ok test_anydbm_not_existing (test.test_dbm.TestCase-dbm.dumb) ... ok test_anydbm_read (test.test_dbm.TestCase-dbm.dumb) ... ok test_error (test.test_dbm.TestCase-dbm.dumb) ... ok

====================================================================== ERROR: test_anydbm_creation (test.test_dbm.TestCase-dbm.ndbm)

Traceback (most recent call last): File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 73, in test_anydbm_creation self.read_helper(f) File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 114, in read_helper self.assertEqual(self._dict[key], f[key.encode("ascii")]) KeyError: b'0'

====================================================================== ERROR: test_anydbm_modification (test.test_dbm.TestCase-dbm.ndbm)

Traceback (most recent call last): File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 88, in test_anydbm_modification self.read_helper(f) File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 114, in read_helper self.assertEqual(self._dict[key], f[key.encode("ascii")]) KeyError: b'0'

====================================================================== ERROR: test_anydbm_read (test.test_dbm.TestCase-dbm.ndbm)

Traceback (most recent call last): File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 94, in test_anydbm_read self.read_helper(f) File "/home/data/src/cpython/3.6/Lib/test/test_dbm.py", line 114, in read_helper self.assertEqual(self._dict[key], f[key.encode("ascii")]) KeyError: b'0'

Ran 19 tests in 0.052s

FAILED (errors=3) test test_dbm failed test_dbm failed

1 test failed: test_dbm

Total duration: 77 ms Tests result: FAILURE

As suggested in http://bugs.python.org/issue14120, I installed libgdbm-dev, re-configured, and re-compiled. That fixes the problem.

IMHO that's not good enough: if we're missing a dependency, then either configuring or building should fail. It's nice that the test failure is now rock-solid reliable rather than intermittent, but it's still a test failure due to missing dependency. Yuck.

Is the problem something like a missing C function prototype? Maybe you see compiler warnings, but the compiler and linker carry on with the wrong prototype. If you build with “make -s”, warnings might be easier to see.

If my guess is right, this would be similar to Issue 27659, where a module half builds with warnings about a missing crypt() function prototype, although it later fails when linking. Maybe more configure or setup.py checks? (I’m not a fan of configure, but it often seems the easiest short-term solution.)