Issue 4774: threding, bsddb and double free or corruption (fasttop) (original) (raw)

This is a test script, to help me to understand why I have unexpected result in application. But I got a more unexpected result, and probably wrong error message about the read-only cursor. The full script is at the end.

I got this only one ! I restart the script after the crash, but this has freezed my xen virtual machine after some minutes. I had to "destroy" it and restart it. After restart was not working anymore, blocked, not responding except to CTRL-C I had to drop the old DB, and then i worked again

I run python 2.5.2 inside a kolab 2.2.0 (aka openpkg) environement on a xen centos 5.x running kernel 2.6.18-8.el5xen and glibc-2.5-12

The host is a P4 running centos 5.X and kernel 2.6.18-53.1.4.el5xen

Here are the errors, I waited for long time after the firts backtrace, to press CTRL-C and got immediately the second backtrace.

Regards

db cleanup, 326 records deleted, 9990 remains Exception in thread Thread-1: Traceback (most recent call last): File "/kolab/lib/python/threading.py", line 486, in __bootstrap_inner self.run() File "/kolab/lib/python/threading.py", line 446, in run self.__target(*self.__args, **self.__kwargs) File "./test/test_bdb.py", line 85, in update_thread=threading.Thread(target=lambda : cleanup(db)) File "./test/test_bdb.py", line 72, in cleanup cursor.delete() DBPermissionsError: (1, 'Operation not permitted -- Write attempted on read- only cursor')

Traceback (most recent call last): File "./test/test_bdb.py", line 130, in server() File "./test/test_bdb.py", line 92, in server db[str(last)]=str(last) KeyboardInterrupt *** glibc detected *** /kolab/bin/python: double free or corruption (fasttop): 0 x09d521a8 *** ======= Backtrace: ========= /lib/i686/nosegneg/libc.so.6[0x4444ccfd] /lib/i686/nosegneg/libc.so.6(cfree+0x90)[0x444503b0] /kolab/bin/python(__db_c_destroy+0x60)[0x81fa6f0] /kolab/bin/python(__db_refresh+0x165)[0x81f3515] /kolab/bin/python(__db_close+0x64)[0x81f3cb4] /kolab/bin/python(__db_close_pp+0xa1)[0x8204b61] /kolab/bin/python[0x812123f] /kolab/bin/python[0x8159534] /kolab/bin/python[0x815fb0a] /kolab/bin/python[0x811abbc] /kolab/bin/python[0x811abc9] /kolab/bin/python[0x80b0229] /kolab/bin/python(PyDict_SetItem+0x6e)[0x80b1c0e] /kolab/bin/python(PyDict_SetItemString+0x42)[0x80b1ce2] /kolab/bin/python(PyImport_Cleanup+0xd4)[0x8108294] /kolab/bin/python(Py_Finalize+0xbf)[0x8113f1f] /kolab/bin/python(Py_Main+0x468)[0x80845c8] /kolab/bin/python(main+0x22)[0x8084052] /lib/i686/nosegneg/libc.so.6(__libc_start_main+0xdc)[0x443fbdec] /kolab/bin/python[0x8083fa1] ======= Memory map: ======== 00110000-001d7000 rwxp 00110000 00:00 0 001d7000-001da000 r-xp 00000000 ca:03 973941 /kolab/lib/python/lib- dynload/ t ime.so 001da000-001dc000 rwxp 00002000 ca:03 973941 /kolab/lib/python/lib- dynload/ t ime.so 001dc000-001e0000 r-xp 00000000 ca:03 973938 /kolab/lib/python/lib- dynload/ s trop.so 001e0000-001e2000 rwxp 00003000 ca:03 973938 /kolab/lib/python/lib- dynload/ s trop.so 001e2000-001e3000 r-xp 00000000 ca:03 973911 /kolab/lib/python/lib- dynload/ _ weakref.so 001e3000-001e4000 rwxp 00000000 ca:03 973911 /kolab/lib/python/lib- dynload/ _ weakref.so 001e4000-001ea000 rwxs 00000000 ca:03 1747650 /tmp/db/__db.001 001ea000-001fc000 rwxs 00000000 ca:03 1747651 /tmp/db/__db.002 001fc000-0023e000 rwxs 00000000 ca:03 1747652 /tmp/db/__db.003 00388000-003e0000 rwxs 00000000 ca:03 1747653 /tmp/db/__db.004 00450000-00452000 r-xp 00000000 ca:03 1132355 /lib/libutil-2.5.so 00452000-00453000 r-xp 00001000 ca:03 1132355 /lib/libutil-2.5.so 00453000-00454000 rwxp 00002000 ca:03 1132355 /lib/libutil-2.5.so 00459000-0047e000 r-xp 00000000 ca:03 1132349 /lib/i686/nosegneg/ libm-2.5.so 0047e000-0047f000 r-xp 00024000 ca:03 1132349 /lib/i686/nosegneg/ libm-2.5.so 0047f000-00480000 rwxp 00025000 ca:03 1132349 /lib/i686/nosegneg/ libm-2.5.so 0063e000-0063f000 r-xp 0063e000 00:00 0 [vdso] 0097a000-0098d000 r-xp 00000000 ca:03 1134447 /lib/i686/nosegneg/ libpthread-2
. 5.so 0098d000-0098e000 r-xp 00012000 ca:03 1134447 /lib/i686/nosegneg/ libpthread-2
. 5.so 0098e000-0098f000 rwxp 00013000 ca:03 1134447 /lib/i686/nosegneg/ libpthread-2
. 5.so 0098f000-00991000 rwxp 0098f000 00:00 0 009a8000-009e9000 rwxp 009a8000 00:00 0 00b80000-00d80000 r-xp 00000000 ca:03 845325 /usr/lib/locale/ locale-archive 00ec9000-00ecc000 r-xp 00000000 ca:03 973916 /kolab/lib/python/lib- dynload/ c StringIO.so 00ecc000-00ecd000 rwxp 00003000 ca:03 973916 /kolab/lib/python/lib- dynload/ c StringIO.so 00fd2000-00fd6000 r-xp 00000000 ca:03 973918 /kolab/lib/python/lib- dynload/ c ollections.so 00fd6000-00fd7000 rwxp 00004000 ca:03 973918 /kolab/lib/python/lib- dynload/ c ollections.so 00fd7000-00fd8000 --xp 00fd7000 00:00 0 00fd8000-019d8000 rwxp 00fd8000 00:00 0 08048000-08433000 r-xp 00000000 ca:03 430731 /kolab/bin/python 08433000-0846e000 rwxp 003eb000 ca:03 430731 /kolab/bin/python 0846e000-08478000 rwxp 0846e000 00:00 0 09ce5000-09da5000 rwxp 09ce5000 00:00 0 443c4000-443dd000 r-xp 00000000 ca:03 1132323 /lib/ld-2.5.so 443dd000-443de000 r-xp 00018000 ca:03 1132323 /lib/ld-2.5.so 443de000-443df000 rwxp 00019000 ca:03 1132323 /lib/ld-2.5.so 443e6000-44521000 r-xp 00000000 ca:03 1134399 /lib/i686/nosegneg/ libc-2.5.so 44521000-44523000 r-xp 0013a000 ca:03 1134399 /lib/i686/nosegneg/ libc-2.5.so 44523000-44524000 rwxp 0013c000 ca:03 1134399 /lib/i686/nosegneg/ libc-2.5.so 44524000-44527000 rwxp 44524000 00:00 0 44529000-4452b000 r-xp 00000000 ca:03 1134402 /lib/libdl-2.5.so 4452b000-4452c000 r-xp 00001000 ca:03 1134402 /lib/libdl-2.5.so 4452c000-4452d000 rwxp 00002000 ca:03 1134402 /lib/libdl-2.5.so 44622000-4462d000 r-xp 00000000 ca:03 1134413 /lib/ libgcc_s-4.1.1-20070105.so

1Aborted

#!/kolab/bin/python

import os, sys, logging, time, threading import marshal import bsddb.db

this is a test to see how bsddb is reliable

The server count in the db, and the client search for missing value

run on server in a terminal, and a client in another

====================================================================== # db functions #

db_dir='/tmp/db' db_filename=os.path.join(db_dir, 'test.bdb')

try: os.mkdir(db_dir) except OSError: pass

freq=100 # new sample per/sec timeout=3 # in sec, how long I wait for them retention=100 # in sec, how long they stay in the db

def init_db(read_only=True): # open the environement dbenv=db=None try: db_env=bsddb.db.DBEnv(0)

    flags=bsddb.db.DB_CREATE | bsddb.db.DB_INIT_CDB |

bsddb.db.DB_INIT_MPOOL db_env.open(db_dir, flags) db=bsddb.db.DB(db_env) flags = bsddb.db.DB_DIRTY_READ | ( bsddb.db.DB_RDONLY if read_only else bsddb.db.DB_CREATE )

    db.open(db_filename, dbtype=bsddb.db.DB_HASH,flags=flags)
except Exception:
    print 'error opening db'
    raise
else:
    print 'db opened'

return dbenv, db

def close_db(dbenv, db): try: if db: db.close() if dbenv: db_env.close() except Exception: log.rcpt.exception('closing db') else: log.rcpt.info('db closed')

def now(): return int(time.time()*freq)

def cleanup(db):

while True:
    cursor=db.cursor(None, bsddb.db.DB_WRITECURSOR)
    count=remain=0
    c=cursor.first()
    limit=now()-retention*freq
    while c:
        k, v=c
        t=int(k)
        if t<=limit:
            cursor.delete()
            count+=1
        else:
            remain+=1
        c=cursor.next()
    cursor.close()
    print 'db cleanup, %d records deleted, %d remains' % (count,

remain) time.sleep(timeout)

def server(): dbenv, db=init_db(read_only=False)

update_thread=threading.Thread(target=lambda : cleanup(db))
update_thread.setDaemon(True)
update_thread.start()
last=now()
while True:

    while last<now():
        db[str(last)]=str(last)
        last+=1

    time.sleep(1/freq)

close_db(dbenv, db)

def client():

dbenv, db=init_db(read_only=True)

max_delay=0
last=now()

while True:
    try:
        value=db[str(last)]
    except KeyError:
        if last<now()-timeout*freq:
            print 'not found', last, float(now()-last)/freq
            last+=1
        else:
          time.sleep(1/freq)
    else:
        delay=now()-last
        if delay>max_delay:
            max_delay=delay
            print 'delay', max_delay

        last+=1

close_db(dbenv, db)

if sys.argv[1]=='c': client() elif sys.argv[1]=='s': server()

Alain, I can not reproduce this issue in current bsddb code (as running under Python 2.6.1 and Berkeley DB 4.7.25). Since Python 2.5 branch has been marked as "security fixes only", this bug would be an "out of date" one.

Please, confirm that this bug is not present in Python 2.6.1 or, if you need to run Python 2.5, please install an updated pybsddb library and check it.

I will wait your comments for a week or two. Then mark this issue as "out of date", if you don't indicate the contrary.

Thanks for your time.