| msg39518 - (view) |
Author: Gregory H. Ball (greg_ball) |
Date: 2002-04-09 19:15 |
| Attached patch is a first crack at a unit test for whichdb. I think that all functionality required for use by the anydbm module is tested, but only for the database modules found in a given installation. The test case is built up at runtime to cover all the available modules, so it is a bit introspective, but I think it is obvious that it should run correctly. Unfortunately it crashes on my box (Redhat 6.2) and this seems to be a real problem with whichdb: it assumes things about the dbm format which turn out to be wrong sometimes. I only discovered this because test_anydbm was crashing, when whichdb failed to work on dbm files. It would not have crashed if dbhash was available... and dbhash was not available because bsddb was not built correctly. So I think there is a build bug there, but I have little idea how to solve that one at this point. Would I be correct in thinking that if this test really uncovers bugs in whichdb, it can't be checked in until they are fixed? Unfortunately I don't know much about the various databases, but I'll try to work with someone on it. |
|
|
| msg39519 - (view) |
Author: Guido van Rossum (gvanrossum) *  |
Date: 2002-04-15 01:10 |
| Logged In: YES user_id=6380 What kind of crash do you experience? Do you have a patch that fixes whichdb? |
|
|
| msg39520 - (view) |
Author: Gregory H. Ball (greg_ball) |
Date: 2002-04-16 03:41 |
| Logged In: YES user_id=11365 I get two failures... First, using the dbm module as the engine, whichdb fails to indentify the type. This is apparently a platform problem... whichdbm.py has the comment # Check for dbm first -- this has a .pag and a .dir file but on my system the dbm modules creates a .db file. The 'file' utility says duh.db: Berkeley DB 2.X Hash/Little Endian (Version 5, Logical sequence number: file - 0, offset - 0, Bucket Size 4096, Overflow Point 1, Last Freed 0, Max Bucket 1, High Mask 0x1, Low Mask 0x0, Fill Factor 40, Number of Keys 0) Now, a very simple patch would be to look for .db files and call them 'dbm'. I have no idea though whether there might be other database formats which use this extension. So the thing to do might be to look for .db files and test their magic. Actually, the .db files are identified as "dbhash" databases if named explicitly to whichdb... But the dbhash module isn't available due to missing bsddb! I'm not sure what to make of all this. I could just assume .db files with dbhash magic are always of kind dbm... sound reasonable? Secondly, dumbdbm doesn't work either, if the database is empty... f.read(1) in ["'", '"'] doesn't turn out to be true, since the .dir file is empty. Ok, I've attached a naive patch. Note I'm not even looking at testing dbhash or gdbm since they're not built on my system. On the other hand since anydbm tries these first, maybe they are effectively tested by test_anydbm. |
|
|
| msg39521 - (view) |
Author: Gregory H. Ball (greg_ball) |
Date: 2002-04-16 03:49 |
| Logged In: YES user_id=11365 More detail... the failure mode of test_anydbm is that a database freshly created with anydbm.open() can't be reopened using the 'r' mode. Since whichdb returns None we wind up at raise error, "need 'c' or 'n' flag to open new db" Of course, whichdb is to blame for this. |
|
|
| msg39522 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2002-04-16 12:46 |
| Logged In: YES user_id=6380 Greg, you assigned this to Neal Norwitz. Why? Usually bug reports stay unassigned until a developer shows interest. I doubt that this is Neal's kind of bug: he's not commented on the bug report, nor does this match the other bugs that he's interested in. |
|
|
| msg39523 - (view) |
Author: Neal Norwitz (nnorwitz) * |
Date: 2002-04-16 12:52 |
| Logged In: YES user_id=33168 I have reviewed it, but others have stayed on top of this and I didn't have anything to contribute. I will be glad to check the patch in when it is in the proper state. But I don't know much about anydbm. |
|
|
| msg39524 - (view) |
Author: Gregory H. Ball (greg_ball) |
Date: 2002-04-16 16:24 |
| Logged In: YES user_id=11365 Neal posted a list to python-dev of standard library modules without unit tests. (<3CB3093C.B7A22727@metaslash.com>, 1 week ago, subject Re: Stability and change) That prompted me to address the breakage that I was seeing in test_anydbm due to whichdb. So I thought he might be interested. If this interrupts the workflow I'll refrain from making assignments with so little justification... |
|
|
| msg39525 - (view) |
Author: Neal Norwitz (nnorwitz) * |
Date: 2002-04-23 00:01 |
| Logged In: YES user_id=33168 Greg, what is the status of this patch? Do you think it's ready? It's not clear to me if you think there is a bug in whichdb or not (your last paragraph in the report). The current code uses os.extsep, but you use '.' inline. This should probably be kept consistent. I don't understand why you add the check for filename.endswith('.db') and whichdb(filename+'.db') == 'dbhash'. Was this to fix your problem on redhat 6.2? Is it possible to test more cases generically in whichdb? Not directed at Greg, I don't understand why the code does: char in ["'", '"'], rather than char in "\"'". I don't love the code duplication (_fname & _delete_files), but I'm not sure there's any particular way to deal with that, rather than add the code to unittest. |
|
|
| msg39526 - (view) |
Author: Guido van Rossum (gvanrossum) * |
Date: 2002-05-31 14:06 |
| Logged In: YES user_id=6380 Assigning to Skip on a lark, since he seems to be interested in these things. |
|
|
| msg39527 - (view) |
Author: Gregory H. Ball (greg_ball) |
Date: 2002-05-31 14:37 |
| Logged In: YES user_id=11365 Skip, The whichdb patch here is naive. But I think the unit test might be a good starting point for catching the sort of problems you've been seeing. Sorry I didn't respond to Neal's comment, I don't remember getting an email for that one. |
|
|
| msg39528 - (view) |
Author: Skip Montanaro (skip.montanaro) *  |
Date: 2002-06-04 03:35 |
| Logged In: YES user_id=44345 I'm having trouble getting the unit test to run successfully on my system... Barry and I are fiddling with bsddb building. I think everything in this area is related, so hopefully some of what we come up with will improve the chances that this unit test passes. Skip |
|
|
| msg39529 - (view) |
Author: Skip Montanaro (skip.montanaro) * |
Date: 2002-07-09 23:42 |
| Logged In: YES user_id=44345 I've attached a somewhat different patch to whichdb.py that I hope helps the test case pass in the situation Greg described: anydbm uses dbm as the underlying database module and dbmmodule.so was linked with the dbm emulation library available with Berkeley DB. In this situation, dbm.open("foo", "c") will actually create a Berkeley DB Hash file named "foo.db". The patch adds a test for "foo.db" and an attempt to open that file using dbm.open() to the check for "dbmness". I don't think this problem can be completely solved by whichdb.py, since you can always construct a test case using multiple db modules where you'll have a filename collision. This may be the best we can do though. Skip |
|
|
| msg39530 - (view) |
Author: Skip Montanaro (skip.montanaro) * |
Date: 2002-08-02 17:10 |
| Logged In: YES user_id=44345 Checked in v 1.1 |
|
|