Issue 18153: python imaplib - error 'unexpected response' (original) (raw)

Issue18153

Created on 2013-06-07 00:20 by tahnoon, last changed 2022-04-11 14:57 by admin.

Messages (10)
msg190733 - (view)	Author: tahnoon pasha (tahnoon)	Date: 2013-06-07 00:20
Hi I've suddenly encountered an error using imaplib on some code that worked fine before. import imaplib m = imaplib.IMAP4('myserver','port') m.login(r'username','password') m.select() gives me the error Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.7/imaplib.py", line 649, in select typ, dat = self._simple_command(name, mailbox) File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command return self._command_complete(name, self._command(name, *args)) File "/usr/lib/python2.7/imaplib.py", line 899, in _command_complete raise self.abort('command: %s => %s' % (name, val)) imaplib.abort: command: SELECT => unexpected response: '* 1520 EXISTS' I'm not sure what it means. Emails are otherwise coming through fine, and I'm using davmail as a server. The program in its entirety saves attachments with a certain name in a specific folder. I've stepped through it and its definitely the `m.select()` that is where its falling over. This same program worked absolutely fine until recently. What am I doing wrong, and how do I fix it? The log of activity is as follows >>> import imaplib >>> m = imaplib.IMAP4('server','port') >>> Debug=4 >>> m.debug 0 >>> m.debug=4 >>> m.debug 4 >>> m.login(r'username','password') 01:26.55 > HLFI1 LOGIN "username" "password" 01:30.76 < HLFI1 OK Authenticated ('OK', ['Authenticated']) >>> m.list() 01:56.33 > HLFI2 LIST "" * 02:00.04 < * LIST (\HasNoChildren) "/" "Trash/Sent Messages" 02:00.04 < * LIST (\HasNoChildren) "/" "Sync Issues/Server Failures" 02:00.04 < * LIST (\HasNoChildren) "/" "Sync Issues/Local Failures" 02:00.04 < * LIST (\HasNoChildren) "/" "Sync Issues/Conflicts" 02:00.04 < * LIST (\HasChildren) "/" "Sync Issues" 02:00.04 < * LIST (\HasNoChildren) "/" "Junk E-mail" 02:00.04 < * LIST (\HasNoChildren) "/" "Drafts" 02:00.04 < * LIST (\HasChildren) "/" "Trash" 02:00.04 < * LIST (\HasNoChildren) "/" "Sent" 02:00.04 < * LIST (\HasNoChildren) "/" "Outbox" 02:00.04 < * LIST (\HasNoChildren) "/" "INBOX" 02:00.04 < HLFI2 OK LIST completed ('OK', ['(\\HasNoChildren) "/" "Trash/Sent Messages"', '(\\HasNoChildren) "/" "Sync Issues/Server Failures"', '(\\HasNoChildren) "/" "Sync Issues/Local Failures"', '(\\HasNoChildren) "/" "Sync Issues/Conflicts"', '(\\HasChildren) "/" "Sync Issues"', '(\\HasNoChildren) "/" "Junk E-mail"', '(\\HasNoChildren) "/" "Drafts"', '(\\HasChildren) "/" "Trash"', '(\\HasNoChildren) "/" "Sent"', '(\\HasNoChildren) "/" "Outbox"', '(\\HasNoChildren) "/" "INBOX"']) >>> m.select() 02:21.37 > HLFI3 SELECT INBOX 02:30.87 < * 1548 EXISTS 02:30.87 last 4 IMAP4 interactions: 00:16.73 < * OK [CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE] IMAP4rev1 DavMail 4.3.0-2125 server ready 00:16.73 > HLFI0 CAPABILITY 00:16.74 < * CAPABILITY IMAP4REV1 AUTH=LOGIN MOVE 00:16.77 < HLFI0 OK CAPABILITY completed Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.7/imaplib.py", line 649, in select typ, dat = self._simple_command(name, mailbox) File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command return self._command_complete(name, self._command(name, *args)) File "/usr/lib/python2.7/imaplib.py", line 899, in _command_complete raise self.abort('command: %s => %s' % (name, val)) imaplib.abort: command: SELECT => unexpected response: '* 1548 EXISTS' I understand that this seems to be occurring because of the extra spaces in the final RETURN from a query on stack overflow http://stackoverflow.com/questions/16911238/python-imaplib-error-unexpected-repsonse and it was suggested I file an issue report. I'm using Davmail as the server on Ubuntu 13.04 and the server works fine with Thunderbird, Fetchmail and Evolution My first attempt at filing an issue so apologies if I've done something wrong.
msg190734 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2013-06-07 01:13
I think that technically the server is out of spec with the RFC. It isn't 100% clear, though since while the RFC says extra spaces are invalid, it also says that an untagged response is formed by "prefixing the token '*'", without otherwise mentioning it in the BNF. So one assumes, based on the rest of the document, that a single space follows the '*', and anything else is invalid. Now, the RFC says the server MUST reject malformed commands from the client, but does not speak to what the client should do given malformed responses from the server. So, it might indeed be reasonable to "fix" imaplib to handle an arbitrary number of spaces after the '*'. IMO Davmail should be fixed, though. IMAP is a finicky protocol.
msg191000 - (view)	Author: tahnoon pasha (tahnoon)	Date: 2013-06-12 00:55
I'll log this at the davmail forums too and report back if I get a response there.
msg191161 - (view)	Author: tahnoon pasha (tahnoon)	Date: 2013-06-14 22:06
http://sourceforge.net/p/davmail/bugs/532/ The response back from the davmail software maintainer is that this is a non RFC mechanism with a purpose (used as a keep alive on a very slow responding server - SELECT can be very slow when getting mail this way.) I guess it's a request back to imaplib maintainers to see how this might be fixed in imaplib. I'm very new to python and not a programmer by trade but if its a straightforward fix I'd be happy to have a go if someone can point me in the right direction.
msg191166 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2013-06-14 23:01
Well, the first thing to do would be to write a test that reproduces the problem. There is test infrastructure in Lib/test/test_imaplib.py, but I will admit that writing the server side of imaplib tests is not a walk in the park. You are welcome to take a look. If you can figure it out, you'll be on your way to being a non-newbie Python programmer :)
msg191168 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2013-06-14 23:03
Oh, by the way there's a bit more of the test infrastructure in python3 compared to python2.
msg191170 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2013-06-14 23:16
I'll also note that if you start from the traceback and look at the code involved in the exception (keeping in mind that since the exception is caught, in python2 you lose the original cause, which is therefore in the function called in the try block), there is actually a comment in the code that is directly on point to this issue. If we can get a reproducible test case the fix will probably be simple.
msg191194 - (view)	Author: tahnoon pasha (tahnoon)	Date: 2013-06-15 03:05
Okay David. Thanks for the pointer. With great trepidation Ill hit google and try and figure out how to create a test case for this.
msg191320 - (view)	Author: tahnoon pasha (tahnoon)	Date: 2013-06-17 09:34
Hi David Adding the following post and response from the davmail author/ maintainers site. He seems to have fixed it in davmail and suggests the following fix in imaplib.py if there is a desire to amend it to allow stray spaces Le 15/06/2013 08:19, tahnoon a écrit : Hi Mickael. I've continued to post on https://bugs.python.org and the imaplib maintainer has agreed to look at fixing this on imaplib if I can provide the appropriate unit test to replicate the problem. I'm not really a programmer so any pointers you can give me on how to set up a server simulation that approximates davmail or if you have one already set up as part of your development that would help solve this at the client end. Thanks Well, it's not easy to reproduce without a backend Exchange server. Basically it should accept multiple spaces after star in untagged response. untested fix of imaplib.py: replace Untagged_response = re.compile(r'* (?P[A-Z-]+)( (?P.))?') with Untagged_response = re.compile(r'*[ ]+(?P[A-Z-]+)( (?P.))?') Note that I disabled this non standard behavior in latest DavMail release, you can enable it in DavMail settings (KeepAlive). Not sure if that closes this issue or if you still think a test case is needed?
msg191336 - (view)	Author: R. David Murray (r.david.murray) *	Date: 2013-06-17 14:07
I don't understand why his keepalive requires the extra spaces...and I'm not 100% sure that accepting them will fix the problem, though in theory it should. I still lean toward making the spaces fix in imaplib, based on the postel principle. I'd prefer to have a unit test, but if you or I or someone else doesn't get around to writing one before the next time we start working on releases, I'll probably commit the (pretty much obviously correct) patch without a test.