process_message needs to have access to the channel state since it needs to make decisions based on the authentication or transport associated with the channel. It should be either the first or the last arg. I can provide a patch for this. Should backwards compatibility be achieved by using the inspect module to check the signature of process_message and call it accordingly?
I think this is reasonable. A patch would be welcome. You could use inspect. When I had to do something similar I just did the call inside a try/except, caught TypeError and retried without the extra argument. See the __init__ of email.feedparser.FeedParser. I don't know which approach is better; although, with the new signature support in 3.4 perhaps inspecting the signature is better. Another approach would be to designate a new method name for the new signature, and use hasattr to decide which to call. That's actually a bit more consistent with the way the SMTPChannel works.
At this point (3.5) we have settled on having process_message be required to have a **kwargs parameter in its signature in order to support new features (ex: enable_SMTPUTF8). So the solution here would be to use inspect.signature to find out if process_message has a **kwargs parameter (kind=VAR_KEYWORD), and if so to supply a 'channel' keyword and value.
I'm going to close this as won't fix since smtpd.py is deprecated, and there's little chance that folks are still interested in working on it. See aiosmtpd as a much better third party replacement.
