Many applications would benefit from 'connectionless' queues, i.e. they don't want to care whether anything is reading from the other end. Using current queue module classes this is not practical, because there is a choice between unbounded memory consumption or blocking. I propose adding a 'LossyQueue' class in the queue module which would allow bounded memory consumption without blocking on put. (i.e. items are dropped in fifo manner beyond a certain limit). In my view this is at least as natural as the PriorityQueue and LifoQueue extensions in that module. Outline as follows: class LossyQueue(Queue): "Queue subclass which drops items on overflow" def _init(self, maxsize): if maxsize > 0: # build the deque with maxsize limit self.queue = deque(maxlen=maxsize) else: # same as normal Queue instance self.queue = collections.deque() # deque alone handles maxsize, # so we pretend we have none self.maxsize = 0 if there is interest in this I will offer a proper patch with docs and tests.
I'm curious about your "many applications would benefit from 'connectionless' queues". What do you have in mind? Is there any reason those apps cannot use collections.deque() directly?
'connectionless' is from how I see it as an analogy with UDP (vs TCP); why not just use a deque is primarily about having the same API - a client (getter) of the queue shouldn't know or care whether it is a 'lossy' queue or a normal queue. I guess most uses of a normal queue (excepting the 'task' functions) could just use a deque, but it wouldn't feel natural. Use cases: non-critical event/status reporting is my canonical example. Specific examples: - a program which executes a long running process in a thread. It wants to update a GUI progress bar or similar, which must occur in a different thread because of the GUI model. By using a LossyQueue, the server thread is simplified; it doesn't have to care whether anything is listening on the other end, allowing greater decoupling (e.g. no changes required if there isn't a GUI). LossyQueues become part of the interface which can be used or not as required. - emulating/providing wrapper around UDP sockets - many application protocols support a get/set/report type interface with the addition of asynchronous events (e.g. SNMP, Netconf, SCPI). In these type of applications a suitable abstraction might be a normal Queue(s) for the standard commands and a LossyQueue for the events (which some applications might not care about). The point is that to the user of this abstraction, these two interfaces look the same. The 'server' doesn't care if a client is listening or not (it won't block and it won't use unlimited memory) The 'client' (if it wants to use it) doesn't know that it isn't a normal queue (same API). -> decouples server and client tasks.
I think it would be better to put this in the ASPN recipes cookbook to let it mature and gather a following. Right now, it is not at all clear that this is the right thing to do.
History
Date
User
Action
Args
2022-04-11 14:56:54
admin
set
github: 51586
2010-04-03 03:17:50
rhettinger
set
status: open -> closedresolution: rejectedmessages: + versions: - Python 2.7
