[Python-Dev] parallelizing (original) (raw)

Matthieu Bec mdcb808 at gmail.com
Wed Sep 13 15:11:29 EDT 2017

Thank you, I'll take your advice.

Regarding your example, I think it gives the illusion to work because sleep() is GIL aware under the hood.

I don't think it works for process() that mainly runs bytecode, because of the GIL.

Sorry if I wrongly thought that was a language level discussion.

Regards,

Matthieu

On 9/13/17 10:49 AM, Chris Barker wrote:

This really isn't the place to ask this kind of question.

If you want to know how to do something with python, try python-users , stack overflow, etc. If you have an idea about a new feature you think python could have, then the python-ideas list is the place for that. But if you want anyone to take it seriously, it should be a better formed idea before you post there. But: On Tue, Sep 12, 2017 at 4:43 PM, Matthieu Bec <mdcb808 at gmail.com_ _<mailto:mdcb808 at gmail.com>> wrote: There are times when you deal with completely independent input/output 'pipes' - where parallelizing would really help speed things up. Can't there be a way to capture that idiom and multi thread it in the language itself? Example: loop: read an XML produce a JSON like

Regular old threading works fine for this: import time import random import threading def process(infile, outfile): "fake function to simulate a process that takes a random amount of time" time.sleep(random.random()) print("processing: {} to make {}".format(infile, outfile)) for i in range(10): threading.Thread(target=process, args=("file%i.xml" % i, "file%i.xml" % i)).start() It gets complicated if you need to pass information back and forth, or worry about race conditions, or manage a queue, or .... But just running a nice self-contained thread safe function in another thread is pretty straightforward. -CHB

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov <mailto:Chris.Barker at noaa.gov>

-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20170913/4c56e26f/attachment.html>

More information about the Python-Dev mailing list