Issue 29659: Expose the length arg from shutil.copyfileobj for public use (original) (raw)

Created on 2017-02-27 00:57 by goodboy, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 328 closed goodboy,2017-02-27 01:19
Messages (5)
msg288616 - (view) Author: Tyler Goodlet (goodboy) * Date: 2017-02-27 00:57
It would be handy to be able to pass the `length` default argument defined by `shutil.copyfileobj()` to other public functions througout the module. When copying very large files (1GB +), increasing the memory buffer can divide the copy time in half and currently the only way to accomplish this is by overriding the needed function(s) which call `shutil.copyfileobj()`. I propose a simple non-invasive change where the following functions will also expose the `length` kwarg and pass it downwards to `copyfileobj`: - copyfile - copy - copy2
msg288962 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-03-04 05:26
This seems reasonable to me.
msg288974 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2017-03-04 09:06
How about increasing default value to 32KiB or 64KiB too? binutils's cp works well for most cases, while it doesn't have option to specify blocksize. I want Python's copy functions works nice enough for common cases too. binutils cp uses 128KiB block for normal file. see more information in https://github.com/coreutils/coreutils/blob/master/src/ioblksize.h But copyfileobj can be used other file-likes. On Linux, typical socket buffer size is 128KiB, and typical pipe buffer size is 64KiB. So block size larger than 64KiB may cause unneeded blocking. That's why I suggested 32KiB or 64KiB.
msg305625 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-06 10:11
Could you please provide any benchmarks Tyler?
msg336504 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2019-02-25 09:27
bpo-33671 fixed this.
History
Date User Action Args
2022-04-11 14:58:43 admin set github: 73845
2019-02-25 09:27:42 methane set status: open -> closedversions: + Python 3.8, - Python 3.7messages: + resolution: fixedstage: test needed -> resolved
2017-11-06 10:11:21 serhiy.storchaka set nosy: + serhiy.storchakamessages: +
2017-03-04 09:06:50 methane set nosy: + methanemessages: +
2017-03-04 05:26:00 rhettinger set nosy: + rhettingermessages: +
2017-03-03 21:55:28 terry.reedy set nosy: + terry.reedytype: enhancementstage: test needed
2017-02-27 19:16:23 Mariatta set components: + Library (Lib)
2017-02-27 01:19:32 goodboy set pull_requests: + <pull%5Frequest288>
2017-02-27 00:57:24 goodboy create