child_process: change the defaults maxBuffer size by koh110 · Pull Request #27179 · nodejs/node (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation21 Commits1 Checks0 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

I changed the defaults for all the maxBuffer sizes to a much larger value.

Refs: #23027

Requested in #23027 (review) by @mcollina

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

Strikes me as an enormously large value, but I don't have a good feel for what value is reasonable. This could lead to sharply increased memory usage for some use cases... but then, if people care, the maxBuffer option should be used to set limits appropriate to the use.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #9829 and #11196, notably #9829 (comment) - it's been discussed before.

Changing the default isn't completely out of the question but since it's intended as a containment mechanism for runaway processes, increasing it 320x from 200k to 64M seems like a tremendously bad idea.

Changing the default isn't completely out of the question but since it's intended as a containment mechanism for runaway processes, increasing it 320x from 200k to 64M seems like a tremendously bad idea.

We had Infinity before. This is smaller. From a user point of view, if we are executing a binary and it suddenly got output truncated because of a Node.js update, it would be a bad user experience.

I doubt that citgm would find these things, its likely going to only show up at load (not that more testing is bad).

But I think that exec and spawnSync should be set same value.

Definitely.

I like the 1 meg suggestion (#9829 (comment)).

Its really hard to get these numbers right, I'd also be OK with no limit, or a child_process.DEFAULT_MAX_BUFFER that was writable.

To follow on, 1meg is only 5 times larger, which seems to deal with output that is larger than a normal hand-created text file, more log-file size. Bigger enough it may cover some more use cases, but not so big its useless as protection against a run-away process, and not so controversial as removing all limits, and letting the only limit be node's ability to allocate memory (aka "infininite").

I like this suggestion too. I think It is suitable.

I like the 1 meg suggestion

@bnoordhuis How about this idea?

maxBuffer should not be setted Infinity for run-away process. But I think it is necessary to set maxBuffer to a higher value, because changing value of spawnSync give big impact.

1M seems like a good middle ground.

BridgeAR added the semver-major

PRs that contain breaking changes and should be released in the next major version.

label

Apr 16, 2019

Changing a default should likely be semver-major.

Changing a default should likely be semver-major.

I'm on the fence about that. Its increasing the maxBuffer, is it a breaking change that some apps will stop failing? But then, it will use more memory. I guess there is no hurry to land in 12.x, it'll be in 13.x in a month.

I think it should be landed in 12.x. Because, it will have an impact on developers who are using over 200 * 1024.
I think that we should decrease impact with this version up.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Semver-major seems appropriate.

@BethGriggs @nodejs/tsc Should this land in 12.x? Its not a complex change, so I think its low risk of instability, even though it is late in 12.x rc cycle.

@mcollina @nodejs/tsc: this requires 2 TSC approvals, it currently has 0

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

sam-github pushed a commit that referenced this pull request

Apr 17, 2019

PR-URL: #27179 Refs: #23027 Reviewed-By: Ben Noordhuis info@bnoordhuis.nl Reviewed-By: Sam Roberts vieuxtech@gmail.com Reviewed-By: Ruben Bridgewater ruben@bridgewater.de Reviewed-By: Michaël Zasso targos@protonmail.com Reviewed-By: Beth Griggs Bethany.Griggs@uk.ibm.com Reviewed-By: Michael Dawson michael_dawson@ca.ibm.com

BethGriggs added a commit that referenced this pull request

Apr 22, 2019

Notable changes:

• assert:
  • improve performance to instantiate errors (Ruben Bridgewater) #26738
  • validate required arguments (Ruben Bridgewater) #26641
  • adjust loose assertions (Ruben Bridgewater) #25008
• async_hooks:
  • remove deprecated emitBefore and emitAfter (Matteo Collina) #26530
  • remove promise object from resource (Andreas Madsen) #23443
• bootstrap
  • make Buffer and process non-enumerable (Ruben Bridgewater) #24874
• buffer:
  • use stricter range checks (Ruben Bridgewater) #27045
  • harden SlowBuffer creation (ZYSzys) #26272
  • harden validation of buffer allocation size (ZYSzys) #26162
  • do proper error propagation in addon methods (Anna Henningsen) #23939
• child_process:
  • change the defaults maxBuffer size (kohta ito) #27179
  • harden fork arguments validation (ZYSzys) #27039
  • use non-infinite maxBuffer defaults (kohta ito) #23027
• console:
  • don't use ANSI escape codes when TERM=dumb (Vladislav Kaminsky) #26261
• crypto:
  • remove legacy native handles (Tobias Nießen) #27011
  • decode missing passphrase errors (Tobias Nießen) #25208
  • move DEP0113 to End-of-Life (Tobias Nießen) #26249
  • remove deprecated crypto._toBuf (Tobias Nießen) #25338
  • set DEFAULT\_ENCODING property to non-enumerable (Antoine du Hamel) #23222
• deps:
  • silence irrelevant V8 warning (Michaël Zasso) #26685
  • update postmortem metadata generation script (cjihrig) #26685
  • V8: un-cherry-pick bd019bd (Refael Ackermann) #26685
  • V8: cherry-pick 6 commits (Michaël Zasso) #26685
  • V8: cherry-pick d82c9af (Anna Henningsen) #26685
  • V8: cherry-pick e5f01ba (Anna Henningsen) #26685
  • V8: cherry-pick d5f08e4 (Anna Henningsen) #26685
  • V8: cherry-pick 6b09d21 (Anna Henningsen) #26685
  • V8: cherry-pick f0bb5d2 (Anna Henningsen) #26685
  • V8: cherry-pick 5b0510d (Anna Henningsen) #26685
  • V8: cherry-pick 91f0cd0 (Anna Henningsen) #26685
  • V8: cherry-pick 392316d (Anna Henningsen) #26685
  • V8: cherry-pick 2f79d68 (Anna Henningsen) #26685
  • sync V8 gypfiles with 7.4 (Ujjwal Sharma) #26685
  • update V8 to 7.4.288.13 (Ujjwal Sharma) #26685
  • bump minimum icu version to 63 (Ujjwal Sharma) #25852
  • silence irrelevant V8 warnings (Michaël Zasso) #25852
  • V8: cherry-pick 7803fa6 (Jon Kunkee) #25852
  • V8: cherry-pick 58cefed (Jon Kunkee) #25852
  • V8: cherry-pick d3308d0 (Michaël Zasso) #25852
  • V8: cherry-pick 74571c8 (Michaël Zasso) #25852
  • cherry-pick fc0ddf5 from upstream V8 (Anna Henningsen) #25852
  • sync V8 gypfiles with 7.3 (Ujjwal Sharma) #25852
  • sync V8 gypfiles with 7.2 (Michaël Zasso) #25852
  • update V8 to 7.3.492.25 (Michaël Zasso) #25852
  • add s390 asm rules for OpenSSL-1.1.1 (Shigeki Ohtsu) #19794
  • sync V8 gypfiles with 7.1 (Refael Ackermann) #23423
  • update V8 to 7.1.302.28 (Michaël Zasso) #23423
• doc:
  • update behaviour of fs.writeFile (Sakthipriyan Vairamani (thefourtheye)) #25080
  • add internal functionality details of util.inherits (Ruben Bridgewater) #24755
• errors:
  • update error name (Ruben Bridgewater) #26738
• fs:
  • use proper .destroy() implementation for SyncWriteStream (Matteo Collina) #26690
  • improve mode validation (Ruben Bridgewater) #26575
  • harden validation of start option in createWriteStream (ZYSzys) #25579
  • make writeFile consistent with readFile wrt fd (Sakthipriyan Vairamani (thefourtheye)) #23709
• http:
  • validate timeout in ClientRequest() (cjihrig) #26214
  • return HTTP 431 on HPE_HEADER_OVERFLOW error (Albert Still) #25605
  • switch default parser to llhttp (Anna Henningsen) #24870
  • change DEP0066 to a runtime deprecation (Morgan Roderick) #24167
  • else case is not reachable (szabolcsit) #24176
• lib:
  • move DEP0021 to end of life (cjihrig) #27127
  • remove Atomics.wake (Gus Caplan) #27033
  • validate Error.captureStackTrace() calls (Ruben Bridgewater) #26738
  • refactor Error.captureStackTrace() usage (Ruben Bridgewater) #26738
  • move DTRACE_* probes out of global scope (James M Snell) #26541
  • deprecate _stream_wrap (Sam Roberts) [#26245] (#26245)
  • don't use util.inspect() internals (Ruben Bridgewater) #24971
  • improve error message for MODULE_NOT_FOUND (Ali Ijaz Sheikh) #25690
  • requireStack property for MODULE_NOT_FOUND (Ali Ijaz Sheikh) #25690
  • move DEP0029 to end of life (cjihrig) #25377
  • move DEP0028 to end of life (cjihrig) #25377
  • move DEP0027 to end of life (cjihrig) #25377
  • move DEP0026 to end of life (cjihrig) #25377
  • move DEP0023 to end of life (cjihrig) #25280
  • move DEP0006 to end of life (cjihrig) #25279
  • remove unintended access to deps/ (Anna Henningsen) #25138
  • move DEP0120 to end of life (cjihrig) #24862
  • use ES6 class inheritance style (Ruben Bridgewater) #24755
  • remove inherits() usage (Ruben Bridgewater) #24755
• module:
  • remove dead code (Ruben Bridgewater) #26983
  • mark DEP0019 as End-of-Life (Ruben Bridgewater) #26973
  • throw an error for invalid package.json main entries (Ruben Bridgewater) #26823
  • don't search in require.resolve.paths (cjihrig) #23683
• n-api:
  • remove code from error name (Ruben Bridgewater) #26738
• net:
  • do not manipulate potential user code (Ruben Bridgewater) #26751
  • emit "write after end" errors in the next tick (Ouyang Yadong) #24457
  • deprecate _setSimultaneousAccepts() undocumented function (James M Snell) #23760
• net,http2:
  • merge setTimeout code (ZYSzys) #25084
• os:
  • implement os.type() using uv_os_uname() (cjihrig) #25659
• process:
  • global.process, global.Buffer getters (Guy Bedford) #26882
  • move DEP0062 (node --debug) to end-of-life (Joyee Cheung) #25828
  • exit on --debug and --debug-brk after option parsing (Joyee Cheung) #25828
  • improve --redirect-warnings handling (Ruben Bridgewater) #24965
• readline:
  • support TERM=dumb (Vladislav Kaminsky) #26261
• repl:
  • add welcome message (gengjiawen) #25947
  • fix terminal default setting (Ruben Bridgewater) #26518
  • check colors with .getColorDepth() (Vladislav Kaminsky) #26261
  • deprecate REPLServer.rli (Ruben Bridgewater) #26260
• src:
  • remove unused INT_MAX constant (Sam Roberts) #27078
  • update NODE_MODULE_VERSION to 72 (Ujjwal Sharma) #26685
  • remove AddPromiseHook() (Anna Henningsen) #26574
  • update NODE_MODULE_VERSION to 71 (Michaël Zasso) #25852
  • clean up MultiIsolatePlatform interface (Anna Henningsen) #26384
  • properly configure default heap limits (Ali Ijaz Sheikh) #25576
  • remove icuDataDir from node config (GauthamBanasandra) #24780
  • explicitly allow JS in ReadHostObject (Yang Guo) #23423
  • update postmortem constant (cjihrig) #23423
  • update NODE_MODULE_VERSION to 68 (Michaël Zasso) #23423
• tls:
  • support TLSv1.3 (Sam Roberts) #26209
  • return correct version from getCipher() (Sam Roberts) #26625
  • check arg types of renegotiate() (Sam Roberts) #25876
  • add code for ERR_TLS_INVALID_PROTOCOL_METHOD (Sam Roberts) #24729
  • emit a warning when servername is an IP address (Rodger Combs) #23329
  • disable TLS v1.0 and v1.1 by default (Ben Noordhuis) #23814
  • remove unused arg to createSecureContext() (Sam Roberts) #24241
  • deprecate Server.prototype.setOptions() (cjihrig) #23820
  • load NODE_EXTRA_CA_CERTS at startup (Ouyang Yadong) #23354
• util:
  • change inspect compact and breakLength default (Ruben Bridgewater) #27109
  • improve inspect edge cases (Ruben Bridgewater) #27109
  • only the first line of the error message (Simon Zünd) #26685
  • don't set the prototype of callbackified functions (Ruben Bridgewater) #26893
  • rename callbackified function (Ruben Bridgewater) #26893
  • increase function length when using callbackify() (Ruben Bridgewater) #26893
  • prevent tampering with internals in inspect() (Ruben Bridgewater) #26577
  • fix proxy inspection (Ruben Bridgewater) #26241
  • prevent leaking internal properties (Ruben Bridgewater) #24971
  • protect against monkeypatched Object prototype for inspect() (Rich Trott) #25953
  • treat format arguments equally (Roman Reiss) #23162
• win, fs:
  • detect if symlink target is a directory (Bartosz Sosnowski) #23724
• zlib:
  • throw TypeError if callback is missing (Anna Henningsen) #24929
  • make “bare” constants un-enumerable (Anna Henningsen) #24824

PR-URL: #26930

This was referenced

Apr 23, 2019

Reviewers

sam-github sam-github approved these changes

bnoordhuis bnoordhuis approved these changes

targos targos approved these changes

BethGriggs BethGriggs approved these changes

BridgeAR BridgeAR approved these changes

mhdawson mhdawson approved these changes

mcollina Awaiting requested review from mcollina

Labels

child_process

Issues and PRs related to the child_process subsystem.

semver-major

PRs that contain breaking changes and should be released in the next major version.