JS stdio protocol doesn't handle massive metafile (2179827550 bytes) (original) (raw)

After completing a build of a very large project with metafile enabled, the JS wrapper for esbuild fails with:

Error: Invalid packet
    at ByteBuffer._read (node_modules\esbuild\lib\main.js:162:13)
    at ByteBuffer.read32 (node_modules\esbuild\lib\main.js:171:40)
    at decodePacket (node_modules\esbuild\lib\main.js:123:15)
    at handleIncomingPacket (node_modules\esbuild\lib\main.js:652:18)
    at Socket.readFromStdout (node_modules\esbuild\lib\main.js:582:7)
    at Socket.emit (node:events:518:28)
    at addChunk (node:internal/streams/readable:561:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:512:3)
    at Readable.push (node:internal/streams/readable:392:5)
    at Pipe.onStreamRead (node:internal/stream_base_commons:189:23)

I have tracked this down to the readUInt32LE implementation:

export function readUInt32LE(buffer: Uint8Array, offset: number): number {
return buffer[offset++] \|
(buffer[offset++] << 8) \|
(buffer[offset++] << 16) \|
(buffer[offset++] << 24)
}

Because (most) JS bitwise operators cast the operands down to signed 32 bit integers, this actually behaves like "readInt32LE" and ends up with a negative number when read as a length.

Potential fix:

export function readUInt32LE(buffer: Uint8Array, offset: number): number { return buffer[offset++] + (buffer[offset++] * 28) + (buffer[offset++] * 216) + (buffer[offset++] * 2**24) }

I'm not sure what consequences that would have elsewhere readUInt32LE is used, but it works for me locally.

Apologies I don't have a reproduction to share; it would need to be a project with a lot of inputs and outputs that explodes the size of the metafile.