bug#15578: Parameter -d or --direct to open files with flag O_DIRECT? (original) (raw)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

From: Kyle Sallee
Subject: bug#15578: Parameter -d or --direct to open files with flag O_DIRECT?
Date: Sat, 12 Oct 2013 13:02:53 -0700

On Thu, Oct 10, 2013 at 4:21 PM, Pádraig Brady <address@hidden> wrote:

One possibility worth mentioning, would be to add a files0-from=F option to dd, like du,sort,wc already have.

Wonderful compromise. I agree. I appreciate how it already continues an established and understood convention in parameters for coreutils utilities rather than creating something new.

Oddly enough I might have never used the -files0-from= parameter for du sort wc However, that might sometimes save on an invocation of cat within a pipe. But I am not often working with null terminated lists of files. Merely creating that list might sometimes require an invocation of "tr" to translate the linefeed to null. However, I am aware that "find" can generate lists of null terminated output.

Now those have it because they need to operate on the complete input set, for accumulation or sorting, and thus can't resort to separated runs with xargs or whatever.

That makes sense.

dd might use it as it has a very different command syntax to the standard tools. So that would allow a general method to efficiently read many files.

Since dd might be the only utility from coreutils that allows specification of additional open flags then dd would have been my second choice for modification after cat.

Another related thing to consider is the above would allow a single process to handle everything, but it might be better to split the load into a process per CPU.

The type of processing that dd does to transliterate character sets and do conversion can be processor intensive when the amount of data being processed is immense. Therefore, the benefit of concurrent threaded processing of blocks while outputting the processed blocks linearly could allow dd to expedite processing in proportion to the amount of processors allowed. However, I anticipate problems could occur when processing around block edges. Yet individual processing of files concurrently avoids that potential problem.

Perhaps concurrent processing of files with dd would most often be accomplished with xargs? That is because the output might not be desired as a single data block. Therefore, if dd allowed specification of multiple input files then it should probably also allow the specification of a list of an equal amount of output files?

The cost for invoking dd through xargs to gain concurrent processing of individual files is low in comparison for the boon gained from using two or more processors concurrently. Therefore, the added complexity might not be worth the additional speed when nearly the same benefit can be gained by invoking dd by xargs -P

Aside from the rejected modification to cat, and the proposed modification to dd, and writing a small C helper program; I wonder what possibilities exist that have not yet been contemplated?

Perhaps a BASH plugin can be created that reads a list of file names on stdin and a list of open flags as parameters and then opens those files and writes their output to stdout? Chet might accept a patch for a such a plugin. Because BASH does not install plugins by default and lacks an inter-POSIX suggested installation location; the availability and use of the plugins go unnoticed by almost everyone who lacks specialization in shell scripting. Or more succinctly stated few people might possess an opinion concerning BASH plugins. And if the plugin is rejected then I can still regress to the idea of writing a standalone C program for the task.

Although my original suggestion might have seemed otherwise, I am not in favor of rapid mutation of coreutils as opposed to carefully considered evolution. I appreciate exploring different possibilities for implementing functionality and carefully weighing the boons and the banes to arrive at a conclusion that becomes best for the long term goals of coreutils.

I am 100% satisfied with coreutils. I wanted an eloquent solution for a spliter of an idea. I arrogantly and hastily expected that coreutils should immediately accommodate. Please forgive my lack of objectivity, insistence, and persistence.

Please pardon the delayed response. I wanted to avoid a rapid rate of communication becoming a burden or a cause for irritation. And I hoped that a day's worth of other activities would grant opportunity for a fresh perspective upon resuming consideration.

thanks, Pádraig.

Thanks for the continued attention. I am satisfied with the outcome.

From the conversation I gained a better understanding of how to accomplish the task. The outcome is appreciated.

I previously expected that "cat" --direct would suffice. Yet now I also want to mitigate the necessity of invoking "cat" using "xargs." I should have realized that before sending the first email.

coreutils should be expected to do what coreutils does, but not expected to be the most expeditious implementation. I wanted unreasonable performance.

Please forgive that I created a bother and selfishly utilized other Pádraig Brady's time instead of thinking for myself. That aside I should not omit saying; thanks for maintaining and sharing coreutils. Shell scripting would be insufferable without coreutils. :)