#25388 - ls-quotes: kills existing scripts reading "ls" -1 as input (original) (raw)

Reported by: L A Walsh <coreutils tlinx.org>

Date: Sun, 8 Jan 2017 03:53:01 UTC

Severity: normal

Done: Assaf Gordon <assafgordon gmail.com>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 25388 in the body.
You can then email your comments to 25388 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Report forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Sun, 08 Jan 2017 03:53:01 GMT) Full text and rfc822 format available.

Acknowledgement sentto L A Walsh <coreutils <at> tlinx.org>:
New bug report received and forwarded. Copy sent to bug-coreutils <at> gnu.org. (Sun, 08 Jan 2017 03:53:01 GMT) Full text and rfc822 format available.

Message #5 received at submit debbugs.gnu.org (full text, mbox):

The new ls doesn't maintain backwards compatibility. The default options now add extraneous quotes to some filenames.

Anyplace one uses 'ls -1' to read 1 file/line now breaks.

This is a regression as it breaks existing scripts and behavior.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Sun, 08 Jan 2017 09:03:02 GMT) Full text and rfc822 format available.

Message #8 received at 25388 debbugs.gnu.org (full text, mbox):

On Jan 06 2017, L A Walsh <coreutils tlinx.org> wrote:

The new ls doesn't maintain backwards compatibility. The default options now add extraneous quotes to some filenames.

Anyplace one uses 'ls -1' to read 1 file/line now breaks.

This is a regression as it breaks existing scripts and behavior.

Does it?

$ touch 'a b' $ ls -1 | grep "'"

Andreas.

-- Andreas Schwab, schwab linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different."

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 05:22:01 GMT) Full text and rfc822 format available.

Message #11 received at 25388 debbugs.gnu.org (full text, mbox):

Andreas Schwab wrote: > On Jan 06 2017, L A Walsh <coreutils tlinx.org> wrote: >> The new ls doesn't maintain backwards compatibility. >> The default options now add extraneous quotes to >> some filenames.

Does it?

$ touch 'a b' $ ls -1 | grep "'"

Sorry, you are right.

The problem is 'ls' doesn't show the same

information on the screen when it is passed through a pager like "less" or "more".

In all but the smallest of directories, it's virtually

a requirement to page directory output using 'more' or 'less'.

Having 'ls' show different output through a pager

than it does when scrolling off the screen is not a standard feature. The user has no idea how to page through 'ls's output as they saw it scroll by on the screen. It's inconsistent with any other tools that work with pagers as well as being inconsistent with itself in how it handles COLOR options.

'ls' should default to its regular behavior and

only engage quoting when asked to.

Let distros and users decide what to enable

to add new behaviors -- like the various color options. My distro added color by default in their defaults w/it only on when output is not a pipe.

I changed it on my machine to put out color

even if through a pipe as I usually am piping it through 'less' and I want to see the color. If I want no color -- I can get the normal text by adding a backslash before the command.

The new quoting also messes up copy/pasting into

any place other than the shell. I tend to copy/paste between a shell and gvim when doing script development. I wouldn't mind having such a switch as an option, but it is usually the case that I can copy/paste between windows and have it pick up whether or not a tab or a space was used. Having tabs converted to a shell code would prevent them being expanded anyplace but in the same shell.

Follow the example of COLOR and let users (including distros) decide what to enable. It will let those who want such to have it on, but not cause confusion or compatibility problems.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 05:26:01 GMT) Full text and rfc822 format available.

Message #14 received at 25388 debbugs.gnu.org (full text, mbox):

L A Walsh wrote:

Having 'ls' show different output through a pager
than it does when scrolling off the screen is not a standard feature

Sure it is. 'ls' has done that since then 1980s. 'ls' shows multicolumn output when the output is a tty, and single-column output when piped into a pager.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 18:49:02 GMT) Full text and rfc822 format available.

Message #17 received at 25388 debbugs.gnu.org (full text, mbox):

[Message part 1 (text/plain, inline)]

That's not what I'm used to:

Ishtar:/tmp/tmp> ls 12345678 a.exe* cccccccccccc.tar ssh-3LKwkRvOQM/ UiGUI_log_20161216-1442.htm bbbbbbbbb.gz ddddddddddddddd tall/ Ishtar:/tmp/tmp> ls |more
12345678 a.exe* cccccccccccc.tar ssh-3LKwkRvOQM/ UiGUI_log_20161216-1442.htm bbbbbbbbb.gz ddddddddddddddd tall/ Ishtar:/tmp/tmp> ls >../tmpdir.txt Ishtar:/tmp/tmp> wc ../tmpdir.txt 2 8 235 ../tmpdir.txt

i.e. they look the same either way -- even down to the same colors (attaching tmpdir.txt which should display as above but with color)... depends on your LS_COLOR settings... Will include that here (very long line):

export LS_COLORS='~=01;30:.bak=01;30:.orig=01;30:no=00:fi=00:di=01;35:ln=01;36:pi=40;33:so=04;01;34:do=04;01;34:bd=40;33;01:cd=40;33;01:or=41;33;01:su=01;04;36:sg=01;04;35:ca=01;04;36:tw=30;42:ow=34;42:st=37;44:ex=00;32:.tar=01;31:.tgz=01;31:.arj=01;31:.taz=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tlzma=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lz=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.rpm=01:30:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;3 5:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.axa=00;36:.oga=00;36:.spx=00;36:.xspf=00;36:'

ls alias on my machines: alias ls='ls -CF --show-control-chars --color=always'

[tmpdir.txt (text/plain, inline)]

[0m[00m12345678[0m [00;32ma.exe[0m* [01;31mcccccccccccc.tar[0m [01;35mssh-3LKwkRvOQM[0m/ [00mUiGUI_log_20161216-1442.htm[0m [01;31mbbbbbbbbb.gz[0m [00mddddddddddddddd[0m [30;42mtall[0m/

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 19:01:02 GMT) Full text and rfc822 format available.

Message #20 received at 25388 debbugs.gnu.org (full text, mbox):

That would be really annoying if it were true. It doesn't do that on any of my *nix terms. When I put it through a pager I want it to page the output I just saw. That's what it's for.

I.e. using a pager, should not, by default, change the output of programs. It's purpose is to moderate the output stream so a user can read what was output when they ran it without the pager. Changing the output, by default, depending on pager usage defeats the primary purpose of pagers.

If it became common place for programs to detect pager use and generate different output, it would seriously harm the intended use of pagers, since you would no longer be able to page the output of those programs (since what you are paging isn't the original output).

FWIW: If I want 1-column output, I usually use "-1".
That's what it is for, no?

-l

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 19:08:02 GMT) Full text and rfc822 format available.

Message #23 received at 25388 debbugs.gnu.org (full text, mbox):

On 01/09/2017 10:48 AM, L A Walsh wrote:

That's not what I'm used to:

I couldn't parse your email. I was describing the common behavior in BSD and GNU/Linux for ages, like this:

$ touch a b $ ls a b $ ls | more a b

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 19:34:01 GMT) Full text and rfc822 format available.

Message #26 received at 25388 debbugs.gnu.org (full text, mbox):

[Message part 1 (text/plain, inline)]

On 01/09/2017 01:00 PM, L A Walsh wrote:

Paul Eggert wrote: > L A Walsh wrote: >> Having 'ls' show different output through a pager than it >> does when scrolling off the screen is not a standard feature. > > Sure it is. 'ls' has done that since then 1980s. 'ls' shows > multicolumn output when the output is a tty, and single-column output > when piped into a pager.
That would be really annoying if it were true. It doesn't do that on any of my *nix terms. When I put it through a pager I want it to page the output I just saw. That's what it's for.

But that's EXACTLY what POSIX has specified, because it has been existing practice for YEARS.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ls.html "STDOUT The default format shall be to list one entry per line to standard output; the exceptions are to terminals or when one of the -C, -m, or -x options is specified. If the output is to a terminal, the format is implementation-defined. "

And POSIX merely codified existing practice (this is nothing new - it has been this way since the 70's) - the earliest versions of ls used isatty(), and implicitly turned on at least -q if stdout was a terminal or -1 if it was not, which means that you have ALWAYS had different behavior between 'ls' and 'ls | more', at least when ls is not some alias that explicitly specifies other command line options to override the defaults.

The GNU Coding Standards explicitly discourage NEW programs from changing default behavior based solely on whether stdout is a tty or not, but existing programs like ls are grandfathered in to have their historical behavior, which predates the GNU Coding Standards.

I.e. using a pager, should not, by default, change the output of programs. It's purpose is to moderate the output stream so a user can read what was output when they ran it without the pager. Changing the output, by default, depending on pager usage defeats the primary purpose of pagers.

I agree that it is not nice to change output merely based on the type of fd that stdout is, and GNU Coding Standards agrees in general. But like it or not, that's precisely what happens in ls due to backwards-compatibility - using a pager through a pipeline changes stdout from being a tty to instead being a pipeline, and therefore changes default output of at least 'ls', and such change in behavior is POSIX-compliant.

-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 19:39:01 GMT) Full text and rfc822 format available.

Message #29 received at 25388 debbugs.gnu.org (full text, mbox):

Paul Eggert wrote: > On 01/09/2017 10:48 AM, L A Walsh wrote: >> That's not what I'm used to: > > I couldn't parse your email.

I ran the ls command on a directory, shows 4 columns (that were in color).

Ishtar:/tmp/tmp> ls 12345678 a.exe* cccccccccccc.tar ssh-3LKwkRvOQM/ UiGUI_log_20161216-1442.htm bbbbbbbbb.gz ddddddddddddddd tall/

Next I ran the same ls command through more (really 'less' -- something set @ distro level, but found convenient, so kept it). Output is identical.

Ishtar:/tmp/tmp> ls |more
12345678 a.exe* cccccccccccc.tar ssh-3LKwkRvOQM/ UiGUI_log_20161216-1442.htm bbbbbbbbb.gz ddddddddddddddd tall/

Then I ran the same command and sent it to a file (which I attached to the email) and ran 'wc' on it -- which showed it had 2 lines and 8 "words" (filenames in this case).

Ishtar:/tmp/tmp> ls >../tmpdir.txt Ishtar:/tmp/tmp> wc ../tmpdir.txt 2 8 235 ../tmpdir.txt

I was describing the common behavior in BSD > and GNU/Linux for ages, like this: > > $ touch a b > $ ls > a b > $ ls | more > a > b

When I do the same, I get this:

ls a b ls | more a b

Which is how it has been for over 15 years on linux on my machines.

If my problem is that the 'ls' listing scrolled off the screen too fast for me to read, then the "default" behavior (that you describe) doesn't seem well thought out. If a user couldn't read it in multi-column mode, what would prompt anyone to think that they would automatically want it in 1-column mode when they put it through a pager -- doesn't make sense.

If I wanted 1 column format, why wouldn't I use "-1".
That's the entire purpose of the switch.

Again, I state that showing different output through a pager vs. what was shot out to the screen could allow a security problem in the form of a stenographic attack.

While dumping the output to a screen, maybe, a SW hack couldn't be covered up, but if they altered the sources of 'more/less', to hide their tracks, it would be noticeably harder to find.

It's not good practice, IMO, to be altering output based on what (or who) is reading it (at least not by default).

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 19:45:02 GMT) Full text and rfc822 format available.

Message #32 received at 25388 debbugs.gnu.org (full text, mbox):

[Message part 1 (text/plain, inline)]

On 01/09/2017 12:48 PM, L A Walsh wrote:

Sure it is. 'ls' has done that since then 1980s. 'ls' shows multicolumn output when the output is a tty, and single-column output when piped into a pager.
That's not what I'm used to:

ls alias on my machines: alias ls='ls -CF --show-control-chars --color=always'

That's because you are using an alias to alter 'ls' to non-default behavior for your personal default.

Try again with '\ls' everywhere you previously used 'ls', and you will see that output to the terminal is different than output to a file or pipeline.

-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 19:48:01 GMT) Full text and rfc822 format available.

Message #35 received at 25388 debbugs.gnu.org (full text, mbox):

[Message part 1 (text/plain, inline)]

On 01/09/2017 01:38 PM, L A Walsh wrote:

Paul Eggert wrote: > On 01/09/2017 10:48 AM, L A Walsh wrote: >> That's not what I'm used to: > > I couldn't parse your email.

I ran the ls command on a directory, shows 4 columns (that were in color).

That right there is a clue that your 'ls' is a non-standard alias that is adding additional command lines. 'ls' by default does not output in color.

Next I ran the same ls command through more (really 'less' -- something set @ distro level, but found convenient, so kept it). Output is identical.

Again, output is identical BECAUSE of your alias. But if you run:

\ls | more

you will get DIFFERENT output.

It's not good practice, IMO, to be altering output based on what (or who) is reading it (at least not by default).

Good or not, it IS what happens, and we can't change it due to years of usage.

-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 19:54:02 GMT) Full text and rfc822 format available.

Message #38 received at 25388 debbugs.gnu.org (full text, mbox):

Eric Blake wrote: > But that's EXACTLY what POSIX has specified, because it has been > existing practice for YEARS.

A bit louder Eric, you can't be heard.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ls.html "STDOUT The default format shall be to list one entry per line to standard output; the exceptions are to terminals or when one of the -C, -m, or -x options is specified. If the output is to a terminal, the format is implementation-defined.

Perhaps you could read my email -- especially the part

where I listed the alias I use for running 'ls'.

And POSIX merely codified existing practice (this is nothing new - it has been this way since the 70's)

Not anymore.

Breaking "rm -fr ." wasn't an existing practice except

at BSD-using dists (like BSD & SunOS). While Solaris was SysV, since it was bought up, it has changed.

The GNU Coding Standards explicitly discourage NEW programs from changing default behavior based solely on whether stdout is a tty or not, but existing programs like ls are grandfathered in to have their historical behavior, which predates the GNU Coding Standards.

Good -- that makes the issue pretty black & white.

Given those standards, then adding a NEW behavior that

does change the output depending on destination being tty or not would be "explicitly discouraged". Making the matter more clear is that it is NOT a case of grandfathering in a "historic behavior"

I agree that it is not nice to change output merely based on the type of fd that stdout is, and GNU Coding Standards agrees in general. But like it or not, that's precisely what happens in ls due to backwards-compatibility.

The old behavior of multi or single columns may be

historical and backwards compat -- I have no problem with that.

However, the new behavior of adding new-shell-encoded output as the default is not historical. It encodes output for bash and other new shells -- hindering cut&paste to other programs.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 19:58:02 GMT) Full text and rfc822 format available.

Message #41 received at 25388 debbugs.gnu.org (full text, mbox):

Gee guess you missed my 1st response to Andreas:

I changed it on my machine to put out color
even if through a pipe as I usually am piping it through 'less' and I want to see the color. If I want no color -- I can get the normal text by adding a backslash before the command.

So 'backslash' usage -- got that -- disables alias usage.

Thanks. :-)

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 20:01:01 GMT) Full text and rfc822 format available.

Message #44 received at 25388 debbugs.gnu.org (full text, mbox):

Eric Blake wrote: > Good or not, it IS what happens, and we can't change it due to years of > usage.

You are coming into the middle of the discussion.
I wasn't asking for a change to _that_ specific behavior.
I'm against adding more/new behaviors that also change

output and do it by default.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 20:08:02 GMT) Full text and rfc822 format available.

Message #47 received at 25388 debbugs.gnu.org (full text, mbox):

On 01/09/2017 12:00 PM, L A Walsh wrote:

I wasn't asking for a change to that specific behavior.

You argued that it's not standard for 'ls' to behave differently when piping output to a pager, and that the recent change was therefore undesirable. As the premise for this argument was incorrect, it shouldn't be surprising if readers disagree with its conclusion.

There are obviously advantages and disadvantages of the recent change, and we thought that the former outweighed the latter for typical users. If this isn't the case for you, you can change your personal alias to reflect your preferences. You're already using such an alias, since you already prefer alternatives to the default in other areas, so this shouldn't be much trouble.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 20:11:01 GMT) Full text and rfc822 format available.

Message #50 received at 25388 debbugs.gnu.org (full text, mbox):

tag 25388 notabug close 25388 stop

On 09/01/17 18:48, L A Walsh wrote:

ls alias on my machines: alias ls='ls -CF --show-control-chars --color=always'

Note a caveat of the above is that it changes the display width depending on whether you pipe to more or not. You could avoid that by changing the alias to:

alias ls='COLUMNS=$COLUMNS ls -CF --show-control-chars --color'

Also you can stick a -N option in that alias to disable the default quoting.

cheers, Pádraig.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 20:36:02 GMT) Full text and rfc822 format available.

Message #53 received at 25388 debbugs.gnu.org (full text, mbox):

I stand corrected for the historical behavior.

I also stand for it being strongly against GNU

standards to add more such behaviors.

As the premise for this argument was incorrect, it shouldn't be surprising if readers disagree with its conclusion.

Now you are deliberately excluding the fact.  While my

initial stance was that it was engaging in non-standard behavior (which is true) -- it is also the case that an exception was made for 'ls' for historical behavior.

This is not historical behavior.  'ls' doesn't get

a carte blanc to change output anyway it wants because of a specific historical behavior.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 20:38:01 GMT) Full text and rfc822 format available.

Message #56 received at 25388 debbugs.gnu.org (full text, mbox):

How do you reopen a bug that was closed before I even replied?

Pádraig Brady wrote:

tag 25388 notabug close 25388 stop

On 09/01/17 18:48, L A Walsh wrote:

ls alias on my machines: alias ls='ls -CF --show-control-chars --color=always'

Note a caveat of the above is that it changes the display width depending on whether you pipe to more or not. You could avoid that by changing the alias to:

alias ls='COLUMNS=$COLUMNS ls -CF --show-control-chars --color'

Also you can stick a -N option in that alias to disable the default quoting.

cheers, Pádraig.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 20:40:01 GMT) Full text and rfc822 format available.

Message #59 received at 25388 debbugs.gnu.org (full text, mbox):

reopen 25388 thanks

It's still a bug as it goes against adding new behaviors that generate different output based on destination = tty or not.

Pádraig Brady wrote:

tag 25388 notabug close 25388 stop

On 09/01/17 18:48, L A Walsh wrote:

ls alias on my machines: alias ls='ls -CF --show-control-chars --color=always'

Note a caveat of the above is that it changes the display width depending on whether you pipe to more or not. You could avoid that by changing the alias to:

alias ls='COLUMNS=$COLUMNS ls -CF --show-control-chars --color'

Also you can stick a -N option in that alias to disable the default quoting.

cheers, Pádraig.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 20:52:01 GMT) Full text and rfc822 format available.

Message #62 received at 25388 debbugs.gnu.org (full text, mbox):

[Message part 1 (text/plain, inline)]

On 01/09/2017 02:35 PM, L A Walsh wrote:

I also stand for it being strongly against GNU
standards to add more such behaviors.

'ls' did not recently add any more cases where tty output differs from non-tty output when all other things are equal in the default state. All that changed was that tty output is formatted differently than it has been in the past. And, as always, if you don't like the default, you can override it.

As the premise for this argument was incorrect, it shouldn't be surprising if readers disagree with its conclusion.
Now you are deliberately excluding the fact.  While my
initial stance was that it was engaging in non-standard behavior (which is true)

No, your claim that ls' behavior is non-standard has been proven to be false; I quoted the line from POSIX that explicitly permits ls to use implementation-defined behavior when outputting to a terminal, and if it is permitted by the standard, then the behavior can't be considered non-standard. It may not be what you want, but that does not make it non-standard.

-- it is also the case that an exception was made for 'ls' for historical behavior.
This is not historical behavior.  'ls' doesn't get
a carte blanc to change output anyway it wants because of a specific historical behavior.

Now you are excluding a fact. POSIX does state that the output of 'ls' when targetting a tty is implementation-defined, so we do get carte blanche permission to define our implementation to do what we think is the sanest default, even if our definition of sanest changes over time as we gain more experience on what types of problems trip up the most default users. The fact that ls output to a tty differs from what you get when outputting to a pipeline or file is irrelevant - ls output to a tty has ALWAYS differed from pipeline and file, when relying on the defaults. And if you don't LIKE the difference, then don't use the defaults - write your alias to override the defaults to ask explicitly for a format that does not add quoting you don't like.

And remember, scripts are UNLIKELY to be broken by the recent change in what gets output to a tty, because scripts don't usually direct 'ls' output to a tty, at least not if that output is going to affect the control flow of the script (as it is, scripts that try to parse 'ls' output are generally already broken; there are much better tools for handling directory contents than trying to parse 'ls' output).

-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 20:53:02 GMT) Full text and rfc822 format available.

Message #65 received at 25388 debbugs.gnu.org (full text, mbox):

[Message part 1 (text/plain, inline)]

On 01/09/2017 02:39 PM, L A Walsh wrote:

reopen 25388 thanks

It's still a bug as it goes against adding new behaviors that generate different output based on destination = tty or not.

This is NOT adding a new behavior that is based on isatty() checks. The output may differ, but there were no new isatty() checks added, therefore the new behavior is NOT a violation of GNU's stance that program output should not differ based solely on whether stdout is a tty.

-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 20:56:02 GMT) Full text and rfc822 format available.

Message #68 received at 25388 debbugs.gnu.org (full text, mbox):

[Message part 1 (text/plain, inline)]

On 01/09/2017 01:53 PM, L A Walsh wrote:

And POSIX merely codified existing practice (this is nothing new - it has been this way since the 70's)
Not anymore.

Breaking "rm -fr ." wasn't an existing practice except
at BSD-using dists (like BSD & SunOS). While Solaris was SysV, since it was bought up, it has changed.

Please quit trying to change the topic. My sentence about POSIX codifying the existing practice of 'ls' using different output to tty than it does to a pipeline is unrelated to your attempt to drag in 'rm' behavior to every single bug report. Whether or not 'rm' behavior changes have occurred over time, and whether such changes were codified by POSIX as existing practice, are not relevant to the current topic of 'ls' changing behavior.

-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 21:23:02 GMT) Full text and rfc822 format available.

Message #71 received at 25388 debbugs.gnu.org (full text, mbox):

Eric Blake wrote: > On 01/09/2017 02:35 PM, L A Walsh wrote: >> I also stand for it being strongly against GNU >> standards to add more such behaviors. > > 'ls' did not recently add any more cases where tty output differs from > non-tty output when all other things are equal in the default state. > All that changed was that tty output is formatted differently than it > has been in the past.

That is not the case.

Created dir with common files w/spaces in them from people's Doc dirs. I also replaced the space in "My Pictures w/a tab to see the effect -- though it wasn't necessary.

ls All Settings My Pictures Show Desktop.scf browser - logitech Bluetooth Software My Documents Start Menu Local Settings Saved Games Virtual Machines

Then I did ls the standard way:

And quoted way:

'All Settings' 'Bluetooth Software' 'Local Settings' 'My'$'\t''Pictures' 'My Documents' 'Saved Games' 'Show Desktop.scf' 'Start Menu' 'Virtual Machines' 'browser - logitech'

I cut & pasted into bash, and read them into array vars using

readarray -t std

and the same for the quoted version but to "readarray -t quoted".

Now I look for the files 1 at a time in a loop: First old way (using -gG to shorten line): -rw-rw-r-- 1 0 Jan 9 13:06 All Settings -rw-rw-r-- 1 0 Jan 9 13:06 Bluetooth Software -rw-rw-r-- 1 0 Jan 9 13:06 Local Settings -rw-rw-r-- 1 0 Jan 9 13:06 My Pictures -rw-rw-r-- 1 0 Jan 9 13:06 My Documents -rw-rw-r-- 1 0 Jan 9 13:06 Saved Games -rw-rw-r-- 1 0 Jan 9 13:06 Show Desktop.scf -rw-rw-r-- 1 0 Jan 9 13:06 Start Menu -rw-rw-r-- 1 0 Jan 9 13:06 Virtual Machines -rw-rw-r-- 1 0 Jan 9 13:06 browser - logitech

And now the new way that you claim is "the same output":

ls: cannot access 'All Settings': No such file or directory ls: cannot access 'Bluetooth Software': No such file or directory ls: cannot access 'Local Settings': No such file or directory ls: cannot access 'My'$'t''Pictures': No such file or directory ls: cannot access 'My Documents': No such file or directory ls: cannot access 'Saved Games': No such file or directory ls: cannot access 'Show Desktop.scf': No such file or directory ls: cannot access 'Start Menu': No such file or directory ls: cannot access 'Virtual Machines': No such file or directory ls: cannot access 'browser - logitech': No such file or directory

your claim that ls' behavior is non-standard has been proven to be false; I quoted the line from POSIX that explicitly permits ls

By permitting it as an exception, you are admitting it

is non-standard compared to other programs.

implementation-defined behavior when outputting to a terminal, and if it is permitted by the standard, then the behavior can't be considered non-standard. It may not be what you want, but that does not make it non-standard.

Now you are excluding a fact. POSIX does state that the output of 'ls' when targetting a tty is implementation-defined,

I'm not quoting posix -- I'm quoting the GNU

standards you mention.

Anyway, as demonstrated above, the output is not the same and varies based on whether or not the destination is a tty.

Your claim that the output is the same is demonstrably false. They are not functionally equivalent.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 21:24:01 GMT) Full text and rfc822 format available.

Message #74 received at 25388 debbugs.gnu.org (full text, mbox):

Eric Blake wrote: > On 01/09/2017 01:53 PM, L A Walsh wrote: >>> And POSIX merely codified existing practice (this is nothing new - it >>> has been this way since the 70's) >> --- >> Not anymore. >> >> Breaking "rm -fr ." wasn't an existing practice except >> at BSD-using dists (like BSD & SunOS). While Solaris was SysV, since it >> was bought up, it has changed. > > Please quit trying to change the topic.

You said that posix codified existing practice and that it has been this way since the 70's. This is not true.

You are making overly broad statements, or, more likely, using indeterminant pronouns.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 21:54:02 GMT) Full text and rfc822 format available.

Message #77 received at 25388 debbugs.gnu.org (full text, mbox):

[Message part 1 (text/plain, inline)]

On 01/09/2017 03:23 PM, L A Walsh wrote:

Eric Blake wrote: > On 01/09/2017 01:53 PM, L A Walsh wrote: >>> And POSIX merely codified existing practice (this is nothing new - it >>> has been this way since the 70's) >> --- >> Not anymore. >> >> Breaking "rm -fr ." wasn't an existing practice except >> at BSD-using dists (like BSD & SunOS). While Solaris was SysV, since it >> was bought up, it has changed. > > Please quit trying to change the topic.
You said that posix codified existing practice and that it has been this way since the 70's. This is not true.

You're quoting me out of context.

When coupled with the sentences prior to what got quoted in this email, I was trying to emphasize only that that POSIX codified the existing practice of ls outputting different data to a tty than to a non-tty, and that the behavior of different ls output to a tty has been around since the 70s, which explains why POSIX codified that particular aspect of ls as standard-conforming behavior.

By eliding the context, you are then attempting to broaden my statement into me making a blanket claim that POSIX can only ever codify existing practice (which may are may not be the case, but it was not the point I was trying to make), by twisting my claim into something unrelated to ls and dragging rm into the mix.

You are making overly broad statements, or, more likely, using indeterminant pronouns.

That was not my intent. Email is a lousy medium for communications.

Here's the full context, for reference:

Sure it is. 'ls' has done that since then 1980s. 'ls' shows multicolumn output when the output is a tty, and single-column output when piped into a pager.

That would be really annoying if it were true. It doesn't do that on any of my *nix terms. When I put it through a pager I want it to page the output I just saw. That's what it's for.

But that's EXACTLY what POSIX has specified, because it has been existing practice for YEARS.

So in my sentence, as originally written, the pronoun of "that's EXACTLY" is referring only to the earlier wording "'ls' has done that since then 1980s", and not a sweeping generalization to all of POSIX.

And in my other sentence:

And POSIX merely codified existing practice (this is nothing new - it has been this way since the 70's)

the "it" is referring, again, to the behavior of 'ls' outputting something different for ttys. That is, read it as "(this is nothing new

ls having different tty output has been this way since the 70's)", and NOT as "(this is nothing new - POSIX has been codifying existing practice in this way since the 70's)" - particularly since POSIX did not exist in the 70's and therefore could not have been codifying existing practice for a duration of time that long.

-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 22:54:02 GMT) Full text and rfc822 format available.

Message #80 received at 25388 debbugs.gnu.org (full text, mbox):

Eric Blake wrote:

You're quoting me out of context.

Your context was not clear.

When coupled with the sentences

Your "But" started that sentence responding to my statement.

While you may have thought you included all previous context, it was interwoven with my statements, so the context was muddied.

You are making overly broad statements, or, more likely, using indeterminant pronouns.

That was not my intent. Email is a lousy medium for communications.

That was how I took it, especially since in other areas

posix has had no problem being more than descriptive.

As soon as you claimed I was dragging in unrelated items,

I figured out what you had meant. But the thing about posix only describing -- which was their original mission statement, to becoming a prescriptive body, is more than a bit of a sore spot, especially since one or more BSDer's have gone off on me in the past about how BSD won the standards war against SysV, since now the BSD'ers were able to be a majority in posix and change the standards to along BSD lines vs. SysV lines. That made me realize what a waste of energy was going to go into making bunches of tools incompatible with how they have been. ...And it progresses...

I liked BSD, but now I see its followers as more than a bit

religious, and unwilling to see anyone else's view. It's probably no coincidence that Apple chose a BSD-system as their base.

As for email being a lousy medium... yeah, it's not the best,

but not the worst. You went off on me because you thought I was changing ls's historical behavior, when my issue is compounding those incompatibilities by adding flaky features that can't be cut/pasted into anything. It's a poor option to do by default.

Information forwardedto bug-coreutils <at> gnu.org:
bug#25388; Package coreutils. (Mon, 09 Jan 2017 22:58:02 GMT) Full text and rfc822 format available.

Message #83 received at 25388 debbugs.gnu.org (full text, mbox):

[Message part 1 (text/plain, inline)]

On 01/09/2017 03:22 PM, L A Walsh wrote:

Eric Blake wrote: > On 01/09/2017 02:35 PM, L A Walsh wrote: >> I also stand for it being strongly against GNU >> standards to add more such behaviors. > > 'ls' did not recently add any more cases where tty output differs from > non-tty output when all other things are equal in the default state. > All that changed was that tty output is formatted differently than it > has been in the past.
That is not the case.
Created dir with common files w/spaces in them from people's Doc dirs. I also replaced the space in "My Pictures w/a tab to see the effect -- though it wasn't necessary.

Okay, I'm going to follow along, using two different versions of ls: 8.25 (Fedora 25) and 8.4 (RHEL 6).

$ mkdir -p /tmp/ls $ cd /tmp/ls $ touch 'a b' c

Now, let's check whether output differed based on tty in the old version, 8.4:

$ \ls a b c $ \ls | cat a b c $ \ls -1 | cat a b c $ \ls --version | head -n1 ls (GNU coreutils) 8.4

Yep, the output is different based on whether stdout was a tty.

Now, let's check again with a modern version:

$ \ls 'a b' c $ \ls | cat a b c $ \ls -1 | cat a b c $ \ls --version | head -n1 ls (GNU coreutils) 8.25

Yep, the output STILL differs based on whether stdout was a tty. But note, that in BOTH cases, the output of '\ls -1' is identical when stdout is NOT a tty. If you are parsing the output of ls -1 in a script, chances are you are NOT creating a pty, and therefore the output you are parsing did NOT change. The only content that changed between ls versions is the CONTENT of the tty output, but NOT the logic of whether to use tty output, nor the content when tty output is not involved.

The GNU Coding Standards do NOT forbid the content of tty output to change between versions (that is, the tty output of version x.1 and x.2 need not be the same), they only state that if version x.1 outputs a particular text to a non-tty, it should default to outputting that same text to a tty, IF that difference in behavior was not already grandfathered in.

Let's go read the actual GNU Coding Standards, instead of paraphrasing it:

https://www.gnu.org/prep/standards/standards.html#User-Interfaces

"Likewise, please don’t make the behavior of a command-line program depend on the type of output device it gets as standard output or standard input. Device independence is an important principle of the system’s design; do not compromise it merely to save someone from typing an option now and then. (Variation in error message syntax when using a terminal is ok, because that is a side issue that people do not depend on.)

If you think one behavior is most useful when the output is to a terminal, and another is most useful when the output is a file or a pipe, then it is usually best to make the default behavior the one that is useful with output to a terminal, and have an option for the other behavior. You can also build two different versions of the program with different names.

There is an exception for programs whose output in certain cases is binary data. Sending such output to a terminal is useless and can cause trouble. If such a program normally sends its output to stdout, it should detect, in these cases, when the output is a terminal and give an error message instead. The -f option should override this exception, thus permitting the output to go to the terminal.

Compatibility requires certain programs to depend on the type of output device. It would be disastrous if ls or sh did not do so in the way all users expect. In some of these cases, we supplement the program with a preferred alternate version that does not depend on the output device type. For example, we provide a dir program much like ls except that its default output format is always multi-column format."

Nothing in there says that we can't change what ls outputs to a termianl between versions of ls, merely that the reason that ls DEFAULTS to different output to terminal than to non-terminal is historical. Furthermore, it encourages output to a terminal to default to being safe in the presence of binary data (filenames can contain control characters, which if not properly quoted are indeed disastrous to the terminal), and does not forbid us from using a version of quoting that is less ambiguous in newer versions of ls than it was in previous versions.

your claim that ls' behavior is non-standard has been proven to be false; I quoted the line from POSIX that explicitly permits ls
By permitting it as an exception, you are admitting it
is non-standard compared to other programs.

Huh? The fact that ls is allowed to have implementation-defined output when writing to a tty is standard. It may not be consistent with other applications, but the point being discussed here wasn't consistency, but standards-compliant. You can't be non-standard by comparing to another app - either the behavior of ls is permitted by the standard or it is not permitted, regardless of what other apps do.

Now you are excluding a fact. POSIX does state that the output of 'ls' when targetting a tty is implementation-defined,
I'm not quoting posix -- I'm quoting the GNU
standards you mention.

I actually quoted it above. So far, you have only paraphrased it, and I think you are trying to claim that the GNU Coding Standards do not let us change the tty-only output between versions, but that is not what it says.

Anyway, as demonstrated above, the output is not the same and varies based on whether or not the destination is a tty.

Indeed - and we've been telling you that default ls output has ALWAYS varied based on whether or not the destination is a tty. The fact that it STILL varies (although with different content than in older versions of ls) is nothing new. We are not violating the requirement that no NEW code should have a difference based on tty, because we already HAVE a difference based on tty.

Your claim that the output is the same is demonstrably false.

No, you have only managed to prove that the tty-only output of ls has changed between versions, and NOT that the tty-only and non-tty output used to be the same but now differ. The tty-only and non-tty output have ALWAYS been different, and I was not claiming that they have been the same.

They are not functionally equivalent.

You are trying to feed tty output into a shell script. In general, this is not portable. But this is not ls' fault. It has NEVER been portable to try and parse the tty-only output of ls in a shell script. Just because the outcome of trying to parse non-portable output has changed doesn't mean that the output is broken - rather, your attempt to feed non-portable output to the shell in the first place is broken.

-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

**Changed bug title to 'ls-quotes: kills existing scripts reading "ls" -1 as input' from 'Bug in ls, kills existing scripts reading "ls" -1 as input'**Request was from Assaf Gordon <assafgordon <at> gmail.com>to control <at> debbugs.gnu.org. (Sun, 28 Oct 2018 07:50:02 GMT) Full text and rfc822 format available.

Reply sentto Assaf Gordon <assafgordon <at> gmail.com>:
You have taken responsibility. (Thu, 13 Dec 2018 20:30:03 GMT) Full text and rfc822 format available.

Notification sentto L A Walsh <coreutils <at> tlinx.org>:
bug acknowledged by developer. (Thu, 13 Dec 2018 20:30:03 GMT) Full text and rfc822 format available.

Message #90 received at 25388-done debbugs.gnu.org (full text, mbox):

Hello,

'ls' did not recently add any more cases where tty output differs from non-tty output when all other things are equal in the default state. All that changed was that tty output is formatted differently than it has been in the past.

We created a summary of common issues and FAQs regarding the quoting change in ls(1): https://www.gnu.org/software/coreutils/quotes.html

If there is an issue that is not addressed there, please send an email to coreutils gnu.org .

regards,

assaf

**bug archived.**Request was from Debbugs Internal Request <help-debbugs <at> gnu.org>to internal_control <at> debbugs.gnu.org. (Fri, 11 Jan 2019 12:24:10 GMT) Full text and rfc822 format available.

This bug report was last modified 6 years and 154 days ago.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.

#25388 - ls-quotes: kills existing scripts reading "ls" -1 as input (original) (raw)

Andreas Schwab wrote: > On Jan 06 2017, L A Walsh <coreutils tlinx.org> wrote: >> The new ls doesn't maintain backwards compatibility. >> The default options now add extraneous quotes to >> some filenames.

Paul Eggert wrote: > On 01/09/2017 10:48 AM, L A Walsh wrote: >> That's not what I'm used to: > > I couldn't parse your email.

I was describing the common behavior in BSD > and GNU/Linux for ages, like this: > > $ touch a b > $ ls > a b > $ ls | more > a > b

Paul Eggert wrote: > On 01/09/2017 10:48 AM, L A Walsh wrote: >> That's not what I'm used to: > > I couldn't parse your email.

Eric Blake wrote: > But that's EXACTLY what POSIX has specified, because it has been > existing practice for YEARS.

Eric Blake wrote: > Good or not, it IS what happens, and we can't change it due to years of > usage.

Paul Eggert wrote: > On 01/09/2017 12:00 PM, L A Walsh wrote: >> I wasn't asking for a change to that specific behavior. > > You argued that it's not standard for 'ls' to behave differently when > piping output to a pager, and that the recent change was therefore > undesirable.

All Settings Bluetooth Software Local Settings My Pictures My Documents Saved Games Show Desktop.scf Start Menu Virtual Machines browser - logitech