How to parse lines in shlex? (original) (raw)

October 26, 2024, 3:49pm 1

so if i try this:

import shlex
print(list(shlex.shlex("""
rm -rf /
echo Done
""")))

i get a flat list of all arguments, same as if i had one line shlex.shlex("rm -rf / echo Done"), not separated commands

does anyone know how to parse separate commands with it?

i know i could just split the string on '\n' but that would break multiline arguments

MegaIng (Cornelius Krupp) October 26, 2024, 4:01pm 2

You can’t, shlex is not designed for this usecase. There might be a third party library that will help you with this.

chepner (Clint Hepner) October 26, 2024, 4:43pm 3

You can try changing what the tokenizer considers whitespace (which it discards):

text = """rm -rf /
echo "Almost done...
done"
"""

p = shlex.shlex(text)
p.whitespace = ' \r\t'  # default minus \n

Then

>>> print(list(p))
['rm', '-', 'rf', '/', '\n', 'echo', '"Almost done...\ndone"', '\n']

Keep in mind that shlex implements a shell-like tokenizer, not an actual shell tokenizer, let alone a shell parser, which is what actually identifies lines as constructs consisting of multiple tokens.

two (porgraming) October 27, 2024, 11:19am 4

thank you

two (porgraming) October 27, 2024, 11:44am 5

that will still parse it same as an argument of one newline, so i think i’ll still just split on lines, because i’ll have a clear error if an input is not working properly