this post was submitted on 20 Feb 2024
-1 points (46.2% liked)

Linux

2359 readers
10 users here now

Shit, just linux.

Use this community for anything related to linux for now, if it gets too huge maybe there will be some sort of meme/gaming/shitpost spinoff. Currently though… go nuts

founded 1 year ago
MODERATORS
[email protected]
-1
mv *.jpg is complete bullshit. (sh.itjust.works)
submitted 9 months ago by [email protected] to c/[email protected]
28 comments fedilink hide all child comments
 

'Well it's only passing mv a list of--' yeah yeah yeah, I know, and that's why I'm calling bullshit. It should be massively harder to execute filenames. Even if 1970s decisions make that the eternal hideous default: the lack of any idiot-proof standard workaround is incomprehensible.

StackOverflow's full of competing one-liners and people pointing out how each one is considered harmful. The least-skeezy options use exec. That sentence should make anyone recoil in horror.

This is not a filename problem. This is a tool problem. If a single printable character is going to silently expand into a list of names, then for god's sake, having it put each name in quotes should be fucking trivial.

top 28 comments
sorted by: hot top controversial new old
[–] [email protected] 4 points 9 months ago* (last edited 9 months ago) (2 children)

This is not standard behavior. Got links to StackOverflow?

This works just fine in bash and zsh:

encoded_funky_name=IUAjJCVeJiooKVtde30nIjo7IGxzOyAmYW1wOyA8Plx8YH5gJChscykke1BBVEh9ISF8fCYmLmpwZw==
# decodes to: !@#$%^&*()[]{}'":; ls; &amp; <>\|`~`$(ls)${PATH}!!||&&.jpg

mkdir temp
cd temp
mkdir temp2

touch "$(echo "${encoded_funky_name}" | base64 -d)"

mv ./*.jpg temp2
ls -lR

mv temp2/*.jpg .
ls -lR

Note that it is recommended to use "./" before a raw * in globs to avoid filenames beginning with "-" being interpreted by the command as special arguments (this is not necessary in the above example, but it is a good habit). See https://dwheeler.com/essays/filenames-in-shell.html#globbing for more info. e.g.:

touch ./-laR 

# Bad ls command:
ls *

# Good ls command:
ls ./*

having it put each name in quotes should be fucking trivial

This is effectively what happens, except that it's actually more robust than that, since it also accounts for names with quotes and other special characters. Glob expansion does NOT output file names into the shell; it passes them as arguments to the command with no further shell processing. Arguments can contain any special characters.

Not sure why you're seeing this behavior. Are you doing something with these file names before passing them to mv? Storing them in variables? Saving them to files?

[–] [email protected] 0 points 9 months ago* (last edited 9 months ago)

Note that it is recommended to use “./” before a raw * in globs to avoid filenames beginning with “-” being interpreted by the command as special arguments

Jackpot! I think! Based on OP's reply to me, this is almost definitely the real problem:

mv *.jpg /mnt/Example/Pictures

Where it will then die with an error like mv: invalid option -- '1' depending on the contents of the current directory.

[–] [email protected] 2 points 9 months ago (1 children)

It's literally just mv *.jpg /mnt/Example/Pictures.

Multiple StackOverflow answers take this as read.

mv ./*.jpg /mnt/Example/Pictures works fine, though, so thank you for the practical solution. I'm still disgusted there's any filename that gets treated as... not a filename. That broken stair might be older than I am.

[–] [email protected] 4 points 9 months ago* (last edited 9 months ago) (1 children)

Ahhh. That's a confusing thread, with a couple confounding factors.

The OP in that thread is attempting to a batch rename, which is out of scope of the mv command.

If you are NOT trying to change the name of the file, I recommend always making the last argument of mv your destination directory, rather than the destination file. This is strictly mandatory if you are moving multiple files.

OP in that SO thread used the filename as the destination path — a filename which did not exist, and therefore could not be resolved by the wildcard. And if it had existed, the move would have failed. Because of that, mv /data/*/Sample_*/logs/*_Data_time.err /data/*/Sample_*/logs/*_Data_time_orig.err is not a valid command.

OP cannot accomplish what they want with a single mv command. They would need to use a loop or something like find.

I agree that the lack of universal consistency in how commands handle arguments is unfortunate. There's no way around that without reinventing the entire OS. Many commands let you pass -- as an argument to signify that any subsequent arguments should not be interpreted as options even if they begin with -. But that's entirely up to the implementation of each program; it's not a shell feature.

Future OS designers: consider typed arguments. Abandon all legacy cruft.

[–] [email protected] 1 points 9 months ago (1 children)

There’s no way around that without reinventing the entire OS.

Disagree. If wildcard expansion is not a feature of the tool, it shouldn't be up to the tool to shape how each expanded input is passed. Some hideous escape syntax ought to force *.jpg to appear as "-a.jpg" instead of -a.jpg, once it reaches mv.

Instead * is acting as a cheeky little ls alternative, minus any degree of formatting control, and piping actual ls input into mv is similarly fraught with complications.

[–] [email protected] 2 points 9 months ago (1 children)

Some hideous escape syntax ought to force *.jpg to appear as "-a.jpg" instead of -a.jpg, once it reaches mv

The issue here is that there's no difference here between -a.jpg and "-a.jpg".

Go ahead and try these commands. They are 100% equivalent:

ls -laR
ls "-laR"
ls '-laR'

The reason these are equivalent is because quoting (and expansion, and globbing) is processed in the shell. The shell then passes the evaluated string as an argument to the tool. All of those evaluate to the same string, so they look exactly the same to the ls command.

What is the solution if there's a file name that matches something the tool thinks is an argument? What would the correct behavior be?

The only solution is to include some way to communicate to the tool what each argument is. In practice, you could do ls -- * and it would work even if the first filename was "-laR". However, not all tools use the -- convention. This is really core to the design of the OS, in that command arguments have no type.

[–] [email protected] -1 points 9 months ago (1 children)

Jesus. That's beyond Javascript levels of "helpfully" reinterpreting strings. That's borderline Excel behavior.

What is the point of strings allowing every character besides \0 if they're gonna eat quotation marks?

[–] [email protected] 3 points 9 months ago* (last edited 9 months ago) (2 children)

All I can tell you is that, in my opinion, it's ridiculous and terrible that old-school terminals haven't been replaced yet with something more user-friendly and self-explanatory, at least in the same-machine user space. But given that they are what they are, some basic understanding of what shells do is required in order to use them, and you don't have that understanding (I don't fault you for this).

The key point here is that programs/commands always receive an array of string arguments, and it's the shell's job to translate your command line into that. Quoting (like in -m="my message"), variable expansion (like $HOME) and various other things are processed by the shell and not the program, and the expectation is that the user knows this. So quotes are never visible to programs, and the upside is that programs never need to process such quotes - making these features universal without having various random bugs in each program's implementation.

[–] [email protected] 1 points 9 months ago (1 children)

I've been using command-line programs for twenty-five years. "Basic" is not the issue, here. This is obscene edge-case behavior for what honestly should be a Hello World level example.

Thoroughly explicable causes do not make the outcome any less of a problem.

[–] [email protected] 1 points 9 months ago

I’ve been using command-line programs for twenty-five years.

A general statement (not about you):

I've been using my human body for over 25 years (exact number withheld) and still don't know shit about how it works and what it does. I don't even know the muscle groups, which most people who care about their health learn. But I still use them. Years of experience don't always translate to understanding and knowledge.

More specifically about you (and me): I'm sure you know a lot about how to use the shell, and I'll absolutely bet you know more tricks than me to get shit done. But you still didn't know about the quoting thing. I have a tendency to dig into understanding how shit works while not developing my actual skills in using them. You might have the opposite tendency. The best is somewhere in the middle, I think.

[–] [email protected] 2 points 9 months ago (1 children)

This is exactly correct. This is why I say it would require a whole new OS with no regard for compatibility with current systems.

As long as arguments are typeless and transmitted in a single array, there is no universal way to make programs distinguish between an option and file name that are identical. Full stop. No way. This has nothing to do with quoting schemes.

There are conventions to work around this (like --) but that is at the command level, NOT the shell level. It is not universal.

[–] [email protected] 1 points 9 months ago (1 children)

Okay, I pretty much agree with you in theory, but in practice you're basically describing PowerShell which I just cannot. :P

[–] [email protected] 2 points 9 months ago

LOL. Now I feel dirty.

To be fair, there's a lot that PowerShell does much better than Bash and similar shells. Piping tables as tables is rad.

[–] [email protected] 2 points 9 months ago (1 children)

I have two questions:

  1. Is this some recent news that you expect us to have heard of? Your post demands context that is not easily understood (but it's possible to understand through multiple reads)
  2. What shell are you using?

I'm using bash (GNU bash, version 5.1.16(1)-release) and don't observe this behavior:

$ ls
'first file.txt'  'second file.txt'
$ cat 'first file.txt' 
this is originally named "first file.txt"
$ cat 'second file.txt' 
this is originally named "second file.txt"
$ mv *.txt
$ ls
'second file.txt'
$ cat 'second file.txt' 
this is originally named "first file.txt"

It's possible that whatever shell you're using has a bug, but it's definitely not universal. It's also possible that this bug only affects builtins like mv and not non-builtin commands.

[–] [email protected] 1 points 9 months ago (2 children)

I expected people have tried to move files... using the move command.

I did not expect I'd have to specify that there's a destination, in an explanation of what *.jpg does, not an explanation of what mv does.

For clarity:

GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)

mv *.jpg /mnt/Example/Pictures

Where it will then die with an error like mv: invalid option -- '1' depending on the contents of the current directory.

Every search about this has led back to this StackOverflow page sooner or later. All answers address wildcard expansion.

[–] [email protected] 1 points 9 months ago (1 children)

When I run into situations like this, I use the commands that work to write out a script. Eg, in your case the wildcard isn't working with the mv command, so do something like this:

ls -1 *.jpg | awk '{print "mv ""$1"" /mnt/example/Pictures"}' > /tmp/movefiles.sh

Then check the movefiles.sh and make sure it has all of the commands and files properly stated, make that executable, and then run that.

[–] [email protected] 1 points 9 months ago (1 children)

ls | grep | mv would work, except the StackOverflow discussion also highlights how parsing ls can have the same issues.

I am moving thousands of files at once. If I have to check each one, it's still wrong.

The pragmatic answer turns out to be ./* instead of *.

[–] [email protected] 1 points 9 months ago* (last edited 9 months ago)

I saw that answer and was just offering another option. I am sure xargs might work, but you would need to test as you need a destination passed on each line. Back to my way, I have used it for a lot more than just the move command. I think I used it to do a chmod once where I wanted to check and make sure before I committed to actually running the command(s). You could also use find and the -exec option, which I think was also mentioned here.

Edit: also, you wouldn't need to check each one, just the first few and last few to make sure the syntax is correct. Maybe do a wc -l to make sure it's got the right number of entries and then let it run.

[–] [email protected] 1 points 9 months ago* (last edited 9 months ago) (1 children)

Can you run printf %s\\n *.jpg in that same directory and share the result? I'm really intrigued. If there's private information in there, I'd be satisfied with just a handful of lines showing the issue with private info replaced with ****s

Edit: also run alias mv to check whether you have an alias messing things up

[–] [email protected] 2 points 9 months ago

bash: alias: mv: not found

printf %s\\n *.jpg is just a list of filenames on newlines. There's 75,000 of them. You'll have to take my word for it.

Fortunately, GenderNeutralBro's aside about using ./* instead of * works as expected. Could have been files beginning with _-. None begin with -, exactly.

[–] [email protected] 2 points 9 months ago (2 children)

What is wrong with mv *.jpg?

[–] [email protected] 3 points 9 months ago

I'm also a bit lost. From the final sentence "having it put each name in quotes should be fucking trivial." OP seems to complain that filenames with spaces expand into multiple arguments. I can't recreate this with this command in bash:

$ printf %s\\n *.jpg

This command prints each argument in its own line. Replace *.jpg with whatever to prove it to yourself. Filenames with spaces show in full in a single line for me.

[–] [email protected] 1 points 9 months ago (2 children)

It expands *.jpg into a list of names.

Which you'd think would mean mv "thing.jpg" "other thing.jpg", but no, it does mv thing.jpg other thing.jpg - and then chokes, because other is not a valid option or filename. Possibly after writing the contents of other thing.jpg onto the destination thing.jpg.

If any filenames contain characters like ; or &amp;, you better hope the next word doesn't look like a command.

[–] [email protected] 5 points 9 months ago

Which shell are you using? This does not happen in Bash, Zsh, or any modern shell I'm familiar with.

[–] [email protected] 1 points 9 months ago* (last edited 9 months ago) (1 children)

What shell are you using? Is this exclusive to mv (and maybe other builtins)?

Edit: also why would you ever want to move files like this, what is wrong with you

[–] [email protected] -2 points 9 months ago (2 children)

Edit: also why would you ever want to move files like this, what is wrong with you

... like what?

The fuck do you mean, why would I want to move files in a terminal? Because I'm doing shit in a terminal! What, do you expect people to Super+E, find the directory they're in, and drag-and-drop? I'm using tools that only exist in the command line.

I'm using whatever GNOME Terminal came with Linux Mint 20 by default. StackOverflow answers sound like any POSIX terminal will do this. I assume it's standard behavior.

[–] [email protected] 2 points 9 months ago

I’m using whatever GNOME Terminal came with Linux Mint 20 by default.

Should be bash. Type help it should tell you.

StackOverflow answers sound like any POSIX terminal will do this. I assume it’s standard behavior.

Link?

[–] [email protected] 1 points 9 months ago* (last edited 9 months ago)

I mean with * where you have no control of the order of the files and you never explicitly say which files are being moved. "yeah, just rename files into other files, I don't fucking care lol" is what your command does