this post was submitted on 15 Jan 2025
84 points (97.7% liked)

Programming

18001 readers
111 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]



founded 2 years ago
MODERATORS
 

This may make some people pull their hair out, but I’d love to hear some arguments. I’ve had the impression that people really don’t like bash, not from here, but just from people I’ve worked with.

There was a task at work where we wanted something that’ll run on a regular basis, and doesn’t do anything complex aside from reading from the database and sending the output to some web API. Pretty common these days.

I can’t think of a simpler scripting language to use than bash. Here are my reasons:

  • Reading from the environment is easy, and so is falling back to some value; just do ${VAR:-fallback}; no need to write another if-statement to check for nullity. Wanna check if a variable’s set to something expected? if [[ <test goes here> ]]; then <handle>; fi
  • Reading from arguments is also straightforward; instead of a import os; os.args[1] in Python, you just do $1.
  • Sending a file via HTTP as part of an application/x-www-form-urlencoded request is super easy with curl. In most programming languages, you’d have to manually open the file, read them into bytes, before putting it into your request for the http library that you need to import. curl already does all that.
  • Need to read from a curl response and it’s JSON? Reach for jq.
  • Instead of having to set up a connection object/instance to your database, give sqlite, psql, duckdb or whichever cli db client a connection string with your query and be on your way.
  • Shipping is… fairly easy? Especially if docker is common in your infrastructure. Pull Ubuntu or debian or alpine, install your dependencies through the package manager, and you’re good to go. If you stay within Linux and don’t have to deal with differences in bash and core utilities between different OSes (looking at you macOS), and assuming you tried to not to do anything too crazy and bring in necessary dependencies in the form of calling them, it should be fairly portable.

Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.

For most bash gotchas, shellcheck does a great job at warning you about them, and telling how to address those gotchas.

There are probably a bunch of other considerations but I can’t think of them off the top of my head, but I’ve addressed a bunch before.

So what’s the dealeo? What am I missing that may not actually be addressable?

(page 2) 50 comments
sorted by: hot top controversial new old
[–] [email protected] 8 points 2 weeks ago (5 children)

Just make certain the robustness issues of bash do not have security implications. Variable, shell, and path evalutions can have security issues depending on the situation.

load more comments (5 replies)
[–] [email protected] 2 points 2 weeks ago* (last edited 2 weeks ago) (2 children)

Run checkbashisms over your $PATH (grep for #!/bin/sh). That's the problem with Bash.
#!/bin/sh is for POSIX compliant shell scripts only, use #!/bin/bash if you use bash syntax.

Btw, i quite like yash.

[–] [email protected] 1 points 2 weeks ago (1 children)

Any reason to use #!/bin/sh over #!/usr/bin/env sh?

[–] [email protected] 1 points 2 weeks ago* (last edited 2 weeks ago) (3 children)

I personally don't see the point in using the absolute path to a tool to look up the relative path of your shell, because shell is always /bin/sh but the env binary might not even exist.

Maybe use it with bash, some BSD's or whatever might have it in /usr without having /bin symlinked to /usr/bin.

load more comments (3 replies)
load more comments (1 replies)
[–] [email protected] 17 points 2 weeks ago (2 children)

One thing that I don't think anyone else has mentioned is data structures. Bash does have arrays and hashmaps at least but I've found that working with them is significantly more awkward than in e.g. python. This is one of several reasons for why bash doesn't scale up well, but sure for small enough scripts it can be fine (if you don't care about windows)

[–] [email protected] 3 points 2 weeks ago

That’s definitely worth mentioning indeed. Bash variables, aside from arrays and hashmaps that you get with declare, are just strings. Any time you need to start capturing a group of data and do stuff with them, it’s a sign to move on. But there are many many times where that’s unnecessary.

[–] [email protected] 7 points 2 weeks ago (2 children)

I think I mentioned it, but inverse: The only data type I'm comfortable with in bash are simple string scalars; plus some simple integer handling I suppose. Once I have to think about stuff like "${foo[@]}" and the like I feel like I should've switched languages already.

Plus I rarely actually want arrays, it's way more likely I want something in the shape of

@dataclass(frozen=True)
class Foo:
    # …

foos: set[Foo] = …
load more comments (2 replies)
[–] [email protected] 3 points 2 weeks ago (1 children)

Well then you guys will love what this guy (by tha name "icitry") did with bash https://www.youtube.com/watch?v=b_WGoPaNPMY

He created a youtube clone with Bash

[–] [email protected] 3 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

That is definitely not something I would do… for work (totally not implying that I miiiight do it outside of work for shits and giggles :P).

I didn’t create this post trying to be like “y’all should just use Bash”, nor is it an attempt to say that I like Bash, but I guess that’s how people boil others down to these days. Fanatics only. Normalcy is dead. (I’m exaggerating ofc)

load more comments (1 replies)
[–] [email protected] 27 points 2 weeks ago

Honestly, if a script grows to more than a few tens of lines I'm off to a different scripting language because I've written enough shell script to know that it's hard to get right.

Shellcheck is great, but what's greater is a language that doesn't have as many gotchas from the get go.

[–] [email protected] 5 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

As I've matured in my career, I write more and more bash. It is absolutely appropriate for production in the right scenarios. Just make sure the people who might have to maintain it in the future won't come knocking down your door with torches and pitchforks...

[–] [email protected] 2 points 2 weeks ago

That’s my take on the use of bash too. If it’s something that people think it’s worth bring their pitchforks out for, then it’s something you should probably not write in bash.

[–] [email protected] 23 points 2 weeks ago (1 children)

I've worked in bash. I've written tools in bash that ended up having a significant lifetime.

Personally, you lost me at

reading from the database

Database drivers exist for a reason. Shelling out to a database cli interface is full of potential pitfalls that don't exist in any language with a programmatic interface to the database. Dealing with query parameterization in bash sounds un-fun and that's table stakes, security-wise.

Same with making web API calls. Error handling in particular is going to require a lot of boilerplate code that you would get mostly for free in languages like Python or Ruby or Go, especially if there's an existing library that wraps the API you want to use in native language constructs.

[–] [email protected] -3 points 2 weeks ago (1 children)

This is almost a strawman argument.

You don’t have to shell out to a db cli. Most of them will gladly take some SQL and spit out some output. Now that output might be in some tabular format with some pretty borders around them that you have to deal with, if you are about the output within your script, but that’s your choice and so deal with it if it’s within your comfort zone to do so. Now if you don’t care about the output and just want it in some file, that’s pretty straightforward, and it’s not too different from just some cli that spits something out and you’ve redirected that output to a file.

I’ve mentioned in another comment where if you need to accept input and use that for your queries, psql is absolutely not the tool to use. If you can’t do it properly in bash and tools, just don’t. That’s fine.

With web API calls, same story really; you may not be all that concerned about the response. Calling a webhook? They’re designed to be a fire and forget, where we’re fine with losing failed connections. Some APIs don’t really follow strict rules with REST, and will gladly include an “ok” as a value in their response to tell you if a request was successful. If knowing that is important to the needs of the program, then, well, there you have it. Otherwise, there are still ways you can get the HTTP code and handle appropriately. If you need to do anything complex with the contents of the response, then you should probably look elsewhere.

My entire post is not to say that “you can do everything in bash and you should”. My point is that there are many cases where bash seems like a good sufficient tool to get that simple job done, and it can do it more easily with less boilerplate than, say, Python or Ruby.

load more comments (1 replies)
[–] [email protected] 3 points 2 weeks ago
[–] [email protected] 9 points 2 weeks ago

Over the last ten - fifteen years, I've written lots of scripts for production in bash. They've all served their purposes (after thorough testing) and not failed. Pretty sure one of my oldest (and biggest) is called temporary_fixes.sh and is still in use today. Another one (admittedly not in production) was partially responsible for getting me my current job, I guess because the interviewers wanted to see what kind of person would solve a coding challenge in bash.

However, I would generally agree that - while bash is good for many things and perhaps even "good enough" - any moderately complex problem is probably better solved using a different language.

[–] [email protected] 2 points 2 weeks ago* (last edited 2 weeks ago) (1 children)

Pretty much all languages are middleware, and most of the original code was shell/bash. All new employees in platform/devops want to immediately push their preferred language, they want java and rust environments. It's a pretty safe bet if they insist on using a specific language; then they don't know how awk or sed. Bash has all the tools you need, but good developers understand you write libraries for functionality that's missing. Modern languages like Python have been widely adopted and has a friendlier onboarding and will save you time though.

Saw this guy's post in another thread, he's strawmanning because of lack of knowledge.

[–] [email protected] 2 points 2 weeks ago (1 children)

Pretty much all languages are middleware, and most of the original code was shell/bash.

What? I genuinely do not know what you mean by this.

[–] [email protected] 1 points 2 weeks ago (4 children)

2 parts:

  • All languages are middleware. Unless you write in assembly, whatever you write isn't directly being executed, they are being run through a compiler and being translated from your "middle language" or into 0s and 1s the computer can understand. Middleware is code used in between libraries to duplicate their functionality.
    https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-is-middleware/
  • Most original code was written in shell. Most scripting is done in the cli or shell language and stored as a script.shfile, containing instructions to execute tasks. Before python was invented you used the basic shell because nothing else existed yet
load more comments (4 replies)
[–] [email protected] 24 points 2 weeks ago (8 children)

I'm afraid your colleagues are completely right and you are wrong, but it sounds like you genuinely are curious so I'll try to answer.

I think the fundamental thing you're forgetting is robustness. Yes Bash is convenient for making something that works once, in the same way that duct tape is convenient for fixes that work for a bit. But for production use you want something reliable and robust that is going to work all the time.

I suspect you just haven't used Bash enough to hit some of the many many footguns. Or maybe when you did hit them you thought "oops I made a mistake", rather than "this is dumb; I wouldn't have had this issue in a proper programming language".

The main footguns are:

  1. Quoting. Trust me you've got this wrong even with shellcheck. I have too. That's not a criticism. It's basically impossible to get quoting completely right in any vaguely complex Bash script.
  2. Error handling. Sure you can set -e, but then that breaks pipelines and conditionals, and you end up with really monstrous pipelines full of pipefail noise. It's also extremely easy to forget set -e.
  3. General robustness. Bash silently does the wrong thing a lot.

instead of a import os; os.args[1] in Python, you just do $1

No. If it's missing $1 will silently become an empty string. os.args[1] will throw an error. Much more robust.

Sure, there can be security vulnerability concerns, but you’d still have to deal with the same problems with your Pythons your Rubies etc.

Absolutely not. Python is strongly typed, and even statically typed if you want. Light years ahead of Bash's mess. Quoting is pretty easy to get right in Python.

I actually started keeping a list of bugs at work that were caused directly by people using Bash. I'll dig it out tomorrow and give you some real world examples.

[–] [email protected] 7 points 2 weeks ago (3 children)

Agreed.

Also gtfobins is a great resource in addition to shellcheck to try to make secure scripts.

For instance I felt upon a script like this recently:

#!/bin/bash
# ... some stuff ...
tar -caf archive.tar.bz2 "$@"

Quotes are OK, shellcheck is happy, but, according to gtfobins, you can abuse tar, so running the script like this: ./test.sh /dev/null --checkpoint=1 --checkpoint-action=exec=/bin/sh ends up spawning an interactive shell...

So you can add up binaries insanity on top of bash's mess.

[–] [email protected] 1 points 2 weeks ago

gtfobins

Meh, most in that list are just "if it has the SUID bit set, it can be used to break out of your security context".

load more comments (2 replies)
load more comments (7 replies)
[–] [email protected] 7 points 2 weeks ago (1 children)

Wanna check if a variable’s set to something expected? if [[ <test goes here> ]]; then <handle>; fi

Hey, you can't just leave out "test goes here". That's worst part by a long shot.
The rest of the syntax, I will have to look up every time I try to write it, but at least I can mostly guess what it does when reading. The test syntax on the other hand is just impossible to read without looking it up.

I also don't actually know how to look that up for the double brackets, so that's fun. For the single bracket, it took me years to learn that that's actually a command and you can do man [ to view the documentation.

[–] [email protected] 4 points 2 weeks ago (1 children)

To be fair, you don’t always have to use the [[ syntax. I know I don’t, e.g. if I’m just looking for a command that returns 1 or 0, which happens quite a bit if you get to use grep.

That said, man test is my friend.

But I’ve also gotten so used to using it that I remember -z and -n by heart :P

[–] [email protected] 5 points 2 weeks ago (1 children)

If you need to use bash a lot just to learn 2 "keywords", then it's not a good language.

I have looked at bash scripts in the past, and even written some (small amount). I had to look up -z and -n every time. I've written a lot more python than bash, that's for sure. But even if I don't write python for a year, when needed I can just write an entire python script without minimal doc lookups. I just need to search if the function I want is part of syd, os or path.

The first time I want to do an else if my IDE will mark it red and I'll write eliffrom then on, same thing if I try to use { }.

If a bash script requires at least one array and one if statement, I can write the entire thing in python faster than I can search how to do those 2 things in bash.

[–] [email protected] 1 points 2 weeks ago

To each their own really. You have what you’re familiar with, and I have mine. That said, I’m not proposing Bash as a good language. It is by no means that.

Now, to use Python for comparison. With a year of not using it, I’d be asking lots of questions. How do I mkdir? How do I mkdir -p? What about cp or mv and their flags? Did I use to bring in some library to make this less painful?

Cause look, I already use many of these commands in the terminal, basically all the time cause I work in it.

Fwiw, there’s a bash-language-server that can warn you of some syntactical errors.

load more comments
view more: ‹ prev next ›