Right Utility Coding Language for the Job
Python is horrible for most things that can be done from the command line. It's just a fact. Yet novice and veteran coders alike continue to pick this wrong tool for the job. Unless you require complex number crunching, data science libraries, remote task synchronization, quick-n-dirty objects and data structures, or a built-in SQL DB you are probably using the wrong tool for the job if you pick Python. (Cue fan-boy flames.) Instead, use the shell itself or a modern compiled language.
When it comes to utility coding from and for things from the command line nothing beats the following—in order of scale and scope:
- POSIX Shell
A shell interpreter is already running any time you use the command line so functions and aliases in your
.bashrc will be orders of magnitude faster to write and execute. You really do not care about variable scope, strict typing, and objects for this kind of stuff. You want to write and execute stuff really quickly without much extra thought. Waiting around those extra milliseconds for a Python or other interpreter to fire up is really annoying over time and even worst when you combine functions together. If each of those functions were a short Python or Perl or Ruby script you would be waiting for the interpreter to fire up and compile for every one. This is ridiculously inefficient compared to simple shell functions or compiled code.
Occasionally you need to really do some quick magic from the command line that exceeds the capability of shell—particularly anything that needs a lot of file altering and parsing. Often for tools your have nothing but
vim) and a terminal to code it and you certainly have no time to install an interpreter (if the change-management gods even allowed it). This is where Perl shines. It was made for this stuff.
Many programmers don't realize any time you execute
tr that you are firing up a separate process for each. What's worse is that all of these have different ways of writing regular expressions. This was enough to convince Larry Wall to create
perl in the first place, to replace them all with one best-of-breed tool. Larry can be proud that his creation still supremely dominates for this specific need.
💢 I must confess I have come to immediately judge anyone who uses a long pipeline of these smaller commands when one
perlbacktick would have done the job. Definitely a sign of a mediocre sysadmin. I'm not being toxic. I'm being scientific. It is objectively better to use
perlfor such things (no matter what the noobs on StackExchange say).
Perl is already on everything, that is to say, any UNIX/Linux system created within the last 25 years. Python is not. Perl defined the standard for regular expressions and Python is even including the original C library created for Perl. Perl is king of regx, period.
Perl has the added advantage of having a very simple single-line execution making it ideal for those
$( perl -e ...) (previously back ticks, yeah I'm that old) in your shell profile and rc scripts. In fact, it is very common for perl code to be embedded in such scripts the same way
awk might be.
⚠️ Young, immature programmers often equate Perl with old-foggies and make jokes about it being "read only". While it is true Perl was the dominate language of the Web for more than a decade, that was never its original usage. Perl continues to dominate at its original primary usage, to replace and unify all the other small shell languages. Anyone who does an actual comparison of using regular expressions in all of the languages must simply admit Perl is objectively the clearest and cleanest of them all. Perl got a bad rep because programmers loved it so much they pushed it beyond its original scope trying to make it into a systems programming tool, which Python and Go were conceived to be from the beginning.
Anything else that you might run from the command-line that requires more size and structure should be compiled and written in a strictly-typed language. This is even more important if you are required to distribute your tool to other users and systems. You never want to force a user or system administrator to install a language interpreter just to use your tool.
Rob Pike and others joke that Go was conceived while waiting several hours for C++ to finish compiling at Google. It is the first modern compiled language and enjoys the best reputation and widest adoption. Docker, Kubernetes, and many other core systems were created in it. There are a number of other languages for this kind of job (Rust, Julia, Crystal), none is as dominant and pragmatic as Go nor likely to ever catch up with Go's momentum and awesomeness.
Python makes great glue, some call it duct tape. When it comes to running scripts, backups, pen testing, reports and other tasks on multiple machines at the same time, or scripting a multi-hop connection to a remote system Python really dominates. Celery (and Fabric) before simplify the asynchronous connection and task execution and wait around to produce output that can be monitored. This sort of multiple-system-scripting-fu really demands Python. You can even use Python to deliver compiled Go utilities to be run on the targeted systems. Ruby with Chef and Puppet was a thing for a while, but Python has proven itself on this front—particularly because there is even an SQL database build into the language itself allowing all that collected date to temporarily be stored in a very elegant way for transfer to other systems.
🤔 It doesn't hurt that Python is also the leading data science language, but utility tasks are in an entirely different category from that work (and require a lot of other learning to master).
Don't Forget About Change Management
Novice system developers frequently fall into the trap of thinking anything that is needed on a system to run their little tool can simply be installed by the user or admin when needed. This is a ridiculously naive notion. While it is true that some organizations are moving toward continuous integration to speed up the process of tool and application deployment, most still have strict change management in place that require submitting a change request, getting it approved, and then scheduling a window for its deployment. Few will go to the trouble just to install or update an interpreter just to run your code—especially since there is simply no need now that Go and other compiled languages exist for this exact need.