Demystifying The Shebang: How Linux Executes Scripts
What is the Shebang?
The shebang refers to the two characters “#!” that start interpreter directive lines in Linux and other Unix-like operating systems. Acting as a marker, the shebang indicates to the system which interpreter should be used to execute the script that follows.
Definition of the shebang
The term “shebang” is derived from the concatenation of the two characters – sharp (hash, pound, number sign) and exclamation mark – that start these directive lines. Some say it is written as “sha-bang” to reflect its pronunciation. While often called the “hashbang”, shebang relates more closely to its functional purpose.
In Linux and other Unix-like systems, executable files are not required to have filename extensions indicating their type. The shebang provides the clue that a script text file contains code that needs to be executed by a particular interpreter before it can run.
Explaining the #! syntax
A shebang line consists of the two characters “#!”, followed by the full path to the interpreter that should execute the code that follows in the script. Some examples showing typical shebang lines:
- #!/bin/bash
- #!/usr/bin/python3
- #!/usr/bin/perl
- #!/usr/bin/node
Spaces between the characters are allowed by convention, though not commonly seen. Whitespace also often follows the interpreter path to improve readability. The path tells the system where to find the correct interpreter binary executable file on the file system to parse and run the script code.
Examples of common shebang lines
The most ubiquitous shebang line calls on the Bash shell interpreter for executing shell scripts:
#!/bin/bash
For Python scripts, if multiple versions are installed, the shebang can specify whether to use Python 2 or Python 3 (if both available on the system):
#!/usr/bin/python3
Or for a specific minor version like 3.8:
#!/usr/bin/python3.8
Perl, Ruby and Node.js scripts would have similar shebangs for their respective interpreters:
#!/usr/bin/perl #!/usr/bin/ruby #!/usr/bin/node
How the Shebang Works
When a script with a shebang is executed directly on the command line or clicked to launch in a graphical interface, the operating system kernel starts by analyzing the shebang to determine how to handle the file.
Describing the role of the kernel
The Linux kernel examines the start of executable text files, looking for those initial “#!’ characters. Upon seeing the shebang sequence, the kernel references the specified absolute path to locate the correct interpreter binary executable.
It then prepares to execute that interpreter, passing it as the argument the full path of the script file to be parsed along with any additional options or arguments. This enables the appropriate runtime engine to take over reading through and acting upon the code that follows in the script.
Walking through the steps to execute a script
When running a script with a shebang, Linux carries out these key steps under the hood:
- The script is invoked from shell with its filepath as an argument
- The kernel opens the file and inspects the initial bytes for a shebang
- The specified interpreter path is referenced to locate its executable
- A new process is spawned running that interpreter binary
- The script filepath gets passed as an argument to the launched process
- The interpreter parses line-by-line, executing the script’s contents
By handling scripts this way, the shebang syntax provides portability. Users do not need to manually invoke interpreters and pass script filepaths as arguments.
Illustrating the process with a diagram
Below is a visual overview of what happens when a script with a shebang gets called to run:
+-----------------------+
| |
| Script invoked from |
| shell with filepath |
+--------> |
| +-----------------------+
|
| +-----------------------+ +----------------------+
| | | | |
| | Kernel inspects start | | Interpreter binary |
| | of script for |--------->| executable file |
| | shebang & interpreter | | /usr/bin/python3 |
+--------> | +----------------------+
+-----------------------+
+-----------------------+ +----------------------+
| | | |
| Kernel prepares to | | Interpreter runtime |
| launch interpreter |----------> | started as new |
| process | | system process |
+--------> | +----------------------+
| +-----------------------+
|
| +----------------------+
| | |
| | Interpreter receives |
| | script filepath and |
| | processes contents |
+--------> |
| line-by-line |
+----------------------+
Customizing Your Shebang
The shebang can be augmented in useful ways to modify script execution behaviors. Some common customizations include specifying interpreter options, handling multiple interpreters within the same file, and using alternate forms of the shebang syntax.
Useful alternate shebang lines
In addition to absolute paths, these shortcut shebang formats are sometimes utilized:
#!/bin/sh #!/usr/bin/env python3
The /bin/sh version will call on the default system shell, while /usr/bin/env runs whichever version of python3 is first defined in the current user’s PATH environment variable.
Specifying interpreter options
Flags and parameters can be appended to the interpreter path in the shebang line, enclosed in single or double quotes. For example, enabling Python verbose mode and warning output on script errors:
#!/usr/bin/python -v -W error
Or passing a Bash debug mode toggle:
#!/bin/bash -x
Handling multiple interpreters
The kernel only looks at the very first shebang in a file. To leverage multiple scripting languages within the same file, the kernel will execute the first specified interpreter, which can then parse and switch between other contained languages.
For instance, a Python script can execute embedded Bash with:
#!/usr/bin/python3
import os
import subprocess
# Execute inline Bash script code
subprocess.call("#!/bin/bash -l\necho Hello from Bash", shell=True)
This allows crosstalk between languages, though it’s often best to separate code into distinct files organized by language instead.
Troubleshooting Shebang Issues
Sometimes scripts with shebangs don’t execute as expected. Debugging tips can help identify and resolve common problems encountered.
Debugging common problems
If nothing happens or there are permission errors when attempting to run a script with a shebang line, possible issues include:
- No read permission for the current user to access the script
- The interpreter path defined doesn’t exist or resolves to a non-executable
- The interpreter binary isn’t marked executable
- An unsupported or invalid option was passed to the interpreter in the shebang
Syntax issues from code bugs can also lead to runtime crashes or logic errors during execution.
Fixing permissions errors
A common pitfall is forgetting to make scripts readable and executable before trying to run them. The following command adds those needed permissions:
chmod u+rx myscript.sh
If other users should also be able to execute the script, the permissions can be opened up accordingly:
chmod ug+rx,o+r myscript.sh
Changing the interpreter path
If the interpreter is available but fails to run from the defined path, double check the accuracy of the shebang. The which or whereis commands help find the correct full executable path:
which python3 whereis bash
Once the proper interpreter path is identified, edit the shebang line in the script appropriately. Ensure the script file permissions allow overwrite before resaving changes.
Use Cases and Best Practices
Understanding conventions for constructing shebang lines helps ensure scripts are portable, redistributable, and compatible across systems.
Recommended shebangs for shell, Python, Perl etc.
For consistent behavior across Linux distributions, these interpreters in base system directories are good defaults:
#!/bin/sh #!/usr/bin/env python3 #!/usr/bin/perl #!/usr/bin/awk
Whereas these shebangs more explicitly call out specific software versions:
#!/bin/bash #!/usr/bin/python3.6 #!/usr/bin/ruby2.7
Making scripts portable and redistributable
Hardcoding absolute interpreter paths can lead to breakages when scripts are copied between systems. Using env and standard system binary locations improves compatibility:
#!/usr/bin/env python3
Installing required interpreters separately alongside scripts or bundling them into virtual environments also enhances transportability.
Maximizing compatibility across systems
Not all Unix-like operating systems implement the shebang process in precisely the same fashion. Variations in handling shebangs exist across Linux flavors, BSD, Solaris, MacOS, etc.
For best results when sharing or publishing scripts:
- Stick to common interpreters like sh, bash, perl, python(3)
- Reference them in standard base directories whenever possible
- Set permissive file permissions allowing public read/execute
Testing scripts across different target platforms is also advised to catch any unexpected quirks with shebang processing.
