Escaping Special Characters When Using Variables In Sed Substitutions
Why Escaping Characters Matters in sed
The sed text processing utility interprets certain characters in a special way during its substitution commands. Characters like the forward slash (/), backslash (\), and dollar sign ($) have predefined functions in the syntax of sed substitution, so they need to be escaped if you want them to be treated as literal text characters. Failure to properly escape these special characters leads to errors or unexpected behavior when sed attempts to parse and execute substitutions.
For example, the forward slash character is used to delimit the components of a sed substitute command. Not escaping slashes that appear within the search or replace components will cause syntax errors:
sed 's/special\/character/replaced/' # Fails with syntax error
Likewise, the dollar sign indicates the end of a line in sed regular expressions. Unescaped dollar signs may lead sed to unexpectedly match end-of-line instead of a literal $ character:
sed 's/$5.00/$$$5.00/' file # Replaces EOL plus "5.00"
The backslash also has a predefined meaning in sed, indicating that the following character should be treated literally instead of as a special character. So unescaped backslashes can alter the meaning of subsequent characters unexpectedly.
Using Backslashes to Escape Special Characters
To treat these special characters literally, prepend them with a backslash (\). The backslash tells sed to interpret the next character in a literal sense instead of giving it any special meaning or function.
Here are some examples of using backslashes to escape problematic characters correctly in sed:
sed 's/special\/character/replaced/' # Escaped slash treated literally sed 's/\$5\.00/$$$5.00/' file # Escaped $ matches literal $
Note that the backslash itself is also a special character in certain contexts. To insert a literal backslash into a regular expression or replacement string, you need to escape it as well, with a double backslash:
sed 's/\\//\\\\/' # Replaces single backslash with double backslash
Escaping Variables That Contain Special Characters
Sed allows you to use variables within substitute commands, via parameter expansion syntax like $myvar or ${myvar}. This can be useful for dynamically generating sed scripts. However, if the variable value contains any unescaped special characters, trying to reference the variable will fail or behave unexpectedly.
For example, this attempt to use a variable value containing slashes will error out:
myvar="/special/value/" sed 's~$myvar~REPLACED~' file # Fails due to unescaped slashes
To work properly, the entire variable reference needs to be escaped with a backslash. This escapes the whole expansion as a single literal string value:
myvar="/special/value/" sed 's~\$myvar~REPLACED~' file # Variable escapes slashes correctly
The same principle applies when a variable value contains dollar signs, backslashes, or any other special character – escape the entire variable reference to treat its value literally during substitution.
When to Double Escape Backslashes
There is an extra twist regarding backslashes in variable definitions. If your variable needs to contain literal backslashes (e.g. a Windows file path), those need to be double-escaped when defining the variable, before passing to sed:
myvar="C:\\path\\to\\file" # Double escaped when defining sed 's~$myvar~LINUX_PATH~' file # Var value passed to sed
This is because the backslash already has special meaning when defining a variable value – it escapes the next character in the value. By doubling the backslashes, the first backslash escapes the second, resulting in a literal backslash stored in the variable’s value. When this variable value gets expanded in sed, a single literal backslash gets passed through.
Safe Alternatives to Escaping Everything
Although escaping variable references fixes problems with built-in special characters, it can get verbose and challenging to maintain. Some safer alternatives exist:
- Use different delimiter characters for sed s commands – For example, use ~, #, or | instead of / slashes.
- Place variable references outside the regular expression or replacement component of an s command.
- Avoid variables altogether and define full strings literally where possible.
While direct escaping allows dynamic substitutions, moving variables out of the search/replace components reduces escaping complexity and improves maintainability.
Summary
Special characters take on predefined meanings and functions within the syntax of sed substitution commands. To treat them as literal characters instead, they need to be escaped with backslashes:
sed 's/special\/character/replaced/'
If using variables in substitutions, any special chars in the variable’s value also need escaping. Escape the entire variable reference to treat its value literally:
myvar="$special"; sed 's~\$myvar~literal~'
Furthermore, backslashes in variable values require double escaping when defining the variable, in order to pass single literal backslashes to sed:
myvar="\\"; # Double escaped when defining sed 's~$myvar~/~'
Alternatives to heavy escaping include using custom delimiter characters, moving variable references out of the search/replace components, or avoiding variables altogether.
