Using Basic Vs. Extended Regular Expressions In Sed

ByThe Aha Unix Team

Definition and Background of Regular Expressions

A regular expression (regex or regexp for short) is a sequence of characters that defines a search pattern. Regexes provide a concise and flexible means for matching strings of text, such as particular characters, words, or patterns of characters. They are supported by many text editors, programming languages, and utilities such as sed and grep.

Sed is a popular Unix stream editor that filters and transforms text using regexes. The sed utility allows you to perform text manipulation including finding and replacing strings, deletion, insertion, and other types of text filtering. Understanding regular expressions is key to unlocking the power of sed.

There are two levels of regular expression support in sed – basic regular expressions (BRE) and extended regular expressions (ERE). BREs provide a simpler set of metacharacters and regex features, while EREs offer additional metacharacters and capabilities.

Basic Regular Expressions in Sed

Basic regular expressions supported in sed provide anchoring, grouping, quantification, alternation, and character class matching capabilities despite using a limited metacharacter set. Some examples of BRE metacharacters are:

  • . – Matches any single character
  • * – Matches 0 or more repetitions of the preceding regex
  • ^ – Matches the start of the line
  • $ – Matches the end of the line
  • [ ] – Defines a character class to match

Common Metacharacters

The dot metacharacter . is one of the most useful, matching any single character. For example, the regex s.d would match “sad”, “sod”, “said”, or any other three letter string starting with “s” and ending with “d” with a single character in between.

The asterisk metacharacter * matches zero or more occurrences of the preceding character or regex. For example, s.*d would match “sd”, “sxyzd”, “spqrxyzd” or any string containing “s”, then zero or more other characters, followed by “d”.

Character Classes and Ranges

Square brackets define character classes in basic regular expressions, allowing you to match any one out of a set of characters. For example, [abc] would match “a”, “b”, or “c”. Ranges can also be specified using a hyphen, like [a-z] to match any lowercase letter.

Anchors and Assertions

The caret ^ and dollar sign $ anchors allow matching the beginning and end of lines respectively. Using them allows you to write regexes like ^hello to match lines starting with “hello”, or world$ to match lines ending in “world”.

Grouping and Capturing

Basic regular expressions allow grouping parts of a regex together using parentheses “()”. This also captures the matched text for later use. Groups can be quantified with metacharacters like * and +. For example, (ab)* would match 0 or more repetitions of “ab”.

Extended Regular Expressions in Sed

In addition to the metacharacters available for basic regular expressions, sed also supports extended regular expressions (EREs) which provide additional metacharacters and capabilities. These include:

  • + – Matches 1 or more repetitions of the preceding regex
  • ? – Makes the preceding regex optional
  • | – Allows alternation between regexes
  • () – Groups regexes and captures text
  • {} – Specifies numeric repetition qualifiers

Additional Metacharacters

The plus metacharacter + works likes * but requires at least one occurrence of the preceding regex to match. Using a question mark ? after a regex allows it to optionally appear. Alternation with the vertical bar | matches the regex before or after.

As in BREs, parentheses () group parts of a regex together. But in EREs parentheses also create a numbered capturing group for retrieving the matched text later for reuse. Capturing groups can be nested and quantified.

Non-Greedy Matching

By default repetition metacharacters like * and + match as much text as possible. Adding a question mark after them activates non-greedy mode, matching the minimum amount needed rather than the maximum.

Named Capture Groups

Instead of accessing captured text from parentheses groups by numerical index, extended regexes support using named groups with angle bracket syntax like (?<name>…). This allows giving meaningful names to parts of a complex regex.

When to Use Basic vs. Extended Regexes

While extended regular expressions provide more powerful matching capabilities, basic regexes have certain advantages relating to performance, compatibility, and learning curve.

Performance and Compatibility Considerations

Since BREs have a simpler and more limited feature set, regular expression engines can evaluate them faster than more complex EREs. Older versions of tools like sed may not fully support ERE syntax. But modern sed versions handle EREs performantly.

Feature Differences and Limitations

Basic regexes lack greedy/non-greedy matching, named groups, embedded flag expressions, leftmost longest assertions, and other ERE features. Complex regexes needing these should use EREs. But BREs suffice for simpler use cases.

Examples and Common Use Cases

Regular expressions are commonly used with sed for tasks like finding and replacing text, input validation, and extracting information from files. Sed provides the “s” substitute command for regex based search and replace.

Find and Replace Text

This sed command uses an extended regex to replace text:

sed 's/apples(, bananas)*, (cherries|grapes)/fruit/' file

It would replace phrases like “apples”, “apples, bananas, cherries”, “apples, bananas, bananas, grapes” with just “fruit”.

Validate Input

Sed can validate input files line by line using BREs:

sed '/^[0-9]{5}(-[0-9]{4})?$/!d' file

That would delete all lines not containing either a 5-digit number or a 5-digit number followed by a hyphen and 4 more digits.

Extract Data from Files

Capture groups in EREs extract matching text for reuse:

sed -n 's/^Item: \([^,]*\), Price: \([^\$]*\).*$/Name: \1, Value: \2/p' file

This sed script would capture and print just the item name and price from lines with an “Item: ” and “Price: ” format.

Best Practices

Follow these regex and sed best practices in your text manipulation workflows for improved results:

Legibility and Documentation

  • Break long sed commands into multiple steps/lines for legibility
  • Add ample comments explaining complex regexes and script steps
  • Name captured groups meaningfully

Testing and Debugging

  • Use sed options like -n to disable default output and test regex matches
  • Print captured groups and current line with the “p” command to test capture syntax
  • Escape metacharacters to match them literally when needed

Performance Tuning

  • Prefer basic regexes over extended ones for faster evaluation
  • Avoid slow constructs like backreferences, excessive backtracking
  • Balance regex precision vs. simplicity

Additional Resources

For further learning about using regular expressions effectively with sed, refer to these resources:

Leave a Reply

Your email address will not be published. Required fields are marked *