Demystifying Regular Expression Syntax Differences Across Linux/Unix Tools

A regular expression (regex or regexp for short) is a sequence of characters that defines a search pattern. Regexes provide a concise and flexible means for matching strings of text, such as particular characters, words, or patterns of characters. They are supported by many text editors and programming languages to find and manipulate text based on the defined patterns.

Some common uses of regular expressions include:

• Matching text – Finding text that matches a specified pattern. For example, finding all phone numbers or email addresses in a document.
• Search and replace – Matching text to replace with another string. For example, changing all American date formats in a document to European date formats.
• Input validation – Checking if user input matches an expected format. For example, validating if an email address or password is correctly formatted before submitting a form.
• Parsing text – Extracting relevant data from logs, documents or textual output. For example, analyzing web server log files to identify page request details.
• Formatting text – Performing advanced text manipulation and formatting to normalize or transform textual data.

The syntax for writing regular expressions varies across languages and tools, but generally they comprise of a combination of normal characters and special meta-characters that denote rules for the patterns you want to match.

Regex Engines Vary By Tool

The software responsible for handling and interpreting regular expressions behind the scenes is called the regular expression engine. There are actually many different regex engines in use across various programming languages, text editors, and other tools that support regular expressions.

Some of the popular regex engines include:

• PCRE (Perl Compatible Regular Expressions) – The PCRE library is one of the most common regex engines. It was initially developed for Perl but has been ported to many other languages and tools like PHP, Apache, Nginx and more.
• Oniguruma – A regex library used as the default engine for Ruby as well as tools like TextMate, Sublime Text, VS Code, etc.
• .NET Framework Regex – The built-in .NET regex engine for .NET languages like C#, VB.NET, ASP.NET.
• Python re Module – The regex engine for the Python programming language.
• GNU Regex – The default regex library for many Linux/Unix tools like grep, sed, awk, vim, Emacs etc.
• Java Regex – The builtin engine for Java’s java.util.regex API.
• ECMAScript Regex – The standard Javascript regex engine largely based on Perl regex syntax and features.

The differences in the regex engines powering various tools and languages can lead to syntactical variations and limitations:

• Not all features like lookaround assertions or named capture groups are uniformly supported.
• Meta-character syntax sometimes varies like using \d vs [0-9] for digits.
• Matching rules might differ such as greediness in quantifiers being tool-specific.
• Character class definitions and shorthand formats can differ like \w or [:alpha:] across regex flavors.

Understanding what engine your tool uses and the nuances it brings is important for writing robust and portable regular expressions.

Core Syntax Differences

While all regex engines share a lot of common syntax, they also have their differences. Some of the core areas where regex syntax varies the most across different tools and programming languages include:

Character Classes and Ranges

The syntax for defining a set or range of characters to match can vary across regex engines. For example:

[0-9]    - Digits from 0 to 9
\d       - Shorthand for digit character (differs across tools)
[a-zA-Z]- Alphabet characters
\w       - Shorthand for "word" char (meaning differs)

Always check your tool’s documentation for supported character class shorthand and ensured cross-engine compatibility if required.

Quantifiers and Repetition

The syntax used to denote how many times a pattern can repeat continuously also sometimes varies. Examples include:

?        - 0 or 1 matches
*        - 0 or more matches 
+        - 1 or more match
{2}      - Exactly 2 matches
{2,4}    - 2 to 4 matches

Greediness and laziness of quantifiers like * and + can also differ across engines.

Anchors and Assertions

Anchors like ^ and $ to match line starts and ends are quite standard, but positive/negative lookaheads/lookbehinds can differ:

^        - Start of line/string
$        - End of line/string

(?=)     - Positive lookahead 
(?!)     - Negative lookahead
(?<=)    - Positive lookbehind
(?
Many engines only support a subset or variant of these assertions.
Groupings and Subexpressions
Bracket expressions for grouping parts of a regex pattern into subexpressions is quite standard with ( and ) metacharacters. But capturing vs non-capturing groups and accessing matched subgroups differs.
For example: 
(regex)       - Capturing group
(?:regex)     - Non-capturing group
\1            - Backreference to group #1
${groupName}  - Named capturing groups 

These additional group features are not available in all regex engines.
Working Examples
To demonstrate some real-world regex usage across tools, here are examples of patterns defined for common use cases:
Matching Phone Numbers
A regex to match US/European phone number formats:
/^(\+[0-9]{1,3}[- ]?)?[0-9]{10,12}$/

Breakdown:

^ - Start of string
(\+[0-9]{1,3}[- ]?) - Optional country code
[0-9]{10,12} - 10-12 digits for number
$ - End of string

Validating Email Addresses
A more robust regex for matching most valid email addresses:
  
/^([a-zA-Z0-9]+)([\._-]{0,1})([a-zA-Z0-9]+)@([a-zA-Z0-9]+)\.([a-zA-Z]{2,3})(\.[a-zA-Z]{2,3})?$/

Key elements:

Username section
@ symbol separator 
Domain name section
Top level domain checking

Parsing Log Files
A common log format regex for parsing web/app log files:
/^(\S+) (\S+) (\S+) \[([^:]+):(\d+):(\d+) ([^\]]+)\] (.*)$/ 

Breakdown of captured groups:

IP Address
Client Identity
User ID
Date
Time
Time Zone
Request Details

The groups can then be processed individually after extracting them with the overall regex match.
Recommendations
When writing regexes for use across different tools and programming languages, here are some top things to keep in mind:
Feature Checklists
Consult engine/tool capability checklists before using more advanced regex features:

PCRE regex cheatsheets
Python re module support list
JavaScript regex compatibility tables

Cross-Engine Compatibility
For regex portability, avoid engine-specific constructs:

Use simple character classes like \d instead of [:digit:]
Avoid advanced assertions like lookbehinds if unsupported
Prefer non-capturing groups over capturing groups

When to Use Raw POSIX Regex
Some tools like grep, sed, awk support "POSIX basic" and "POSIX extended" modes for more standard regex features across platforms.