Bypassing Stdout Buffering With Unbuffer And Stdbuf
Stdout buffering refers to the storing of standard output text in a temporary memory buffer before sending it to the terminal. Shells like Bash default to block buffering stdout, which means text is collected in 4096 byte chunks before being displayed. Buffer flushing occurs during newline writes or when the buffer fills up.
This buffering causes issues when expecting real-time command output. Since text waits in the buffer, terminal updates are delayed and output seems unresponsive. Mixing buffered stdout with unbuffered stderr also presents problems. The lack of synchronization leads to confusing interleaving of messages.
Definition of stdout buffering
Stdout buffering is controlled by settings in the C standard library. The setvbuf() function configures buffering mode and size for file handles like stdout. In block buffering mode, 4096 bytes are collected before flushing whereas line buffering sends text on newline writes. Unbuffered immediately displays each write.
Situations where buffering causes problems
The delayed and grouped output from buffering causes several issues:
- Output seems sluggish since text waits in the buffer
- Status messages fail to update in real-time
- Unbuffered stderr interleaves with buffered stdout
- Script interaction with the user is hindered
Delayed output to terminal
Since block buffering delays writes to the terminal, output seems unresponsive. Commands will sit silently despite printing text due to the text waiting in memory. This hinders scripts that need to interact with or update the user.
Mixing stderr and buffered stdout
Stderr tends to remain unbuffered while stdout uses block buffering. This discrepancy leads to messy output since the different streams have independent flushes. Status text on stderr may interleave oddly with command output sent to stdout.
Disabling Buffer for Real-Time Output
The most straightforward solution is fully disabling buffering with utilities like unbuffer or stdbuf. The unbuffer command runs a process with unbuffered stdout and stderr streams. This provides completely synchronized and immediate output at the cost of higher CPU usage.
Introducing unbuffer command
The unbuffer program comes from the expect package and disables stream buffering for a passed command. It internally uses the low-level setbuf() library call to configure unbuffered stdout and stderr handles before exec() runs the process.
Syntax and examples of unbuffer
Basic syntax:
unbuffer command [arguments]
Print size of current directory without buffer delays:
unbuffer du -sh .
Monitor log file content changes immediately:
unbuffer tail -f /var/log/messages
Unbuffering common terminal output commands
Unbuffer is especially useful on interactive scripts relying on real-time output. Some common cases include:
- Progress bars and status updates
- Interactive installation wizards
- Real-time log monitoring
- Cursor-based animation
Controlling Buffer Size with stdbuf
For less extreme cases, the stdbuf command simply modifies buffer settings instead of fully disabling. This allows customizing buffer size or switching to line buffering for responsive output without the CPU tradeoffs of being completely unbuffered.
What stdbuf does
The stdbuf utility leverages setvbuf() to alter the stdout and stderr buffer modes before launching a process. Available buffer modes are:
- block: 4096 byte chunks (default)
- line: Flush on newline character
- unbuffered: No buffering
Syntax and examples of stdbuf
Basic syntax:
stdbuf [-i|-o|-e] [buffer_mode] command
Set unbuffered stdout on interactive ping:
stdbuf -o0 ping 8.8.8.8
Use line buffering for a tail -f:
stdbuf -oeL tail -f /var/log/dmesg
Setting smaller buffer sizes for responsiveness
By switching from block to line buffering, output will flush on newline writes instead of filling 4096 byte buffers. This provides improved responsiveness for monitoring logs, status updates, and other interactive output.
Achieving Line Buffering with Python
The Python print() function similarly offers control over buffering choices. Toggling between line and block buffering can improve script output responsiveness.
Buffer modes in Python print function
The print() buffer modes:
- -1: Line buffered stdout
- 0: Unbuffered output
- 1: Block buffer stdout (default)
Switching to line buffering from block buffering
Line buffering via the flush parameter:
import sys sys.stdout = io.TextIOWrapper(buffer=sys.stdout, flush=-1)
Example code for line buffering stdout
import time
import sys
import io
sys.stdout = io.TextIOWrapper(buffer=sys.stdout, flush=-1)
for x in range(10):
print("Line %d" % x)
time.sleep(1)
This loops prints a line number every second, flushing on newlines instead of buffering.
Conclusions
Stdout buffering causes issues for scripts needing interactivity and real-time output. Utilities like unbuffer and stdbuf offer control over disabling buffering or setting sane buffer sizes. For quick fixes, try unbuffering interactive pipes. Switching Python and other programs to line buffering also improves responsiveness.
Summary of buffering behaviors
- Block: 4096 bytes before flush
- Line: Flush on newline
- Unbuffered: Real-time flush each write
Recommendations for responsive output
- Disable buffering entirely with unbuffer when possible
- Or set line buffering with stdbuf or Python print() options
- Watch CPU usage since unbuffered IO has overhead
