How to Monitor and Tune System Performance in Linux - DevOps Prerequisite 5
System Monitoring and Performance Tuning in Linux
System monitoring and performance tuning are essential tasks for ensuring that your Linux environment runs efficiently and effectively. This article will cover a range of tools and techniques for monitoring system performance and tuning various aspects of a Linux system. We will delve into CPU, memory, disk I/O, and network monitoring, as well as provide strategies for optimizing system performance.
Table of Contents
Introduction to System Monitoring and Performance Tuning
Monitoring Tools
top
htop
vmstat
iostat
dstat
netstat and ss
iftop
CPU Monitoring and Tuning
Monitoring CPU Usage
Tuning CPU Performance
Memory Monitoring and Tuning
Monitoring Memory Usage
Tuning Memory Performance
Disk I/O Monitoring and Tuning
Monitoring Disk I/O
Tuning Disk Performance
Network Monitoring and Tuning
Monitoring Network Traffic
Tuning Network Performance
Best Practices for System Monitoring and Performance Tuning
Conclusion
1. Introduction to System Monitoring and Performance Tuning
System monitoring involves continuously checking various system metrics to ensure that your system is running smoothly. Performance tuning involves making adjustments to system parameters and configurations to improve performance. Effective system monitoring and performance tuning can help prevent bottlenecks, reduce downtime, and ensure optimal resource utilization.
2. Monitoring Tools
Linux offers a variety of tools for monitoring system performance. Here are some of the most commonly used tools:
top
The top
command provides a dynamic real-time view of the system's processes, showing CPU and memory usage.
top
htop
htop
is an enhanced version of top
, providing a more user-friendly interface and additional features such as mouse support and visual indicators.
sudo apt install htop
htop
vmstat
vmstat
reports information about processes, memory, paging, block I/O, traps, and CPU activity.
vmstat 2
This command updates every 2 seconds.
iostat
iostat
provides statistics on CPU and I/O usage.
sudo apt install sysstat
iostat -xz 2
This command shows extended statistics (-x) with device utilization (-z) every 2 seconds.
dstat
dstat
combines the functionality of vmstat
, iostat
, netstat
, and ifstat
.
sudo apt install dstat
dstat
netstat and ss
netstat
and ss
are used for network statistics.
netstat -tuln
ss -tuln
iftop
iftop
displays bandwidth usage on an interface by host.
sudo apt install iftop
sudo iftop -i eth0
3. CPU Monitoring and Tuning
Monitoring CPU Usage
Use top
, htop
, vmstat
, and iostat
to monitor CPU usage.
top -d 1
The -d
option sets the delay between updates to 1 second.
Tuning CPU Performance
Adjust CPU Scheduling: Use
chrt
to set real-time scheduling policies.sudo chrt -f -p 99 $(pgrep your_process)
Set CPU Affinity: Use
taskset
to bind processes to specific CPUs.sudo taskset -c 0,1 your_process
Enable/Disable Hyper-Threading: Modify the BIOS/UEFI settings to enable or disable hyper-threading.
4. Memory Monitoring and Tuning
Monitoring Memory Usage
Use free
, vmstat
, and htop
to monitor memory usage.
free -h
The -h
option displays the output in human-readable format.
Tuning Memory Performance
Adjust Swappiness: The
swappiness
parameter controls the tendency of the kernel to move processes out of physical memory and onto the swap disk.sudo sysctl vm.swappiness=10
Cache Pressure: The
vfs_cache_pressure
parameter controls the tendency of the kernel to reclaim memory used for caching.sudo sysctl vm.vfs_cache_pressure=50
Use HugePages: HugePages can improve performance for applications with large memory requirements.
sudo sysctl vm.nr_hugepages=128
5. Disk I/O Monitoring and Tuning
Monitoring Disk I/O
Use iostat
, dstat
, and iotop
to monitor disk I/O.
sudo iotop -o
The -o
option shows only processes or threads actually doing I/O.
Tuning Disk Performance
Use the Correct I/O Scheduler: The I/O scheduler can be changed using
sysfs
.echo noop | sudo tee /sys/block/sda/queue/scheduler
Enable Write Caching: Write caching can improve performance but at the risk of data loss in case of power failure.
sudo hdparm -W1 /dev/sda
Tune Filesystem Parameters: Mount options such as
noatime
can reduce I/O operations.sudo mount -o remount,noatime /dev/sda1
6. Network Monitoring and Tuning
Monitoring Network Traffic
Use iftop
, netstat
, ss
, and nload
to monitor network traffic.
sudo apt install nload
sudo nload
Tuning Network Performance
Adjust TCP Settings: Tune various TCP parameters using
sysctl
.sudo sysctl -w net.ipv4.tcp_fin_timeout=30 sudo sysctl -w net.ipv4.tcp_window_scaling=1
Use NIC Offloading: Enable or disable NIC offloading features such as TCP segmentation offload (TSO).
sudo ethtool -K eth0 tso on
Optimize Network Buffers: Increase the size of network buffers to handle more data.
sudo sysctl -w net.core.rmem_max=16777216 sudo sysctl -w net.core.wmem_max=16777216
7. Best Practices for System Monitoring and Performance Tuning
Regular Monitoring: Set up regular monitoring to catch issues before they become critical.
Automate Tasks: Use tools like
cron
and monitoring software to automate regular checks and alerts.Document Changes: Keep a log of all tuning changes to understand their impact.
Start Small: Make small, incremental changes and monitor their effects before making further adjustments.
Balance Performance and Stability: Ensure that performance improvements do not compromise system stability.
8. Conclusion
System monitoring and performance tuning are critical skills for maintaining a healthy and efficient Linux environment. By using the right tools and techniques, you can monitor CPU, memory, disk I/O, and network performance, and make informed adjustments to optimize your system. Regular monitoring and tuning not only improve performance but also help in proactive problem detection and resolution.
Mastering these skills will ensure that your Linux systems run smoothly, providing a reliable platform for your applications and services. Whether you're managing a single server or a fleet of machines, effective system monitoring and performance tuning are indispensable for any Linux administrator.
Code Snippets Recap
# top command
top
# htop command
sudo apt install htop
htop
# vmstat command
vmstat 2
# iostat command
sudo apt install sysstat
iostat -xz 2
# dstat command
sudo apt install dstat
dstat
# netstat and ss commands
netstat -tuln
ss -tuln
# iftop command
sudo apt install iftop
sudo iftop -i eth0
# Adjust CPU Scheduling
sudo chrt -f -p 99 $(pgrep your_process)
# Set CPU Affinity
sudo taskset -c 0,1 your_process
# Monitoring Memory Usage
free -h
# Adjust Swappiness
sudo sysctl vm.swappiness=10
# Cache Pressure
sudo sysctl vm.vfs_cache_pressure=50
# Use HugePages
sudo sysctl vm.nr_hugepages=128
# Monitoring Disk I/O
sudo iotop -o
# Use the Correct I/O Scheduler
echo noop | sudo tee /sys/block/sda/queue/scheduler
# Enable Write Caching
sudo hdparm -W1 /dev/sda
# Tune Filesystem Parameters
sudo mount -o remount,noatime /dev/sda1
# Monitoring Network Traffic
sudo apt install nload
sudo nload
# Adjust TCP Settings
sudo sysctl -w net.ipv4.tcp_fin_timeout=30
sudo sysctl -w net.ipv4.tcp_window_scaling=1
# Use NIC Offloading
sudo ethtool -K eth0 tso on
# Optimize Network Buffers
sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=167
By implementing these practices and utilizing these tools, you can maintain a robust and high-performing Linux environment. Happy monitoring and tuning!