Linux Server Infrastructure: A System Administrator's Field Guide

Introduction

Linux server administration is a discipline that rewards systematic thinking. Servers are not set-and-forget appliances; they require ongoing attention to security patches, resource utilisation, log management, and capacity planning. This guide consolidates the practices I have developed over years of managing production Linux infrastructure, from single VPS instances to multi-node clusters handling critical workloads.

Initial Server Hardening

Every new server should go through a standard hardening process before it receives any traffic. Automation tools like Ansible make this repeatable:

SSH Configuration

The SSH daemon is the most attacked service on any internet-facing Linux server. These baseline changes reduce the attack surface significantly:

# /etc/ssh/sshd_config
Port 2222                              # Change from default 22
PermitRootLogin no                     # Disable root SSH
PasswordAuthentication no              # Key-based authentication only
PubkeyAuthentication yes
MaxAuthTries 3                         # Rate-limit authentication attempts
ClientAliveInterval 300                # Drop idle connections
ClientAliveCountMax 0
AllowUsers kushal                      # Whitelist specific users

Restart SSH after applying: systemctl restart sshd. Verify connectivity in a separate session before closing the current one.

Firewall Configuration

A firewall should be in place before the server is reachable on the network. ufw provides a straightforward interface:

ufw default deny incoming
ufw default allow outgoing
ufw allow 2222/tcp comment 'SSH'
ufw allow 80/tcp comment 'HTTP'
ufw allow 443/tcp comment 'HTTPS'
ufw enable

For more complex environments, nftables or iptables offer finer control. The principle is the same: deny by default, allow explicitly.

Automatic Security Updates

Critical vulnerabilities demand fast remediation. Configure unattended-upgrades for security patches while keeping application-level updates manual:

apt install unattended-upgrades
dpkg-reconfigure --priority=low unattended-upgrades

Monitor the /var/log/unattended-upgrades/ log for any failures.

Monitoring and Alerting

A server you do not monitor is a server whose failure you will discover from users, not from your dashboard.

Essential Metrics

Track these metrics across every server:

Implementing Monitoring with Prometheus and Node Exporter

The Prometheus stack is the de facto standard for open-source monitoring:

# docker-compose.monitoring.yml
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - 9090:9090

  node_exporter:
    image: prom/node-exporter:latest
    network_mode: host
    pid: host
    volumes:
      - /:/host:ro,rslave
    command:
      - '--path.rootfs=/host'

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - 3000:3000

Alerting rules in Prometheus notify you before a metric crosses into critical territory:

groups:
  - name: node_alerts
    rules:
      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 15
        for: 5m
        annotations:
          summary: "Disk space below 15% on {{ $labels.instance }}"

Log Management

Logs are the first place to look during troubleshooting. Centralised logging prevents the need to SSH into individual servers:

• Filebeat — Ship logs from servers to a central Elasticsearch cluster
• Loki — Lightweight log aggregation integrated with Grafana
• rsyslog — Traditional syslog forwarding for legacy systems

At minimum, every server should forward auth and syslog messages to a central collector.

Backup Strategy

A backup that has never been restored is a hope, not a plan.

3-2-1 Rule

The industry-standard backup framework:

• 3 copies of your data
• 2 different storage media
• 1 copy off-site

Implementing Backups

For database backups, use the native dump tools:

#!/bin/bash
# /usr/local/bin/postgres-backup.sh
BACKUP_DIR="/var/backups/postgres"
RETENTION_DAYS=30
TIMESTAMP=$(date +%Y%m%d-%H%M%S)

pg_dump -U postgres my_database | \
  gzip > "$BACKUP_DIR/my_database-$TIMESTAMP.sql.gz"

# Sync to off-site S3-compatible storage
rclone sync "$BACKUP_DIR" "minio-backup:postgres-dumps"

# Remove local backups older than retention
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETENTION_DAYS -delete

This script runs daily via cron and syncs to an off-site MinIO bucket. The retention policy prevents unbounded storage growth.

Test Restores

Schedule monthly restore drills. A backup that cannot be restored is worthless. Automate the test:

#!/bin/bash
# restore-test.sh — run monthly
pg_restore --test my_database.sql.gz
if [ $? -eq 0 ]; then
  logger "Restore test passed for my_database"
else
  logger -p local0.err "Restore test FAILED for my_database"
fi

Incident Response

When a server goes down or a service degrades, a structured response process minimises downtime and prevents recurrence.

The IR Process

1. Detect — Monitoring alerts or user reports
2. Triage — Determine severity and impact scope
3. Mitigate — Restore service (even temporarily)
4. Diagnose — Identify root cause
5. Resolve — Apply permanent fix
6. Review — Document what happened and update runbooks

Essential Runbook Items

Every server should have a runbook accessible to the team:

Server: web-01
IP: 192.168.1.10
OS: Ubuntu 24.04
Services: Nginx, Node.js (port 3000), PostgreSQL
Backup: Daily pg_dump to MinIO

Common Issues:
- Nginx not starting: Check config with `nginx -t`
- Node.js OOM: Restart with `systemctl restart my-app`
- Disk full: Check with `du -sh /var/log/* | sort -rh`

Performance Tuning

Kernel Parameters

Adjusts sysctl settings for server workloads:

# /etc/sysctl.d/99-server.conf

# Network performance
net.core.somaxconn = 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15

# File descriptors
fs.file-max = 100000

# Virtual memory
vm.swappiness = 10
vm.vfs_cache_pressure = 50

Disk I/O

For database servers, the I/O scheduler matters. Modern SSDs work well with none (NVMe) or mq-deadline:

# Check current scheduler
cat /sys/block/sda/queue/scheduler

# Set via udev rule
echo 'ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"' \
  > /etc/udev/rules.d/60-iosched.rules

Security Maintenance

Patch Cadence

Audit and Compliance

Regularly audit what is running and who has access:

# List all listening services
ss -tlnp

# List all users with shell access
grep -E '/bin/(bash|sh|zsh)' /etc/passwd | cut -d: -f1

# Check failed SSH logins
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn

# Check for world-writable files
find /etc -perm -o+w -type f

Conclusion

Linux server administration is about building reliable, repeatable systems. Hardening every new server consistently, monitoring essential metrics, maintaining tested backups, and following a structured incident response process reduces downtime and operational stress. The practices outlined here form a solid foundation — adapt them to your specific environment and threat model, and revisit them as both the threat landscape and your infrastructure evolve.