Linux & Servers
47 articles in this topic.
The pull model: systemd timers and release polling
When the client's security team refuses a runner outright, you can still ship. A systemd timer on their server asks GitHub what the latest release is, compares it against what is checked out, and deploys when they differ. Nothing external can execute anything.
Anatomy of a Frappe bench
Five directories, one of which is not a directory at all but a collection of independent git repositories. Understanding that is the difference between a deploy that works and a Saturday spent fixing file ownership.
What a runner actually is: GitHub-hosted vs self-hosted
A runner is a process that asks GitHub for work and executes it wherever it happens to live. Once that sentence is true for you, deploying to a firewalled server with no inbound access stops being a trick.
Your Cert Automation Should Default to Let's Encrypt Staging
Five test renewals can exhaust the duplicate certificate limit and block a real renewal for a week. Here is why certbot drifts to production, and the directory split that prevents it.
Certbot connection refused on http-01: diagnosing proxies, IPv6 and port 80
Connection refused is not the same as a timeout, and the difference tells you exactly where to look. A diagnostic order for failed Let's Encrypt http-01 challenges.
Your App Keeps Dying: A systemd Unit File That Actually Keeps It Running
Restart=always is not high availability, it is a bandaid over a bug. Here is the systemd unit file, restart policy, resource cap and sandbox config that keeps a real app alive on a real VPS.
Nginx Reverse Proxy, TLS and Rate Limiting: The Config You Should Actually Be Running
A one-line proxy_pass is not a reverse proxy. Real timeouts, correct forwarded headers, a Mozilla-grade TLS block and limit_req zones that stop credential stuffing without breaking your own app.
Killed by the Kernel: What Really Happens When Your Server Runs Out of RAM
The OOM killer is not random and your server did not just crash. Learn to read the dmesg kill record, tell a cgroup OOM from a global one, and cap the guilty process so MariaDB stops taking the bullet.
Restore First: How to Build Backups You Have Actually Tested
Write the restore script before the backup script. A practical restore-first methodology with restic, verified database dumps, timed recovery drills and the failure modes that turn a backup into an expensive nothing.
Certbot http-01 Fails Behind Your Force-HTTPS Redirect: It Is Not HSTS
Let's Encrypt follows your 301 to HTTPS and ignores HSTS entirely. So the real cause of a failed http-01 renewal is one of four things, and disabling HSTS is not the fix.
Certbot renewal fails silently: alerting before the certificate expires
Certbot can fail twice a day for three months without telling anyone, and Let's Encrypt stopped sending expiry emails in 2025. Three independent alerts that catch it, with the tests.
Lock Down a Fresh VPS in 20 Minutes: SSH Hardening That Survives Real Attacks
A timed, copy-paste SSH hardening run for Ubuntu 24.04 - ed25519 keys, drop-in config that actually wins, UFW, fail2ban - plus the socket activation and drop-in ordering traps that lock people out.
The 3am Outage Playbook: Reading Linux Logs Like a Detective
Build the timeline before you build a theory. A working method for journalctl, grep and awk during a real outage - boot IDs, priority filters, the first anomaly rule, and the three culprits behind most 3am pages.
Kubernetes DNS Timeouts That Come and Go: ndots, conntrack and the Five Second Stall
Intermittent i/o timeout errors on Kubernetes DNS are three different bugs wearing one shirt. How to tell ndots amplification, a kernel conntrack race and a starved CoreDNS apart.
Cron vs systemd Timers vs the Frappe Scheduler: Pick by Failure Mode, Not by Habit
Three schedulers, three completely different failure modes. A decision rule plus working configs for cron with flock, systemd timers with catch-up, and Frappe scheduler_events - and why running all three breaks things.
"Too Many Certificates Already Issued": Fixing Let's Encrypt Rate Limits
The duplicate certificate limit is 5 per exact name set per week and global across all accounts, so switching clients cannot help. Here is what actually works.
'Unable to get local issuer certificate': fix the chain, not the client
The site works in Chrome and fails everywhere else. That is an incomplete nginx certificate chain, and the fix is fullchain.pem, not curl -k or disabling verification.
Certbot Keeps Making New Certificates Instead of Renewing: --cert-name Explained
Certbot identifies a certificate by its lineage name, not its domain list. Here is why it spawns duplicate -0001 certs and how --cert-name fixes it for good.
Your fail2ban Stopped Banning Anyone and Never Told You
fail2ban fails silently: a dead filter looks exactly like a quiet internet. Here is the post-upgrade check that proves your jails still work, and how to alert on zero bans.
OOMKilled and Nothing Else: Finding What Really Ate Your Pod
Kubernetes records OOMKilled and exit code 137, then deletes the evidence. Here is how to read cgroup counters, kernel logs and working set to find the actual cause.
Postgres PANIC 'No space left on device': do not delete pg_wal
A full disk PANICs Postgres into a crash loop, and deleting WAL files to free space turns it into a restore from backup. The safe recovery order, and the slot that caused it.
Redis 'OOM command not allowed': Why Switching to LRU Can Cost You Data
The one-line fix for Redis OOM errors quietly deletes your data. Here is how to tell a safe cache instance from a durable one, and what to set for each.
A Stop Job Is Running for User Manager for UID 1000: The Real Cause
The 90 second shutdown hang is systemd waiting on one user process that ignored SIGTERM. How to find it, and why the DefaultTimeoutStopSec advice is a trap.
Certbot Bypasses systemd for Nginx: Why Renewals Leave Them Out of Sync
Certbot's nginx plugin signals nginx directly instead of via systemctl, so systemd's state drifts from reality after renewal. Here is the PID-file race and the fix.
Stop Using chmod 777: What to Do Instead When You Get Permission Denied
chmod 777 makes a file world-writable and turns an uploads folder into a remote code execution path. Here is how to actually diagnose Permission denied and fix it with chown and 755/644.
'The Server Is Slow' With Zero Budget and No APM: A Command-Line Triage
No Datadog, no New Relic, no dashboards. Here is a free, repeatable command-line order to find why a production server is slow, using tools already on the box.
'Could Not Get Lock' on dpkg: Don't rm the Lock File Before Checking This
The top answer says delete the dpkg lock file. That corrupts your package database if a real process still holds it. Check what holds the lock first, then decide.
Frappe Socket.IO Not Working in Production: Diagnose Before You Reinstall
Realtime updates dead in production? Do not run the blind bench setup reinstall ritual. It is almost always one thing: nginx, supervisor, or a firewalled port.
Frappe Scheduled Jobs Not Running: The Full Checklist Nobody Gives You
Frappe scheduled jobs need four things at once: a cron tick, the per-site flag, running workers, and an unblocked queue. Here is how to find the broken link instead of guessing bench enable-scheduler.
Redis 'Connection refused' in Frappe: Find the Real Cause, Not Just Restart
Error 111 connecting to localhost:13000 is not a wrong port, it means nothing is listening. Read the real port, ping it, then fix the actual cause per deployment type.
Anyone Can Forge X-Forwarded-For: Configuring nginx real_ip Correctly
A raw client request can claim any source IP in X-Forwarded-For. Here is how to scope nginx set_real_ip_from and real_ip_recursive so your rate limits and allowlists trust a verified IP.
'No Space Left on Device' but df Shows Free Space: It's Inodes
df -h shows gigabytes free but every write fails with No space left on device. You are out of inodes, not bytes. Here is how to confirm it with df -i and fix it.
The OOM Killer Took Down My App: Reading dmesg to Find the Real Memory Hog
Your app vanished with exit code 137 and its own logs show nothing. The kernel killed it. Here is how to read the OOM report and find the process that actually ate the RAM.
pip 'externally-managed-environment': What PEP 668 Protects and the Right Fix
The externally-managed-environment error is PEP 668 stopping pip from corrupting your OS packages. Use a venv, not --break-system-packages, and here is exactly why --user is not reliable.
Read-Only File System on a Live Server: Diagnose Before You Remount
When ext4 remounts read-only, the kernel is protecting a failing disk. Read dmesg and check SMART before you remount rw, or you turn recoverable loss into total loss.
'Temporary Failure in Name Resolution': Fixing Ubuntu DNS Without Fighting systemd-resolved
Editing /etc/resolv.conf on modern Ubuntu fails because it is a symlink systemd-resolved regenerates. Set DNS in netplan, restore the symlink, and flush the cache.
Start Request Repeated Too Quickly: The Real Fix, Not Just reset-failed
systemd's 'Start request repeated too quickly' means your service already crashed five times, it is not the bug. Read the real crash in journalctl, fix the root cause, then reset-failed.
'Too Many Open Files' Won't Go Away? You Are Editing the Wrong Limit
Raising limits.conf does nothing for a systemd service. Here is where the file-descriptor limit really lives, how to set LimitNOFILE, and how to prove it reached the process.
What Linux Really Is (And Why It Runs the World Quietly)
Most people think Linux is just “that black screen operating system developers use.” That misunderstanding is one of the biggest reasons many software developers, founders, and even IT managers…
Linux for ERPNext: What a Server Really Is (And Why Businesses Use Linux)
When business owners say,“I host my ERPNext in the cloud,”most of them don’t actually know what that means. They imagine the cloud as something abstract - like magic. But in reality, your entire…
Access Denied for root@localhost After the MariaDB 10.4 Upgrade
MariaDB 10.4 authenticates root through the unix_socket plugin, so the password you have used for years now returns error 1698. What changed, how to get in, and the right fix.
python: command not found on Ubuntu 20.04, and Why You Should Not Symlink It
Ubuntu 20.04 ships no /usr/bin/python at all, so scripts and deploy jobs die with command not found. The correct fixes, and why the popular ln -s answer causes a worse bug later.
502 Bad Gateway: It Is Almost Never Nginx
A 502 means Nginx asked your application for a response and did not get one. Read the error log line, match it to one of four causes, and stop restarting things at random.
Ubuntu 18.04 Ignored /etc/network/interfaces: A Netplan Static IP That Works
18.04 replaced ifupdown with Netplan, so the interfaces file you have edited for a decade does nothing. Here is the YAML that gives a server a static IP, and how to not lock yourself out.
Installing ERPNext on Ubuntu 16.04: What the Easy Install Script Does Not Tell You
The ERPNext install script works until it does not. Here are the four failures I hit on real servers, the MariaDB settings it needs, and how to finish a half-installed bench.
Free HTTPS on Nginx With Let's Encrypt, and the Renewal Everyone Forgets
Getting a free Let's Encrypt certificate on Nginx takes five minutes. Keeping it alive for the next three years is the part that breaks. Here is the issue, the config and the renewal.
Ubuntu 16.04 Killed My Upstart Job: Converting an init Script to systemd
Ubuntu 16.04 is the first LTS with systemd as PID 1, so initctl and your /etc/init job stopped working. Here is the unit file that replaces it, and how to read the failure.