What to Log So 3am-You Can Actually Debug It

It is 03:14. PagerDuty has been screaming for six minutes. Checkout is throwing 500s for some customers but not others, and you are staring at a log stream that says, in its entirety, ERROR: operation failed, forty times a second. Which operation? For which customer? Failed how? The log knows none of this, because the person who wrote it was you, at 14:00 on a calm Tuesday, and calm-Tuesday-you did not think about this exact moment.

Every logging guide on the internet tells you the same four things: use log levels correctly, log in a structured format, do not log secrets, and add a request ID. That advice is not wrong. It is just uselessly generic, because it never shows you the one log line that turns a four-hour incident into a ten-minute one. This article is that line, reverse-engineered from a real incident.

My claim is simple: you are not logging events, you are logging the questions your future self will ask. A log line that cannot answer a question nobody would ask under pressure is noise. A log line missing the field you would kill for at 3am is a bug.

The incident, walked through log line by log line

Here is the real shape of the checkout failure. Orders from one payment provider were failing; orders from another were fine. The service that failed was a payments gateway wrapper calling an external provider. This is what the logs said:

2026-07-22 03:08:41 INFO  Processing checkout
2026-07-22 03:08:42 INFO  Calling payment provider
2026-07-22 03:08:47 ERROR operation failed
2026-07-22 03:08:47 ERROR operation failed
2026-07-22 03:08:48 INFO  Processing checkout

Read that as 3am-you. You cannot tell which checkout failed. You cannot tell which provider. You cannot tell whether the two operation failed lines are the same request retried or two different customers. You cannot tell if line 3 is even related to line 2, because there is nothing connecting them. Every question you have, the log refuses to answer. You end up SSHing into the box, adding print statements, and redeploying into a live incident, which is the worst possible time to be changing code.

Now here is the same failure, logged so that the fix is obvious before you finish reading:

{"ts":"2026-07-22T03:08:47Z","level":"error","msg":"payment authorization failed",
 "event":"payment.authorize","order_id":"ord_8842","tenant":"acme-ke",
 "provider":"pesapal","provider_ref":"PSP-QT19","amount":"KES 4500",
 "attempt":3,"latency_ms":5031,"error":"upstream timeout",
 "upstream_status":504,"trace_id":"a1b2c3d4","user_id":"u_5567"}

You did not need to debug anything. error: upstream timeout, upstream_status: 504, provider: pesapal, latency_ms: 5031, attempt: 3. The provider is timing out at five seconds, on the third retry, for one tenant. The incident is now a conversation with the provider and a change to your timeout budget, not a code archaeology dig. The difference between these two logs is not effort. It is knowing, in advance, which fields matter.

The five fields that turn a message into a diagnosis

Forget "log more". Log these five, on anything that can fail. Each one answers a question you will ask under pressure.

Field	Answers	Example
event (a stable name)	What operation was this?	payment.authorize
The subject ID	Which record failed?	order_id=ord_8842
The tenant / owner	Is it everyone or one customer?	tenant=acme-ke
The dependency + its response	Whose fault is it?	upstream_status=504
trace_id / correlation ID	What is the whole story of this request?	trace_id=a1b2c3d4

Notice what is not on that list: a stack trace on every INFO line, the full request body, or a human-readable sentence. The message field ("payment authorization failed") is for humans skimming; the structured fields are for machines filtering. You want both, kept separate, so you can run grep order_id=ord_8842 and get the entire life of that order.

Correlation IDs: the one thing that pays off most under pressure

If you do only one thing from this article, do this. Generate an ID at the edge of your system (the load balancer, the API gateway, or the first middleware that touches the request), and thread it through every service, queue, and log line that the request touches. In a distributed system this is the difference between "grep for one ID and read the entire causal chain in order" and "manually correlate timestamps across five log streams that are not even clock-synced".

The mechanism matters. An incoming request carries or is assigned an X-Request-Id. Your web layer puts it in a context object. Every log call reads it from context automatically, so nobody has to remember to pass it. When you call another service, you forward it as a header. When you enqueue a background job, you store it in the job payload and restore it when the worker picks the job up. Now a single ID follows the work everywhere, including across the async boundary where causality is usually lost.

This is the same discipline that correlating your logs, metrics and traces depends on, and it is why the 3am outage playbook starts by finding one request and following it rather than reading the firehose. A log without a correlation ID is a fact with no story around it.

What NOT to log

Noise is not neutral. Every line that does not help buries a line that would, raises your bill, and slows every query you run during the incident. Just as important, some fields are actively dangerous.

• Secrets and credentials. No passwords, API keys, tokens, session cookies, or full card numbers. Ever. A log aggregator is a second copy of your database with weaker access controls. Redact at the point of logging, not in a downstream pipeline you might forget to configure.
• PII you do not need for debugging. Log the user_id, not the email and phone number. The ID is enough to find the person in your own database if you genuinely need to; the raw PII in logs is a compliance liability and, in many jurisdictions, a breach if it leaks.
• Full request and response bodies at INFO. They are enormous, they usually contain the PII and secrets above, and they drown the signal. Log the body only at DEBUG, redacted, and only where you have a specific reason.
• Success at volume. One INFO line per successful request is fine. One INFO line per row processed in a million-row batch is a self-inflicted denial of service on your own logging. Log the batch summary, not every row.

What people get wrong

"Just log everything and filter later." This is the most expensive bad advice in the genre. Logging everything means your storage bill is unbounded, your queries during an incident are slow because you are scanning noise, and the one useful line is a needle in a haystack you built on purpose. You do not have a search problem, you have a signal problem, and more hay does not fix it.

"We have log levels, so we are fine." Levels are necessary and insufficient. ERROR: operation failed is a perfectly valid ERROR-level line and it is still useless. The level tells you severity; the fields tell you the cause. A correctly-levelled log with no identifiers is a well-organised way of knowing nothing.

"We will add fields when we need them." You cannot add fields to a log line that has already been written during the incident you are having right now. Logging is the one form of instrumentation that only helps if it was there before the failure. This is why the checklist below belongs in your next PR, not your next quarter.

A checklist for your next PR, not a rewrite

You do not need to re-architect logging across your whole system. You need to apply this to the next error path you touch. On any line that logs a failure, ask:

1. Does it name the operation? A stable event key you can filter on, not a prose sentence that changes wording between versions.
2. Can I find the one affected record? The subject ID (order, invoice, job) is present.
3. Can I tell blast radius? The tenant or user ID is present, so you know instantly whether it is everyone or one account.
4. Does it name the culprit? If a dependency was involved, its identity and its response code or error are logged, so you know whose fault it is before you start reading code.
5. Can I get the whole story? A correlation ID is present and identical to the one on the request that started it.
6. Is it safe? No secret, no card number, no raw PII beyond an ID.

When it is still broken

If you have the fields and still cannot debug an incident, the failure is usually one of three things. First, the correlation ID breaks at an async boundary: check that your queue producers write the ID into the job payload and your consumers restore it into logging context, because this is the most common place the thread is dropped. Second, clock skew makes ordering lie to you: if you are correlating by timestamp instead of by ID across machines, even 200ms of drift will reorder cause and effect, so correlate by ID and treat time as advisory. Third, sampling ate the line you needed: if you sample logs to control cost, never sample errors, and make sure a sampled-out request still keeps its error lines. If all three are clean and you are still blind, you are missing a field, so add it to that error path in the post-incident PR while the pain is fresh.