HikariCP Connection Timeouts Are Not a Pool Size Problem

Traffic goes up. Latency goes up with it, then falls off a cliff, then the log fills with the same stack trace a thousand times a minute. Every request is now waiting thirty seconds to be told it cannot have a database connection. Somebody in the incident channel says "just bump the pool size" and somebody else has already merged it.

It will help for about a day. Then the same thing happens at a slightly higher traffic level, and now you have 50 connections doing the same damage 10 were doing before, except Postgres is also unhappy. The pool is not the problem. The pool is the place where the problem becomes visible.

A connection pool timeout means connections are being held too long, not that there are too few of them. Until you know what is holding them, changing maximumPoolSize is guessing with a config file.

The exact error

java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)

The number varies slightly (30001ms, 30096ms) because it is measured, not configured. The 30000 comes from HikariCP's default connectionTimeout.

Just above it, if you have HikariCP logging at DEBUG, is the line that actually tells you something:

HikariPool-1 - Pool stats (total=10, active=10, idle=0, waiting=37)

Ten connections, all active, nothing idle, thirty-seven threads queued behind them. That is the whole diagnosis in one line: your application can start work far faster than the database can finish it.

Why raising the pool size does not fix it

Think of the pool as a fixed number of servers in a queue system. Throughput is bounded by how long each connection is held, not by how many you have. If a request holds a connection for 500ms and you have 10 connections, you can serve 20 requests per second. Double the pool to 20 and you get 40 per second, but you have also doubled the concurrent load on the database, so each query gets slower, so hold time goes up, and you land somewhere well short of 40.

Past a certain point more connections make things strictly worse. Every extra Postgres backend is a process competing for CPU, for shared buffers, and for locks. HikariCP's own documentation points at the PostgreSQL project's sizing guidance, which lands on a formula in the region of (core_count * 2) + effective_spindle_count. For a 4-core database server on SSD that is somewhere around 8 to 10 connections. Not 100.

The corollary that people find hard to accept: a pool of 10 that times out is telling you something true about your workload. Raising it to 100 does not fix the workload, it just moves the queue from your application into the database, where it is harder to see and more expensive to have.

Find what is holding the connections

Step 1: Turn on leak detection

HikariCP can log a stack trace for any connection held longer than a threshold. It is off by default (leakDetectionThreshold=0). Turn it on in a staging environment, or in production during the incident if you are already on fire.

spring:
  datasource:
    hikari:
      maximum-pool-size: 10
      connection-timeout: 30000
      leak-detection-threshold: 20000
      max-lifetime: 1800000
      register-mbeans: true

Expected output when it triggers:

Connection leak detection triggered for org.postgresql.jdbc.PgConnection@5f2a on thread http-nio-8080-exec-4,
stack trace follows

The stack trace points at the exact method that borrowed the connection. Nine times out of ten it is a service method annotated @Transactional that calls something over the network partway through. That is your bug. It is not a pool sizing question at all.

Step 2: Ask the database what it is doing

On Postgres, this is the query that ends the argument:

SELECT pid,
       state,
       wait_event_type,
       wait_event,
       now() - xact_start  AS xact_age,
       now() - query_start AS query_age,
       left(query, 100)    AS q
FROM pg_stat_activity
WHERE datname = current_database()
  AND pid <> pg_backend_pid()
ORDER BY xact_start NULLS LAST;

Read the state column first.

What you see	What it means	What to do
Many rows active with large query_age	Genuinely slow queries	Take the query text to EXPLAIN (ANALYZE, BUFFERS). Usually a missing index.
Many rows idle in transaction	Transactions opened and left open while the app does something else	Fix the transaction boundary. This is the leak-detection stack trace from step 1.
Rows with wait_event_type = 'Lock'	Contention on the same rows	Look at update ordering and how long you hold row locks.
Almost nothing running, pool still exhausted	Connections held by application code that is not talking to the database	Leak detection again, or a connection never closed on an error path.

On MySQL or MariaDB the equivalent is SHOW FULL PROCESSLIST or SELECT * FROM information_schema.processlist. The reading is the same: sleeping connections inside an open transaction are the enemy.

Step 3: Put a ceiling on how long anything can hold a connection

These two Postgres settings turn an outage into a handful of failed requests. Set them per role, so you do not affect migrations or backups.

ALTER ROLE app_user SET statement_timeout = '15s';
ALTER ROLE app_user SET idle_in_transaction_session_timeout = '30s';

A query that exceeds statement_timeout is cancelled and its connection returns to the pool. A transaction left idle beyond the second setting has its session terminated. Both produce a clear error in your application log naming the offending statement, which is exactly what you want. Without them, one pathological query can hold a connection for hours.

Step 4: Fix the transaction boundary

The concrete change is usually this shape:

-@Transactional
 public void completeOrder(Long orderId) {
-    Order order = orderRepo.findById(orderId).orElseThrow();
-    PaymentResult result = paymentClient.charge(order);   // 2s network call
-    order.setStatus(result.status());
-    orderRepo.save(order);
+    Order order = readOrder(orderId);                     // short transaction
+    PaymentResult result = paymentClient.charge(order);   // no connection held
+    applyResult(orderId, result);                         // short transaction
 }

Same user-visible latency. Roughly fifty times more throughput per pooled connection.

Step 5: Size the pool, last

Now, and only now, pick a number. Start from the database side: what is the total number of connections the database can serve well? Divide that across every application instance, plus migrations, plus your admin sessions, plus whatever else connects. In Kubernetes, remember that maximumPoolSize is per pod, so a pool of 20 across 12 replicas is 240 connections at the database.

Leave minimumIdle alone. HikariCP defaults it to maximumPoolSize, which gives a fixed-size pool, and a fixed-size pool has far more predictable behaviour under load than one that is constantly opening and closing connections.

Step 6: Add PgBouncer when the pod count is the problem

If you genuinely have many application instances and cannot reduce the aggregate, put PgBouncer between them and Postgres in pool_mode = transaction. Each pod keeps a small Hikari pool pointed at PgBouncer; PgBouncer multiplexes them onto a much smaller set of real backends.

[databases]
appdb = host=10.0.0.5 port=5432 dbname=appdb

[pgbouncer]
pool_mode = transaction
max_client_conn = 2000
default_pool_size = 25

Two things to know before you do this. Transaction pooling means no session-level state survives between transactions, so advisory locks held across statements, SET at session scope, and LISTEN/NOTIFY stop behaving as you expect. And prepared statements need max_prepared_statements to be non-zero, which is how modern PgBouncer tracks protocol-level named prepared statements in transaction and statement pooling mode. Read the PgBouncer configuration reference before rolling it out.

Verification

Watch the pool stats line under the same load that broke you:

HikariPool-1 - Pool stats (total=10, active=3, idle=7, waiting=0)

waiting=0 sustained through peak is the goal. If active is consistently equal to total but waiting stays at zero, you are exactly at capacity and should look again at query latency before traffic grows.

On the database side, re-run the pg_stat_activity query during peak. You want no rows in idle in transaction with an age above a second or two, and your slowest query_age comfortably under statement_timeout.

What people get wrong

• "Set maximumPoolSize to 100." This converts a fast failure into a slow, database-wide degradation. You lose the error that was telling you where the bug is.
• "Raise connectionTimeout to 60000 so requests stop failing." Now users wait a minute to receive an error, and your web server's thread pool fills up too. Shorter timeouts are usually the better direction, not longer.
• "HikariCP has a connection leak bug." Occasionally there is a driver-level socket read that hangs, which is why keepaliveTime and maxLifetime exist. But in the overwhelming majority of cases the leak is application code that borrowed a connection and did not return it on an error path. Leak detection tells you which it is in about ten minutes.
• "Just add a read replica." A replica helps if reads are your bottleneck. It does nothing for a transaction held open across an HTTP call, and it adds replication lag bugs on top of the problem you still have.

When it is still broken

1. Check whether the database has hit its own limit. SHOW max_connections and compare against SELECT count(*) FROM pg_stat_activity. If you are at the ceiling, Hikari's attempts to open new connections fail and the pool starves for a different reason.
2. Look for a stuck connection-adder. If total is below maximumPoolSize and never climbs, HikariCP cannot create connections at all: DNS, TLS handshake, or a firewall dropping idle sockets. Set keepaliveTime below the idle timeout of any NAT or load balancer between the app and the database.
3. Profile a single slow request end to end. Distributed tracing that shows the span between connection acquire and release makes this obvious in a way logs never will.
4. Check for N+1 queries. A page that issues 400 tiny queries holds a connection for the duration of all 400 round trips. Each is fast, the total is not. Similar reasoning to the index work in my notes on reading a slow query log properly.

The habit worth building: treat a pool timeout as a latency alert, not a capacity alert. The pool is a queue, and a queue that is full is telling you about service time. Go and measure the service time.