How a Local Agent Found a 50-Minute Lock, Not a Slow Database

Tuesday, 1 PM. One message.

The client's message was short: "The app is throwing 500s." The vendor who builds the application had one diagnosis, repeated in three different threads: the database is slow. That is not a diagnosis. It is a shrug with extra steps.

Here is what we did instead. We did not buy anything, we did not spin up anything, and the client's data did not leave the building. We fired up the same kind of stack we write about on this site: a Hermes Agent running qwen3.8:27b, pointed at the AWS account with a strictly read-only key. No mutations, no console, no one hovering over anyone's shoulder. Just an agent that can read CloudWatch Logs, S3, the RDS slow log, and the general log - and then think about what it read. The whole investigation, from first log pull to both emails drafted, finished in under an hour.

The app logs were lying by omission

The first pass looked at the ECS service in the 13:00-14:00 window. Hundreds of 500s, sure. But every single one of them said the same unhelpful thing: a timeout. The application was not broken. It was waiting. For what?

That question belongs to the database, not the application. So the agent went to the slow log - and this is where the case turned.

The signature of a victim

Every slow entry in the window had the same shape:

• Lock_time and Query_time are both around 50 seconds
• Rows_examined: 0

That combination is not "the query is slow". A slow query examines millions of rows and takes a long time. This examined zero rows and still took a full 50 seconds. Zero rows plus exactly the InnoDB lock-wait timeout means the query never started. It spent its whole life standing in a queue. These were victims, not culprits. The slow log was telling us something was holding a lock - but not who.

One thread, fifty minutes, nine thousand updates

The general log has the answer, if you have the patience to build per-connection timelines across a busy server. The agent did it for us. One connection, thread 260280, opened a transaction at 13:08:35. It then batch-processed roughly 65,000 statements - including 9,001 row-level UPDATE device writes - without ever committing. At 13:59:10 it rolled the whole thing back. Fifty minutes of row locks, held hostage by a worker that forgot to commit between batches.

Everything else - the API's PATCH /devices requests, the retrying deposit jobs, the nightly event pipeline - queued up behind it, hit the 50-second timeout, and surfaced to the client as "500 errors". The vendor's "database is slow" was, technically, true: the database was slow because someone was holding the steering wheel and not driving.

The agent also checked the previous ten days: zero lock-wait errors before this window. Not a chronic condition. A regression. Someone had shipped a code path that batch-processes without committing. That is the kind of thing a code review should catch, and the kind of thing that costs a client real traffic when it does not.

Two emails. One for each reader.

This is the part I actually like. The investigation ends the same way every time, and the format depends on who has to act on it:

To the client, the executive summary. Three bullets: what happened, who is at fault, what happens next. "A single long-lived database transaction, caused by a bug in the vendor's application code, blocked all other writes for 50 minutes. It is an application bug, not an infrastructure issue. The vendor has the forensic report and a specific fix. No changes to your AWS bill are needed." That is the whole email. The client knows what to say to the vendor, and they know it is not their infrastructure.

To the vendor, the detailed report. The full per-connection timeline, the exact thread ID, the 65,000 statements, the 9,001 updates, the 13:08:35 open and 13:59:10 rollback, and the exact commit discipline the batch worker is missing. Not "your DB is slow". Here is the transaction, here is why it blocks, here is the line of behavior to fix. You cannot argue with a timestamp.

The bill

One hour of investigation on hardware we already own. For what it is worth, that is the part the numbers make concrete. The traditional path for an incident like this: a senior DBA or a managed-service team gets paged, schedules a session, pulls read-only access (a day or two of approval), and then works the logs. Realistically 2 to 5 business days before the client gets an answer, and the invoice is measured in hundreds of dollars an hour.

Our path: under one hour, on a machine that was already paying for itself, with a read-only key and no approval chain. The vendor fixed the commit logic. The lock incident did not recur.

That is the private stack doing what it is for: the detective work happens on your side of the building, and the only things that leave are two well-written emails.

Need your incidents diagnosed in-house?

Contact us at support@sevenseven.tech or fill out our contact form.

From read-only AWS forensics to a two-audience incident report, we run the investigation on your hardware with your rules.