Database-first debugging
A lesson from production firefighting
"Assuming that a breakage is caused by a code change is often true, but not always. If we had spent less time looking at the recent changes, and more time looking at the database itself, we would have found it much faster."
From a production debugging session. We were chasing a performance issue through git blame and commit diffs for hours. The actual problem was 12.8 million stale database records - damcologs where only 27k contained actual content.
After cleanup: 406,624 records. Down from 13,279,954. Integration server CPU usage dropped immediately. The "bug" wasn't in the code at all.
The pattern
When something breaks, the instinct is to look at what changed recently. git diff, recent commits, recent deploys. This is usually right. But when it's wrong, you burn hours.
Sometimes the problem has been there for months, slowly accumulating until it tips over. The database grew. The load changed. Something external shifted.
The fix: Before diving into code archaeology, spend 10 minutes looking at the actual data. Database size, record counts, query performance. Often the answer is sitting there, waiting.