At an enterprise customer running employee monitoring (UAM/DLP) software for around 1,500 seats, productivity reports suddenly stopped making sense one morning. Average daily active work time, normally 7-7.7 hours, collapsed to roughly 1.3 hours in a single day — as if the entire company had stopped working overnight. It wasn't a data error; it was a real infrastructure failure.
This is the record of a real case where the vendor's official fix didn't work, and we found and permanently resolved the root cause ourselves.
The symptom: reports collapse overnight
| Period | Avg. daily active time | Users under 4h/day |
|---|---|---|
| Normal period | ~7.7 hours | 6–9% |
| Day of failure | 1.35 hours | 99.3% |
| Weeks after (ongoing) | ~1.3 hours | 99%+ |
The break hit full effect within a single day and persisted for weeks. The customer flagged the data as "broken." We investigated the numbers directly: they were technically accurate — they just didn't reflect normal system performance.
Root cause: a service crashing every 15 seconds
The platform's core monitoring service was crashing and restarting in the background roughly every 15 seconds. The cause: a sequence backing the primary key of a mail-attachment table in PostgreSQL had reached the maximum value of a 32-bit integer (2,147,483,647). Every crash caused the OS to reset all active agent connections.
The result was a cascading failure:
- Monitoring sessions were constantly fragmented into 15-second chunks.
- Activity logs came in truncated and incomplete.
- Calculated "active work time" reflected only a tiny fraction of real time worked.
Why the vendor's official update didn't fix it
We applied the vendor's recommended official update — the issue persisted. The update reset the sequence value but didn't change the column's underlying data type. The sequence climbed back to its maximum within days and the crash loop resumed.
We isolated the root cause ourselves: the problem was a data-type ceiling the update never addressed, and the recurrence of the exact same symptom was the proof.
The permanent fix
Outside the standard update path, we upgraded the column's data type in production from 32-bit integer to 64-bit bigint:
ALTER TABLE <table> ALTER COLUMN <sequence_column> TYPE bigint;
The new ceiling (~9.2 quadrillion) is effectively unreachable in practice. After the change, the service ran without interruption and data flow returned to normal — the failure never recurred.
Had this not been caught proactively, the customer would have opened their dashboards the next business day to find months of broken reporting — a serious escalation. We also reported the finding back to the vendor officially, recommending the data-type migration be included in an official update and that similar deployments be audited proactively.
Takeaways
- "We applied the official update" is not a guarantee. Don't assume a fix covers a problem — verify what the patch actually changed.
- A recurring symptom is the clearest sign the root cause wasn't found. If an issue comes back after a temporary improvement, the fix reset a symptom, not the underlying data.
- Numeric limits (32-bit, etc.) are silent time bombs. High-throughput, long-running systems should have counter columns like these audited periodically for headroom.
- Monitoring your own infrastructure beats waiting on a vendor. Because we found the root cause ourselves, the customer started the next business day with healthy reports.