Database failures rarely make headlines, until they cost companies millions of dollars.

From accidentally creating 2.8 billion phantom shares, to a failed banking migration that locked 1.9 million customers out of their accounts, to a database outage that took down one of the world's largest DevOps platforms, these incidents reshaped the way modern software is built.

In this article, we'll examine three real-world database failures, what went wrong, the impact they had, and the engineering lessons they continue to teach teams building critical systems today.

Samsung Securities (2018)

In April 2018, Samsung Securities intended to distribute dividends to its employees. Due to a system error, the platform issued billions of company shares instead of the intended cash dividends. Employees briefly had access to shares that should never have existed, and some sold them before the mistake was discovered.

• The Error

  The 2018 Samsung Securities incident was triggered by a simple fat-finger error that caused the system to mistakenly distribute 2.8 billion phantom shares, worth approximately $105 billion, to around 2,000 employees participating in the company's employee stock ownership plan.

  The intention was to pay each employee a cash dividend of 1,000 South Korean won (approximately $0.93). Instead, an employee accidentally entered 1,000 shares rather than 1,000 won, causing the system to create billions of shares out of thin air and credit them directly to employees' accounts.

• The Impact

  Before the mistake could be corrected, 16 employees sold a portion of the phantom shares, causing Samsung Securities' stock price to plunge by nearly 12% during intraday trading.

  The incident triggered regulatory investigations, exposed weaknesses in the company's internal controls, and ultimately cost Samsung Securities more than $105 million in penalties, compensation, and legal settlements.

• The Lesson

  Following the incident, Samsung Securities implemented several corrective measures that have since become valuable lessons for software engineers building financial systems:

  • Two-Person Rule: Critical corporate actions and large ledger adjustments now require verification and approval from multiple authorized employees before they can be executed.
  • Hard Caps and Limits: System-enforced safeguards and constraints prevent the issuance of shares beyond the company's authorized or outstanding share limits.
  • Automated Anomaly Detection: Real-time monitoring and circuit breakers automatically detect abnormal asset creation and freeze suspicious transactions before they can impact the market.

TSB Bank (2018)

In April 2018, TSB Bank attempted one of the largest banking technology migrations in UK history, moving 5.2 million customers from its former parent company's infrastructure to a new in-house banking platform. Instead of delivering a seamless transition, the migration triggered weeks of outages that left customers unable to access their accounts, make payments, or use online banking services, making it one of the most significant IT failures in the banking industry.

• The Error

  The migration failed due to a combination of technical and project management mistakes. TSB attempted to migrate its banking platform under an aggressive timeline while simultaneously upgrading the core banking system, increasing the complexity and risk of the project.

  Independent reviews also identified poor coordination across more than 1,400 team members and 70+ third-party suppliers, insufficient end-to-end data testing, and the absence of a dedicated data testing and automation strategy. Together, these failures turned what should have been a controlled migration into one of the largest banking IT failures in UK history.

• The Impact

  The migration failure affected 1.9 million customers, leaving many unable to access their accounts for weeks. TSB ultimately reported £330 million in costs related to the incident, including customer compensation, remediation efforts, and regulatory penalties, making it one of the most expensive banking IT failures in UK history.

• The Lesson

  The TSB incident demonstrated that successful database migrations require far more than moving data from one system to another. They demand careful planning, realistic timelines, comprehensive testing, and well-defined rollback strategies.

  Data migration should never be treated as an afterthought. With proper ownership, automated validation, and end-to-end migration testing, organizations can significantly reduce the risk of costly outages and ensure a smoother transition for their critical systems.

GitLab (2017)

In January 2017, GitLab experienced one of the most well-documented database incidents in software history. A routine maintenance operation accidentally deleted production data, triggering a widespread outage and putting thousands of customer repositories at risk. The incident became a textbook example of why backups alone are not enough.

• The Error

  While attempting to resolve database replication issues, an engineer accidentally executed a destructive command on the production database, deleting critical data. Although GitLab had multiple backup and replication mechanisms in place, several backups were outdated, incomplete, or could not be restored successfully, significantly complicating the recovery process.

• The Impact

  GitLab's services remained unavailable for nearly 18 hours, affecting thousands of developers and organizations worldwide. The outage resulted in the loss of up to six hours of customer data for some projects and required an extensive recovery effort to restore production systems.

• The Lesson

  The GitLab incident reinforced a lesson that every engineering team should remember: backups are only valuable if they can be restored successfully. Following the outage, GitLab improved its backup strategy, disaster recovery procedures, replication architecture, and operational runbooks, while making the entire postmortem publicly available to help the engineering community learn from the incident.

Conclusion

Although these incidents occurred in different industries, they reveal a common pattern: the most expensive software failures are rarely caused by a single bug. They happen when critical safeguards are missing or fail under pressure.

Samsung Securities demonstrated the importance of enforcing business rules and validation for high-impact operations. TSB Bank showed that database migrations require careful planning, comprehensive testing, and reliable rollback strategies. GitLab reminded the engineering community that backups are only useful if they can be restored successfully.

Modern software engineering has evolved because of failures like these. Every database constraint, migration plan, backup test, and recovery drill exists for a reason, often because another team learned the lesson the hard way.

The best engineering practices aren't just theoretical recommendations. They're the result of real incidents that cost organizations hundreds of millions of dollars and permanently changed how critical systems are designed, deployed, and maintained.