Disaster Recovery
Due to a fire at the French cloud service provider OVHcloud, millions of websites are offline and web services are no longer accessible. Strong Ops GmbH is working with affected customers and OVHcloud to limit the damage and take over crisis management to restore the productive operation of the failed systems as quickly as possible.
What happened?
A fire at a French cloud service provider OVH has disabled millions of websites. The portals of government agencies, banks, businesses, and news websites were no longer accessible and part of France’s webspace was destroyed, according to internet observers. Europe’s largest cloud service provider has asked its customers, including the French government, the Pompidou Center and the cryptocurrency exchange Deribit, to activate their emergency plans following the fire.
The challenges
- Unexpected loss of production data including customer data (poor backups)
- Unexpected loss of revenue due to downtime of productive systems
- Loss of trust from end customers
- Access to provider systems is down (offline)
- Lack of infrastructure as code (IaC) and inadequate documentation
- Time pressure leaves little room for action when restoring systems
The solution
Without knowing the actual extent of the outages and data loss caused by the devastating fire, we immediately began to redesign, implement and bring into operation the customer’s paralyzed infrastructure as a cloud solution from scratch.
The main focus was to ensure that the processes and services relevant to end customers (such as user interfaces and purchasing processes) were available for productive use again as quickly as possible, regardless of whether existing data backups could be restored. If backups were available later on, we would be able to successfully migrate them to the new solution or, alternatively, build up a database from existing data that could still be restored.
In this way, we not only demonstrated our professional and efficient crisis management to those affected but were also able to replace legacy applications and establish a future-oriented cloud solution for the customer.
The introduction of proactive monitoring and incident management as well as mature backup and recovery policies enable stable and secure operational processes that can be restored as quickly as possible even in the event of unexpected outages. In addition, profitable side effects for our customers include lower operating expenses and a modern, robust and scalable infrastructure overall.
What does Strong Ops GmbH offer?
- Design & Development of reliable, scalable and cost-efficient cloud architectures
- Optimization strategies for the entire Toolchain and IT processes
- SIEM, Logging, Alerting, and Backup solutions
- Proactive Monitoring and Incident Management
- Coaching and introduction of DevOps principles/strategies
- Technical Due Diligence
- Managed Cloud Infrastructure
During the fire, over 40 servers failed within minutes at affected customers. Strong Ops GmbH was commissioned to restore affected systems and products where possible or to design, implement and commission them from scratch in the shortest possible time.
The fire irrevocably destroyed over 4 TB of data from Postgres clusters, distributed file systems and back offices. Strong Ops GmbH was commissioned to redesign, implement and commission the affected systems and products from scratch within a very short space of time.
