Infrastructure Update: Recovery Complete and Stronger Backups

Last week we reported that one of our servers had started failing at the disk level. That work is now finished. Every affected project has been moved to healthy hardware, the failing server has been taken out of service, and we have added several new layers of backups on top of what we had before.

This post is the follow-up to the earlier incident note.

What We Did

We moved every project off the failing server and onto new machines. Instead of one risky cutover, we rebuilt each application on healthy hardware and confirmed it was serving traffic before moving on to the next one. Most projects came back on their next deploy; the rest we handled by hand.

Once everything was running on the new servers, we shut the old machine down for good. It is no longer part of our infrastructure and will not receive any traffic or data.

New Backup Layers

The hardest lesson from this incident was that our backups were not where they needed to be. We changed that.

Database backups now run automatically on a regular schedule and are stored separately from the servers that run your applications. A problem with one machine no longer touches the copies we keep for recovery. We also keep several points in time instead of a single latest copy, so we can roll back to a known-good state rather than only the most recent one.

We tested restoring from these backups, not just creating them. A backup you have never restored is only a guess, so checking that the restore actually works was part of the job.

What This Means For You

If your project was affected, it is already running on the new infrastructure. You do not need to do anything. If you ever notice your application is unavailable, a redeploy from your dashboard will rebuild it on a healthy server.

Your data is now protected by more than one independent copy, kept away from the machines that serve it.

Closing Note

Incidents like this are not fun to write about, but hiding them helps no one. The short version: the failing server is gone, every project is on healthy hardware, and the backup gap that made this stressful has been closed. If you have a question about your specific project, email us at info@vibenest.net with the project name or URL.