17th July 2018

SSR17 :: Service Interruption

The Reseller server SSR17 is currently experiencing a service interruption. MySQL is offline. Users hosted on this server will experience service degradation as a result. Our System Operations team is investigating the root cause of this interruption. We will provide updates as more details are made available.

UPDATE 22:30 EDT: Engineers are working to restore MySQL service on SSR17. Thank you for your patience. 

UPDATE 23:15 EDT: SSR17 is going down for file system maintenance. The server is fully offline during this time.

UPDATE July 18th 12:00 AM EDT: The server is still being worked on by our system engineers

UPDATE 3:00 AM EDT: The server is back up but we noticed some further MySQL issues. We have currently started MySQL with MYISAM engine only and are working on recovering INNODB

UPDATE 7:00 AM EDT: We are still working on recovering INNODB tables and databases

UPDATE 10:35 AM EDT: Our System Operations team is still working on restoring InnoDB tables and databases. We will provide updates as quickly as possible.

UPDATE 12:55 PM EDT: Our System Operations team is still working on restoring InnoDB tables and databases. We will provide updates as quickly as possible.

UPDATE 3:25 PM EDT: The restore for InnoDB tables and databases is still in progress. We will provide updates as quickly as possible.

UPDATE 5:15 PM EDT: Restoration is in progress. We do not have an ETA. We will continue to provide updates until this issue is resolved.

UPDATE 6:25 PM EDT: Restoration is still in progress. It's a fairly slow process due to the large amounts of data involved. We appreciate your understanding.

UPDATE 7:10 PM EDT: Progress is steady. An ETA is not available, but please be assured that this issue is being actively worked.

UPDATE 8:20 PM EDT: We are still working to restore MySQL databases on SSR17. Thanks for your patience.

UPDATE 9:40 PM EDT: Our Engineering team is still working on SSR17. Many sites are back online and working, and we won't stop until all issues are solved.

UPDATE 10:15 PM EDT: We are glad to report that SSR17 is back online and fully operational. If you experience any further problems, please reach out to our Guru crew via Ticket, Phone or Live Chat.

UPDATE 10:25 PM EDT: Please accept our apologies- a miscommunication caused us to believe the server was fully fixed. It is online, but has latent issues that are still being addressed.

UPDATE 10:50 PM EDT: SSR17 is going down for file system maintenance to resolve an underlying file system issue. The server will be fully offline during this time. 

UPDATE 11:45 PM EDT: The server is still down for file system maintenance.

UPDATE July 19th 1:00AM EDT: The server is still unavailable for the file system maintenance.

UPDATE 2:00 AM EDT: We are still running file system check on the server

UPDATE 3:00 AM EDT: Unfortunately, we do not have any new updates yet. The file system check is still running on the server.

UPDATE 4:00 AM EDT: The server is back online and all services are up and running. We are still monitoring everything and will keep the notice up until we confirm that everything is stable.

UPDATE 6:00 AM EDT: Unfortunately, we have noticed further issues with the MySQL service. We are noiw again shutting down MySQL and will work on recovering InnoDB engine

UPDATE 8:00 AM EDT: Our System Operations team is working on restoring InnoDB tables and databases.

UPDATE 10:00 AM EDT: The server is undergoing a file system check.

UPDATE 11:20 AM EDT: The file system check is still running.

UPDATE 12:55 PM EDT: Our System Operations team is still working on restoring service as quickly as possible.

UPDATE 1:30 PM EDT: Our System Operations team is working on restoring InnoDB tables and databases. We will provide updates as quickly as possible.

UPDATE 2:55 PM EDT: We are still working on restoring InnoDB tables and databases.

UPDATE 4:30 PM EDT: Engineering is still working hard to restore service to SSR17

UPDATE 5:30 PM EDT: Underlying issues have been resolved, and new hardware is being used. A final restore is in progress.

UPDATE 6:25 PM EDT: The final restore is still underway.

UPDATE 7:40 PM EDT: Engineers are working to complete the restore. Data is being verified. We will keep updating this post regularly with known information.

UPDATE 8:45 PM EDT: About 70% of MySQL databases have been restored. The rest are still restoring. Thank you for your continued patience.

UPDATE 10:00 PM EDT: The database restores have reached 90%. We will update as progress continues.

UPDATE 10:30 PM EDT: Restore process has finished. Should you continue to encounter issues with your database, please feel free to reach out to our live support. We sincerely appreciate your patience and understanding in this matter.