21st May 2018

[RESOLVED]nl1-ss2 Service Interruption

Update 2:30 AM EDT: The server has been rebooted and it's back online now

Update 5:00 AM EDT: We have noticed some additional file system errors on the server and our system engineers are looking into it.

Update 6:00 AM EDT: We have brought the server down for a File System Check. We will update this post once the server is back online.

Update 7:00 AM EDT: Our system engineers are still running File System Check on the server

Update 9:30 AM EDT : Our system engineers are still running the File System Check. We will continue to provide updates as quickly as possible.

Update 10:30 AM EDT : The File System Check is still running on the server.

Update 12:00 PM EDT : Our system engineers are still running the File System Check. We will provide updates as quickly as possible.

Update 1:40 PM EDT : The File System Check is still in progress. We will continue to provide updates.

Update 2:45 PM EDT : Our system engineers are still working on restoring service. We will continue to provide updates as quickly as possible.

Update 3:45 PM EDT : The File System Check is continuing to run and we will continue to provide updates.

Update 4:45 PM EDT : Our system engineers are still running the File System Check. We will update this post regularly until this process is complete.

Update 5:55 PM EDT : Our system engineers are still running the File System Check. We will continue to update this post regularly and as more information becomes available.

Update 6:45 PM EDT : The status remains largely the same. Progress is being made on the File System Check. We started a Bare Metal Restore to a new hardware in an effor to expadite restoration of service. 

Update 10:00PM EDT : Engeineers are still working a two-pronged approach to resolving this. We are working hard to restore server as soon as possible.

Update 11:00PM EDT : The previously mentioned processes are still in progress. You can be assured that we are working to restore service as quickly as we can.

Update May 22nd 1:00 AM EDT : Our system engineers are still working on the server at the moment.

Update May 22nd 2:00 AM EDT : Both processes, the full server restoration and File System Check, are still running.

Update May 22nd 3:45 AM EDT : Engineers are still working on the server. More information will be posted as soon as we'll have further updates

Update May 22nd 5:45 AM EDT : The File System Check and a restore are still ongoing. 

Update May 22nd 9:45 AM EDT : The File System Check and Server Restore are still in progress. 

Update May 22nd 11:15 AM EDT : Our engineers are still working on the file system check and a full server restoration.

Update May 22nd 1:00 PM EDT : The File System Check and Server Restore are still in progress.

Update May 22nd 2:45 PM EDT : Our engineers are still working on the file system check and a full server restoration.

Update May 22nd 5:00 PM EDT : Restoration is still in progress. Thank you for your continued patience.

Update May 22nd 7:00 PM EDT : The full server restoration and File System Check are continuing.

Update May 22nd 8:15 PM EDT : The full server restoration is continuing.

Update May 22nd 11:15 PM EDT : The full server restoration is progressing towards completion.

Update May 23rd 01:30 AM EDT : The full server restore is still running.

Update May 23rd 03:45 AM EDT : The server restoration is still running and it should be finished soon.

Update May 23rd 05:45 AM EDT : The restoration process is still running at the moment.

Update May 23rd 08:00 AM EDT : The full server restore is still running.

Update May 23rd 11:45 AM EDT : The restoration process is still running at the moment.

Update May 23rd 1:12 PM EDT : The restoration process has now completed and we are in the final stages of enabling access. During this time frame some sites will be available while others may not. We will have further updates as we progress.

Update May 23rd 2:59 PM EDT : We are in the final stages of enabling access. During this time frame some sites services will be available while others may not. We will have further updates very soon for you.

Update May 23rd 4:40 PM EDT : Web services and email are online. Roundcube (webmail) is offline. MySQL databases are being restored separately. Database dependant websites may not load correctly until this process is completed. The MySQL restore is currently about 8% completed.

Update May 23rd 6:15 PM EDT : MySQL restoration is still progressing toward completion.

Update May 23rd 8:00 PM EDT : The next phase of MySQL restoration has begun.

Update May 23rd 10:30 PM EDT : MySQL restoration is still in progress.

Update May 23rd 11:40 PM EDT : MySQL restoration is mostly completed. Engineers are working to restore the final database tables.

Update May 24th 3:00 AM EDT : We are doing further restoration of MySQL. INNODB will temporary have to be disabled while tables are being reimported. All MYISAM tables and databases should work fine.

Update May 24th 6:00 AM EDT : The restore has been completed and the service is back online now. Please contact our support department if you notice any additional issues with accounts on this server.