Jump to content
Sign in to follow this  
3Fingers

Server Status - Outages, Maintenance, Interruptions -

Recommended Posts

New York servers...

Subject and date Description
Brief upstream (INAP) issue a short time ago 
Aug 09 2018 07:23:40 PM PT
At approximately 10:09pm EDT, we saw between two and three minutes of connectivity loss to our NYC location. During this connectivity loss, overall traffic usage dropped to 0, indicating that no endpoints could connect to our IP space. Our tests indicate that the loss occurred upstream, with INAP or one of its upstreams. We are following up with INAP to ask what happened. 

Update @ 1:10pm EDT on 8/10: INAP reported that this was due to a fiber failure between datacenters and that they have a patching crew on-site to fix it. 

We saw another very similar problem just a few minutes ago, and we are following up again.

Share this post


Link to post
Share on other sites

New York servers...

Subject and date Description
Upcoming Xen upgrade with reboot @ 2:30am Eastern Time on September 14 
Sep 13 2018 09:38:24 AM PT
We are planning to reboot the machine hosting your VDS at approximately 2:30am Eastern Time on September 14, in order to apply critical Intel microcode and Xen updates to address a new speculation-related vulnerability 

Share this post


Link to post
Share on other sites

Chicago Servers...

Subject and date Description
Upcoming Xen upgrade with reboot @ 1:30am Central Time on September 14 
Sep 13 2018 09:32:21 AM PT
We are planning to reboot the machine hosting your VDS at approximately 1:30am Central Time on September 14, in order to apply critical Intel microcode and Xen updates to address a new speculation-related vulnerability 

Share this post


Link to post
Share on other sites

New York servers...

Subject and date Description
Upcoming routine INAP maintenance 9-19 / 9-20 / 9-21 
Sep 18 2018 12:54:46 PM PT
One of our upstream providers will be performing maintenance on one of its routers between 10pm EDT and 2am EDT, starting on 9/19. This maintenance may cause brief periods of connectivity loss or increased latency for some clients to your service. 

INAP also tells us it will be doing an almost identical maintenance event on a different router the next night, from 10pm to 2am EDT again. The same impact may be felt from the second maintenance.

Share this post


Link to post
Share on other sites

 

New York...

Subject and date Description
Upcoming routine INAP maintenance late 10/4 to early 10/5 
Sep 26 2018 11:20:42 PM PT
One of our upstream providers will be performing maintenance on one of its routers between 10pm EDT and 2am EDT, starting on 10/4. This maintenance may cause brief periods of connectivity loss or increased latency for some clients to your service.

Share this post


Link to post
Share on other sites

New York servers...

Subject and date Description
Upcoming routine INAP maintenance starting late 10/9 
Oct 05 2018 11:26:10 PM PT
One of our upstream providers will be performing maintenance on one of its routers between 10pm and 2am EDT, starting on 10/9. This maintenance may cause brief periods of connectivity loss or increased latency for some clients to your service.

Share this post


Link to post
Share on other sites

Chicago...

Subject and date Description
Upstream (Telia) problem 
Oct 11 2018 08:36:03 PM PT
One of our upstream providers in Chicago, Telia Carrier, dropped two of our three 10G connections at approximately 9:15pm CDT this evening, causing a minute or so of routing reconvergence as customer traffic failed over to the third link -- leading to a burst of packet loss and broken connections for many clients. We are following up with Telia and asking for an ETA on when these two links will be repaired so that we can return to full capacity. 

Update @ 1:19am on 10/12:: Telia has resolved the problem on their end that took these circuits down. They tell us that it was a hardware issue with one of their routers.

Share this post


Link to post
Share on other sites

 

NYC lag today...

Subject and date Description
INAP problem in NYC 
Nov 04 2018 11:25:21 AM PT
We are seeing heavy packet loss over one of our four 10G connections to INAP in NYC right now -- starting at approximately 1:50pm EST. We contacted InAP about this and they informed us that it is a known issue that they are actively troubleshooting on their end. We will continue to track this issue and investigate whether we can mitigate it to any degree on our end. (This is not due to an attack on NFO or any other NFO-specific problem, however.) 

Share this post


Link to post
Share on other sites
Subject and date Description
Upcoming upstream maintenance early 3/13 
Mar 09 2019 01:46:19 PM PT
One of our upstream providers will be performing maintenance on the routers that we connect to in Chicago between 2am CDT and 5am CDT on Wednesday, March 13. We are told that this maintenance will break our connection to that provider for some period of time during the maintenance window. Because we have other upstreams in Chicago, there should not be any extended connectivity loss, but some  clients may see brief periods of connectivity loss or increased latency, as traffic automatically shifts to our other upstreams in Chicago and then shifts back.

Share this post


Link to post
Share on other sites
Subject and date Description
Upcoming Xen upgrade with reboot @ 1:30am Central Time on June 20 
Jun 17 2019 10:34:05 PM PT
We are planning to reboot the machine hosting your VDS at approximately 1:30am Central Time on June 20, in order to apply critical Intel microcode and Xen updates to address a new speculation-related vulnerability (Microarchitectural Data Sampling/MDS).

We have generally avoided the need for reboots over the last couple of years by invisibly using the Xen livepatching functionality to update running code. Because this specific new flaw requires the application of updated microcode at boot-time, and because its Xen code updates cannot be patched into a running system, that was not possible in this case. We plan to continue avoiding reboots on our end as much as possible in the future.

We also recommend that all customers take this opportunity to apply the latest security updates from their OS distributions. The vendors for all currently-supported operating systems have released patches for the new vulnerability.

For the reboot, your VDS will need to be shut down for approximately 15-30 minutes. We will attempt to gracefully shut it down on our end through Xen, but on occasion this doesn't work perfectly, so you may consider turning off applications that might write to disk before the maintenance event. If you are running Windows 2012 R2, please also note your VDS may need additional reboots to work properly, due to a bug within Xen related to how it boots up (that only occurs during bootup) -- our system will attempt to detect when this is the case and perform the extra reboot operations, but we recommend checking afterward yourself, as well.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
Sign in to follow this  

×