Jump to content
Sign in to follow this  
3Fingers

Server Status - Outages, Maintenance, Interruptions -

Recommended Posts

Aggregation switch reloaded 
Dec 05 2016 05:13:51 PM PT
Our main aggregation switch in Chicago appears to have reloaded (rebooted) at approximately 6:43pm CST, causing a brief loss of connectivity (a minute or so).

We are still in the process of replacing this switch. We have been waiting (for a week) for the manufacturer to respond to our request for an updated firmware version so that we can finish the configuration and ship it to Chicago; without the updated firmware, the new switch is not usable, because it will have security problems.

When we have a firm date for the installation of the new switch to be completed, we will post a maintenance notification.

Share this post


Link to post
Share on other sites
Subject and date Description
Upcoming switch maintenance early 12/14
Dec 11 2016 10:19:59 PM PT
Between 2am and 6am CST on Wednesday, December 14, we plan to replace our main aggregation switch in Chicago with a newer, faster, and more reliable one. When we swing over the router uplinks to the new switch, we expect for customers to see approximately 5-15 minutes of connectivity loss.

Share this post


Link to post
Share on other sites
Subject and date Description
Comcast maintenance in Chicago
Dec 13 2016 12:21:07 AM PT
Comcast is performing maintenance in the Chicago area that is significantly impacting their network there. As a result, many Comcast clients are currently unable to reach services based in Chicago (including our services there).

Internap normally has a direct Comcast link, which they took offline in response to this, so that traffic would flow over other providers. However, Comcast's maintenance is preventing connectivity more generally, so this did not help.

We have been told that Comcast estimates that their maintenance is expected to be completed by 3:30am CST.

Share this post


Link to post
Share on other sites
On ‎12‎/‎12‎/‎2016 at 10:12 AM, 3Fingers said:
Subject and date Description
Upcoming switch maintenance early 12/14
Dec 11 2016 10:19:59 PM PT
Between 2am and 6am CST on Wednesday, December 14, we plan to replace our main aggregation switch in Chicago with a newer, faster, and more reliable one. When we swing over the router uplinks to the new switch, we expect for customers to see approximately 5-15 minutes of connectivity loss.

Update @ 5:42am CST: This was completed successfully. Connectivity loss lasted around 5 minutes, as expected.

Share this post


Link to post
Share on other sites
Upstream (Internap) problems in NY
Jan 23 2017 09:00:32 AM PT

Internap is having some sort of serious network problem in New York, causing a total loss of connectivity for our services there. They had this same type of problem approximately one hour ago, as well, for about 5 minutes.

We are still awaiting an update from Internap on what happened an hour ago and what is happening again now (presumably they are related).

 

** Update @ 12:20am EST: We continue to wait for Internap to resolve the problem in their New York market. They have told us that there is currently no estimated time to repair, as they don't yet know what is going on

 

** Update @ 12:49pm EST: Internap has now told us now that this is due to an outage within their physical fiber provider.

It usually takes hours, at minimum, to repair fiber cuts, because so much work is required to locate the problem and have a team perform the splicing. Internap says it is exploring other options in the meantime, but we do not expect a quick resolution.

 

** Update @ 2:28pm EST: At 1:37pm EST, Internap was able to bring their facility back online through an alternate fiber link, restoring full connectivity to our services. Internap is continuing to work with their fiber vendor to bring the original fiber back online and restore redundancy.

 

** Update @ 4:54pm EST: We're seeing packet loss occurring within Internap's network in NYC. This is likely due to their temporarily reduced capacity in this market, combined with increased traffic as we enter peak hours. We are asking Internap for another update.

 

** Update @ 12:21am EST: Internap continues to struggle at times in NYC and we have seen two periods of heavy packet loss in the last few hours.

We're told that the current estimate for a repair of their original fiber is 2pm EST later today (1/24).

Share this post


Link to post
Share on other sites
Internap maintenance to use fixed fiber tonight 
Jan 24 2017 03:10:41 PM PT
Internap's fiber provider has told them that it has fixed the fiber path feeding the facility in NYC. Internap plans to test and re-integrate the fiber between 10pm and midnight tonight. This may lead to one or more brief disconnection events or periods of increased latency.

Share this post


Link to post
Share on other sites

Strange routing problems around 11am CST
Feb 24 2017 09:24:20 AM PTWe began observing strange routing issues around 11am CST at our Chicago router. These seem to be tied to specific upstreams, so we have dropped those upstreams now, while we continue to investigate.

Update @ 12:20pm CST: Further investigation has determined that one of our upstream links to Telia stopped responding properly, in an unusual way. We are continuing to work with Telia to investigate and resolve the problematic link. In the meantime, the location is operating normally, with traffic flowing over our other providers.

Share this post


Link to post
Share on other sites
Subject and date Description
Upcoming move to a new machine after 2am EDT on April 1
Mar 28 2017 12:22:44 AM PT

Starting at approximately 2am EDT on April 1, we plan to begin moving all customers off of the machine hosting your server, to other machines at the same location. We are doing this in order to decommission this old machine, which is nearing the end of its useful life because it is no longer acceptably fast or power-efficient.

 

tl;dr 

They are moving our nyc server to a newer machine, yay!

Share this post


Link to post
Share on other sites
Subject and date Description
Upcoming Internap maintenance early 4/1
Mar 30 2017 11:33:57 PM PT
One of our upstreams in Chicago, Internap, will be performing maintenance on two of the three links that we have with them between 11pm CDT and 4am CDT, starting on Friday, March 31 and ending on Saturday, April 1. During this window, some clients in Chicago may notice one or more brief connectivity issues, which should last just a few minutes.

Share this post


Link to post
Share on other sites

Chicago maintenance.....

 

 

Subject and date

Description
Upcoming InterNAP maintenance early 4/21
Apr 19 2017 11:21:39 PM PT
An upstream provider will be performing maintenance on one of its routers between 1am and 4am CDT on Friday, 4/21. This maintenance may cause brief periods of connectivity loss or increased latency for your server.

Share this post


Link to post
Share on other sites

 

New York server ...

Subject and date Description
Upcoming physical move with downtime early 5/3 or 5/4
May 01 2017 10:24:03 AM PT
The machine hosting your service is one of those that we will need to physically move between 10pm EDT on 5/2 and 10am EDT on 5/3, or between 10pm EDT on 5/3 and 10am EDT on 5/4. This means that during one of those two periods, we will need to turn it off (which we will instruct it to do gracefully), disconnect it, load it into a truck, drive it to other facility, place it into the new rack, connect it back up, and turn it back on.

We are splitting up this work into two days to try to keep the downtime to a minimum on each of them. The total downtime for your service as a result of the move will likely be at least a couple of hours, but less than the full 12 that we are allotting for the maintenance -- we are expressing a large window in case the move has to be delayed or takes longer than anticipated.

Share this post


Link to post
Share on other sites
Subject and date Description

Upcoming routine Telia  maintenance
Jun 12 2017 02:05:42 PM PT

One of our upstream providers will be performing maintenance on one of its routers between 2am CDT and 5am CDT on June 15. This maintenance may cause brief periods of connectivity loss or increased latency for some clients to your service.

Share this post


Link to post
Share on other sites
Subject and date Description
Upcoming Xen upgrade with reboot @ 2:30am Eastern Time on June 20
Jun 19 2017 12:40:57 AM PT
     We are planning to reboot the machine hosting your VDS at approximately 2:30am Eastern Time on June 20, in order to upgrade it to the most recent version of Xen. This is necessary because the new version includes a fix for a newly-discovered, critical Xen vulnerability.

For the reboot, your VDS will need to be shut down, then booted back up. We estimate that downtime will last 15-60 minutes.

Share this post


Link to post
Share on other sites
Subject and date Description
Upcoming Xen upgrade with reboot on September 12
Sep 07 2017 11:50:44 PM PT

We are planning to reboot the machine hosting your VDS on September 12, in order to upgrade it to the most recent version of Xen. This is necessary because the new version includes a fix for a newly-discovered, critical Xen vulnerability.

For the reboot, your VDS will need to be shut down, then booted back up. We estimate that downtime will last 15-60 minutes.

 

This is for all servers...

NYC server reboot will be @ 2:30am eastern time.

CHI servers reboot will be @ 1:30am central time.

SEA server reboot will be @ 12:30am pacific time.

Share this post


Link to post
Share on other sites
Upcoming brief connection interruption early on 11/26
Nov 25 2017 12:42:43 AM PT
Between 3am and 4am EST on Sunday, November 26, we plan to upgrade the firmware on our primary Cisco network switch in New York. This should lead to between two and five minutes of connectivity loss for your service at this location.

We are performing this update due to a strange packet loss problem that we are seeing intermittently, which may relate to a bug in the current firmware version.

Share this post


Link to post
Share on other sites

Chicago #1 server...

Subject and date Description
Upcoming downtime with power cycle early on 11/28 due to cabinet migration 
Nov 26 2017 12:17:28 AM PT
Our facilities provider in Chicago, Internap, is requiring that we move several cabinets' worth of gear in this facility over to new cabinets. We have been planning these moves for the last month and coordinating with Internap to try to make them as smooth as possible. We have done such moves successfully many times in the past. 

The moves are set to occur over the course of several mornings (we are doing them early in the morning to minimize customer impact). On November 28, between approximately midnight and 8am CST, we plan to move the equipment in the cabinet hosting your specific service. This means that you will see your service gracefully shut down at some point during the window (close to the beginning of it), and booted back up later in the window (as soon as we have finished moving all of the gear -- at least a few hours later).

Share this post


Link to post
Share on other sites

Upcoming downtime with power cycle early on 11/30 due to cabinet migration

Our facilities provider in Chicago, Internap, is requiring that we move several cabinets' worth of gear in this facility over to new cabinets. We have been planning these moves for the last month and coordinating with Internap to try to make them as smooth as possible. We have done such moves successfully many times in the past.

We have moved other cabinets already, over the course of several mornings (we are doing them early in the morning to minimize customer impact). On November 30, between approximately midnight and 10am CST, we plan to move the equipment in the cabinet hosting your specific service. This means that you will see your service gracefully shut down at some point during the window (close to the beginning of it), and boot back up later in the window (as soon as we have finished moving all of the gear -- at least several hours later).

We will be taking every precaution to be as careful as we can with your hardware. However, there is always the possibility that a hardware problem will be revealed by the move. Because of this, we strongly recommend that you back up any irreplaceable data that you may have stored on the server.

If you require very high levels of uptime and the hours of downtime will cause problems for you on the 30th, please contact us, and we can investigate how we can help, such as by providing a temporary service or moving you to other hardware. (While most of our cabinets in Chicago will need to be moved, a couple will remain in place, and those will only see downtime when our router is migrated this same morning -- likely less than 60 minutes.) 

Share this post


Link to post
Share on other sites

 

Chicago server maintenance...

Subject and date Description
Upcoming routine Internap upstream maintenance on 1/17 and 1/18 
Jan 13 2018 10:21:17 PM PT
One of our upstream providers will be performing maintenance on one of its routers between 10pm CST and 1am CST, starting on 1/17 and again starting on 1/18. This maintenance may cause brief periods of connectivity loss or increased latency for some clients to your service.

Share this post


Link to post
Share on other sites
Subject and date Description
   
Upcoming routine InterNAP maintenance on 1/31 and 2/1 
Jan 28 2018 12:29:50 AM PT
One of our upstream providers will be performing maintenance on one of its routers between 10pm and 2am PST, starting on 1/31 and again on 2/1. This maintenance may cause brief periods of connectivity loss or increased latency for some clients to your service.

 

This is for madhouseservers.com website, Not game servers

Share this post


Link to post
Share on other sites

 

Chicago server maintenance...

Subject and date Description
Upcoming routine upstream maintenance (Telia) 
Apr 11 2018 01:43:46 PM PT
One of our upstream providers will be performing maintenance on one of its routers between 4:30am CDT and 5:30am CDT on April 13. This maintenance may cause brief periods of connectivity loss or increased latency for some clients to your service.

Share this post


Link to post
Share on other sites

Last night, Chicago if anyone was wondering...

Subject and date Description
Brief upstream (Telia) issue a short time ago 
Aug 02 2018 07:04:56 PM PT
Between approximately 8:13pm and 8:27pm CDT, one of our upstreams in Chicago (Telia) experienced some sort of problem with the router that we connect to. This brought down our links to that upstream and caused some lost packets and routing reconvergence for customers coming in, or going out, over that transit provider. 

We have opened a ticket with Telia to ask about this and will continue to follow up with them. 

Update @ 9:52pm CDT: Telia has responded that this was a "very brief major outage" and that they will send us an RFO at a later (unspecified) time.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
Sign in to follow this  

×