One of our distribution switches in NYC is not responding, causing downtime for many customers at this location. We are currently investigating this.
Update @ 6:58pm EDT: Further investigation determined that the PDU that the switch was plugged into has failed. We have moved the switch to another PDU, which has brought many customers back online, but those hosted on machines that also were on the failed PDU remain down. We are currently asking the facility to replace this PDU with an on-site spare, after which the machines that are down can boot back up.
Update @ 7:59pm EDT: The actual problem turned out to be a flipped breaker, and the breaker was flipped when one of the machines on the PDU had its power supply blow out. All other machines are back online now, and we are working to recover the data on the failed one. We will post another event specifically to customers on the failed machine with more details.
