Ever stopped all departures from London Heathrow? This intern, wanting to remain anonymous, sure did! The story starts 18 years ago.
“As a student, I was doing research at Heathrow Airport, working in a room just below the control tower. We were listening to the control tower instructions and timing how long pilots took to respond. My colleague left for lunch one day having turned his radio onto ‘broadcast’ by accident. As radio is one way, it meant that no one was able to send or receive messages on the frequency that was being used to give take off permission.
I returned to my desk and started to eat my lunch to discover that all departures from the airport had been brought to a standstill by someone who sounded like they were eating their lunch. The realisation that it was the sound of MY lunch being eaten hit me about ten minutes later. I rushed over and flicked the switch off, and one of the busiest international airports in the world started to work again.”
Oleksiy Kovyrin, however, turned off a city-wide network with the push of a wrong button.
“It was my second year in university and I worked for a local ISP as a junior system administrator,” he explains. “We had a large wireless network ~60km in radius covering our whole city and many rural areas around it. This network was used by all major commercial banks and many large enterprises in the area (~100 of bank branches, some large factories, radio stations, etc).
To cover such a large area we basically had to build a huge wifi network that had a powerful antenna in the centre and many smaller regional points of presence would connect to it using directional wifi antennas and then distribute the traffic locally. The core router connected to the central antenna was located at the top floor of the highest building in the area about 20 minutes away from our central office.
I was working on some monitoring scripts for the central router (which was a pretty powerful unix server). I’d run those scripts on a local server that I had on my table, make some changes, run it again, etc. While doing my debugging I received an alert from our production monitor saying that our core router had some (non-critical) issues and, since I was on-call that day, I decided to go take a look. After fixing the issue on the router I went back to my debugging.
And that’s where things went wrong… When I wanted to shut down my local server I’ve switched to a terminal window that was connected to the box, typed “poweroff”, pressed enter and only then realized that I did it on a wrong server. I had that second window opened ever since the monitoring alert an hour ago and now I’ve shut down the core router of our city-wide network. We had to grab a car and drive to the central station to power the router back on, our whole banking infrastructure was down for 30+ minutes.”
Share this story