How Brazil tackled internet outages caused by heavy rain

Even supercomputers are helpless when their surrounding optical telecommunications break down because of heavy rain. A collaboration with participation from RNP, the national research and education network (NREN) of Brazil, faced a huge challenge with multiple simultaneous cable ruptures, but managed to reestablish normal operation within a week.

In February 2022, the city of Petrópolis in the Mountain Region of Rio de Janeiro was severely affected by heavy rains. The tragedy is considered the largest ever in the municipality, claiming more than 200 lives, while thousands of citizens were left without housing.

In addition to the unprecedented damage to the population, the rains in Petrópolis caused several ruptures of optical cables, which jeopardised the city’s communication with the outside world. The Petrópolis Metropolitan Network (Remep), operated and maintained by RNP, had 60 % of its infrastructure affected, leaving local research and education institutions without internet access for several days.

One of the largest computers in Latin America

As optical cable ruptures occurred at several points in the network, the Network Engineering team of RNP were faced with a complex task. The NREN joined forced with K2 Telecom and Go Telecom, partners in the maintenance of the Petrópolis metropolitan network.

“Since the event took place, RNP has mobilised together with Go Telecom and K2 Telecom to access the city, assess the damage, and implement emergency network recovery actions, without putting the teams at risk. We followed the guidelines of the Civil Defense and City Hall and, when we were allowed access, we spared no effort to return to communication with all institutions,” says Eduardo Grizendi, Director of Engineering and Operations at RNP.

Among the affected institutions was the National Laboratory of Scientific Computing. The laboratory houses one of the largest supercomputers in Latin America, the SDumont supercomputer which is central node (Tier-0) of the National High Performance Processing System (SINAPAD) and thus vital for the continuity of scientific research.

Radio link provided resilience

Besides the National Laboratory of Scientific Computing, the network serves a campus of the Fluminense Federal University, CEFET, the Catholic University of Petrópolis, the City Hall and the Imperial Museum.

In addition to the emergency actions for partial recovery of optical cables, one of the mitigation actions was the activation of a radio link for resilience. Another action was the emergency contracting of a 1 GB/s link, which served as redundancy, guaranteeing connectivity to user institutions until the network was fully recovered.

Image credit: Landslide natural disaster in Petropolis following heavy rains 25 February 2022/Shutterstock