Guest post by Adam Slagell, NCSA and the Bro team

The National Center for Supercomputing Applications (NCSA) at the University of Illinois faces unique security challenges as it must both maintain an open academic environment accessible to researchers around the world and protect some of the most valuable IT assets in the nation, a point brought home by the recent inauguration of the Blue Waters petascale computing system.

As one of the most powerful supercomputers in the world, Blue Waters enables scientists to carry out research that would be otherwise impossible. Blue Waters provides sustained performance of more than 1 quadrillion calculations per second (1 petaflop) on a range of scientific and engineering applications and also offers big data capabilities (1.5 PB memory, 26 PB disk, 300 PB nearline tape capacity).

Traditional security technologies like inline intrusion prevention systems, stateful firewalls, and security appliances can take a 10 Gbps connection down to 500Mbps easily. NCSA, which has well over 100 Gbps of external WAN connections (and plans to go to 300 Gbps), can't come close to operating efficiently with those kinds of bottlenecks. Instead NCSA relies on passive monitoring techniques, making heavy use of ICSI's open-source Bro network security monitor to understand and protect its network.

Bro is an open-source traffic analyzer that comes out of many years of research by ICSI's Networking and Security scientists. Vern Paxson, who is now the director of the group, began development of the first version in 1995 at the Lawrence Berkeley National Laboratory, and today a group of ICSI researchers and engineers continues to maintain and extend the system, with current funding from the National Science Foundation. Since Bro's early days the system has successfully bridged the traditional gap between academic research and operations. It is now deployed widely for protecting critical cyber-infrastructure at major universities, research labs, supercomputing centers, open-science communities, and industry sites. Bro remains unique in its analysis capabilities as it is not limited to any particular detection strategy - a major restriction of traditional intrusion detection systems. Bro instead provides a flexible platform for implementing a range of sophisticated, in-depth traffic analyses that are tailored to the needs of individual sites.

At NCSA, Bro has been in use for operational security for more than 10 years, and in that time it has become an increasingly important part of its security and networking infrastructure. Starting out, NCSA needed to monitor only a single optical link. But with Blue Waters and the move to a new data center, NCSA is monitoring dozens of 10G connections with 100G connections on the roadmap. Instead of a single host running Bro on that one connection, NCSA now uses a cluster of 80 Bro workers and is rapidly adding workers with the eventual goal of having more than 100.

The move to the new data center gave NCSA a chance to rearchitect how it does network security in general. NCSA now breaks its network up into multiple trust zones. Bro isn't just used at the boundary anymore; it also monitors any traffic that passes zone boundaries on internal networks.

But with 20 10G circuits feeding into their Bro cluster, they had to find a way to aggregate and load balance all that traffic. To that end they are using a suite of specialized hardware that first aggregates all the traffic that could be routed asymmetrically across their network. It then load-balances the flows to 10G network interface controllers (NICs) on each of the servers running Bro. Finally, once the packets reach the Bro servers, they are further load-balanced by the NICs so that flows are distributed among multiple Bro workers. This has proven to be an immensely scalable solution that has helped the NCSA security team keep up with the rapid progress in high-speed networks.

In addition to the increased scale of the Bro deployment at NCSA, the team has continued to find new ways to use it. Early on they began using Bro for intrusion detection, and as they learned more about it, they would ask after each incident, "Could Bro have caught this earlier?" Almost always, the answer was yes and they were able to develop new Bro policies toward that purpose, capturing in them a lot of learned knowledge and a historical record of security incidents.

Beyond security intrusions, NCSA has also found Bro useful for tasks as diverse as finding misconfigurations, auditing security policies, and taking inventory. With Bro, NCSA's security team knows exactly what hosts are in their data center. If a new machine pops up that was not properly vetted, they get an alert. They have also found laptops connecting to networks that they should not be on and services running that shouldn't be due to accidental misconfiguration. This could be as simple as a student test machine accidentally being set up as an open proxy, or a production system that has accidentally reverted to a less secure state after an update undid a previous configuration change. And now NCSA has expanded Bro beyond monitoring network traffic to consuming system events using the instrumented SSH daemon from NERSC, which sends commands entered on key systems to Bro for analysis.

For ICSI's researchers, deployments such as the one at NCSA serve as confirmation that their technology can be used for large-scale deployments in operational settings. As attacks get more complex, and network speeds increase, ICSI's Networking and Security researchers are pursuing a number of active projects that may eventually transition into operations as part of future Bro versions, helping to prepare the system for new challenges. For more information about Bro, visit its home page at http://www.bro.org.

Adam Slagell is a senior research scientist in the Cyber Security Directorate at the NCSA, the Chief Information Security Officer, a member of the University Information Security Committee, and the leader of several projects that blend research and development activities.