Finding Cyber Threats With Big Data Analytics

Traffic on optical transport networks is growing exponentially, leaving cyber intelligence agencies in charge of monitoring these networks with the unenviable task of trying to sift through ever-increasing amounts of data to search for cyber threats. However, new technologies capable of filtering exploding volumes of real-time traffic are being embedded within emerging network monitoring applications supporting big data and analytics capabilities.

Waves of change are washing through long-haul transport networks because of emerging optical transmission technologies. Innovative optical coherent signaling techniques have expanded individual optical wavelength capacity from 10 gigabits per second to 100G+, opening the door for carriers to deploy transport equipment capable of 400 gigabits per second. Of course, this model offers scalability for the future because it can be deployed directly to the existing fiber base to avoid the exorbitant cost of laying new fiber optic cables.

Cyber intelligence officials tasked with searching for potential network attacks are forced into playing defense because of the sheer volume of traffic and the rapidly evolving technologies used to transport data across global networks. Solutions that were deployed to monitor traffic only one or two years ago may already be obsolete or simply overwhelmed by this optical signaling revolution. As transmission speeds accelerate and the volume of traffic expands exponentially, modern cyber tools are commonly not able to keep pace with the incoming traffic and fail to gain real-time visibility across all of the transport network’s data pipes. So how do cyber intelligence agents plan for success and stay relevant amidst these explosions in traffic volume and speed?

It turns out that the evolving network signaling protocols combined with constantly increasing traffic volume has opened up the potential for new applications to emerge that focus on combining big data analytics and network orchestration. These applications hold the potential to deliver automated network access while arming intelligence officials with greater information on where and when network attacks occur. This information could lead to the type of intelligence that turns the table on cyber terrorists.

Modern big data applications are providing scalable solutions capable of sifting through hundreds of terabits of traffic to pinpoint network anomalies. Analysis of these events can trigger intercept of unlawful network activity, such as malware attacks and denial of service attempts.

IP Flow Information Export (IPFIX) and NetFlow generation are modern applications targeting these big data challenges. These functions summarize the traffic on each individual Internet protocol (IP) flow and allow monitoring probes to efficiently search large volumes of optical network traffic while significantly reducing the load on deep packet inspection applications. More efficient sifting leads to more parallel data processing and increased intelligence collection. This increased monitoring efficiency allows those in charge of protecting networks to focus on more offensive cyber tactics.

Cyber intelligence solutions armed with IPFIX and NetFlow are being deployed by service providers to generate IP flow-based analytics and drive real-time, data-driven decisions. With these advanced capabilities built into their cyber tools, operators are improving their ability to proactively react to potential global network service disruptions before they happen, create more profitable services and make more informed capacity decisions. Using analytics based on IPFIX and NetFlow information as actionable intelligence enables swift reactions to advanced threats that typically evade traditional signature-based security or rule-based solutions.

In addition to IP flow-based analytics, cyber intelligence solutions responsible for surveillance of large scale optical networks are focusing on transport protocol layers, which were typically ignored in the previous generation of monitoring applications. These network signaling protocols carry information identifying the service provider responsible for physical transport as well as detailed geographical information that could determine the physical source or destination of the monitored traffic flow. As agents try to gain an advantage in finding criminals who perpetrate network attacks, they will discover that combing traditional IPFIX or NetFlow information with an extra layer of analytics derived from the optical transport layer can open new opportunities to enhance threat detection.

Previously, cyber intelligence tools focused only on Internet protocol traffic analysis would often miss valuable information from the physical transport network. Correlating analytics extracted from optical transport layers with standard Internet protocol flow analysis provides a complete picture of the network across all layers, from the physical network to the application data. By analyzing information collected across an entire monitored network or unique network segments, cyber intelligence officials can automate orchestration of large datasets gathered across multiple probes and claim true network visibility. This constantly shifting database of network information can be used to characterize the optical network and can be tracked over time to gather historical trends over days, weeks, months or years.

With access to current and historical information, network monitoring applications can identify a baseline for how the network is expected to operate and provide visibility into how flows are traversing the network. More importantly, it presents the opportunity to detect abnormal network behavior and provide early warning of a network attack. This information can be analyzed and used to develop strategies to enhance cyber threat hunting tasks.

In summary, hunting for cyber threats on long-haul optical networks is becoming more difficult each day because of the rapidly growing volume of traffic and increasingly varying transport speeds as new signaling technologies are introduced. Fortunately, modern cyber intelligence solutions are using emerging big data applications to rapidly sift through exploding volumes of network traffic to gather analytics that deliver actionable intelligence. As cyber attacks increase, the combination of big data capabilities and network analytics are a welcome development that will allow network monitoring agents to shift from defense to offense.

Jesse Price is the CEO of NetQuest Corp.