The Return of the Shadow...

The next generation of detection

A long time ago, in what seems like a galaxy far, far away, the Shadow IDS was released in 1994. SHADOW was the first publicly available, freely distributed intrusion detection system (followed by Snort in 1998). SHADOW was the acronym for "Secondary Heuristics Analysis for Defensive Online Warfare," a research project led by Stephen Northcutt at the Naval Surface Warfare Center, Dahlgren, Virginia. The toolkit was a network security monitoring system built with Tcpdump and various Perl scripts. The enabling technology was the "intelligent definition of tcpdump filters based on the network environment and educated recognition of known and potential exploits from traffic patterns."¹  

As we enter the present day, it's clear that the underlying technology of network security monitoring has yet to evolve significantly since the release of SHADOW. While the open-source community has made commendable contributions with tools like Snort, Zeek, and Suricata, we still rely on filters and rules that necessitate prior knowledge of malicious traffic patterns. This stagnation over three decades underscores the need for a new generation of open-source network security tools that complement the limitations of signatures-based solutions with anomaly detection. This is especially true with the increasing prevalence of traffic encryption.

Fortunately, generative AI technologies like ChatGPT have overtaken the market, and LLM projects are flooding the open-source community. Some of these technologies should be applied to network anomaly detection, especially in an open-source manner, to foster further research and collaboration.

Open-Source Network Anomaly Detection

To make open-source network anomaly detection convincing and accessible, we need to consider three things:

1) Easy to Understand

First, the AI framework must be practical and easy to understand. I am referring to a high level of knowledge of AI fundamentals. Most security analysts know Python. Therefore, it's a relatively easy stretch for security professionals to segue to a Python-based framework like PyTorch (or TensorFlow). There is still a learning curve. For example, analysts do not need to know the algorithmic details of partial derivatives (calculus) to use a stochastic gradient descent (SGD) function. Instead, an analyst should have a high-level, intuitive understanding of why SGD is necessary and how it functions within the PyTorch framework. The truth is that with a fundamental knowledge of AI and a boilerplate understanding of the PyTorch framework, one can quickly design, train, and deploy powerful AI models.

2) Easy to Use

Next, the AI technology stack needs to be easy to use. Fortunately, gone are the days of monolithic code. Not only is it less likely to be scalable, but it is also less flexible and adaptable to integrate with emerging best-of-breed technologies. On the other hand, microservices based on Docker and Docker Compose make things far easier to deploy, scale, and manage in production. And the code is easier to maintain and upgrade. However, like PyTorch, using Docker and Docker Compose requires some effort to learn. It's not quite as steep as learning PyTorch, but it will take time to master. Learning to use Docker to deploy a PyTorch-based solution comes with a significant payoff, and it's gratifying.

3) Easy to Update

Finally, AI models need to be easy to update. Working with AI models is not a fire-and-forget process. On the contrary, AI models like filters and signatures lose their predictive power and decay over time because either the threat landscape changes, a network's traffic profile changes, or the environment changes. When models lose efficacy, data scientists call this phenomenon "mode drift." Again, this is where production operations with PyTorch to continuously train and deploy models at scale shine. One has the convenience of training and validating PyTorch models in Python, plus the advantage of running the models at scale using the C++ interface. Fortunately, there are abundant tools in the open-source community to facilitate all phases of AI operations with PyTorch.

Project Goals

To this end, the ShadowMeter Project attempts to make a PyTorch-based network anomaly detection framework accessible to security analysts and data scientists. The goal is to make it easy to use, understand, and update detection models in production and research environments. Thanks to YAF, part of CERT’s NetSA Security Suite (open-source), the project is already a fully functional network monitoring solution — minus the deep learning analytics. This summer, work is diligently underway to implement the anomaly detection engine with a working model inspired by the MemStream: Memory-Based Streaming Anomaly Detection research paper. 

1. https://scholarship.richmond.edu/cgi/viewcontent.cgireferer=&httpsredir=1&article=1441&context=honors-theses