Objective
The OpenSOC project is a collaborative open source development project dedicated to providing an extensible
and scalable advanced security analytics tool. It has strong foundations in the Apache Hadoop Framework and
values collaboration for high-quality community-based open source development.
The OpenSOC project has the following goals:
- To provide a collaborative open source community for development of an extensible and scalable advanced security analytics tool
- To encourage open communication for additional features and identification of deficiencies for a stable and functionally usable tool
- To identify key feature enhancements to drive technology efforts around efficient security analytics
The OpenSOC project welcomes participation from all people and organizations for development, enhancements, and/or implementation support.
What is OpenSOC?
OpenSOC is a Big Data security analytics framework designed to consume and monitor network traffic and machine
exhaust data of a data center. OpenSOC is extensible and is designed to work at a massive scale.
The framework provides the following capabilities:
- Extensible spouts and parsers for attaching OpenSOC to monitor any telemetry source
- Extensible enrichment framework for any telemetry stream
- Anomaly detection and real-time rules-based alerts for any telemetry stream
- Hadoop-backed storage for telemetry stream with a customizable retention time
- Automated real-time indexin for telemetry streams backed by Elastic Search
- Telemetry correlation and SQL query capability for data stored in Hadoop backed by Hive
- ODBC/JDBC compatibility and integration with existing analytics tools
OpenSOC is designed to scale up to consume millions of messages per second, enrich them, run them through anomaly
detection algorithms, and issue real-time alerts.
What do I need to run OpenSOC
- 2 Network Capture Cards (Recommend Napatech NT20E2-CAP)
- Apache Flume 1.4.0 +
- Apache Kafka 0.8.1+
- Apache Storm 0.9 +
- Apache Hadoop 2.x (any distribution)
- Apache Hive 12 + (13 recommended)
- Apache Hbase 0.94+
- Elastic Search 1.1 +
- MySQL 5.6+
Components
OpenSOC consists of the following repositories
- OpenSOC-Streaming:This repository contains topologies for processing, enriching, indexing, and
corelating telemetry messages, PCAP reconstruction service, and various other data services. This module is
open source under Apache 2.0 License. It is available on github:
https://github.com/OpenSOC/opensoc-streaming
- OpenSOC-UI:UI for performing log and network packet analytics, displaying alerts, and errors. This module is
open source under Apache 2.0 License. It is available on github: https://github.com/OpenSOC/opensoc-ui
Installation
Instructions for obtaining OpenSOC can be found at the
project's
Primary Wiki.