Suricata and Apache Hudi Integration

Input and output integration overview

<p>This plugin reports internal performance counters of the Suricata IDS/IPS engine and processes the incoming data to fit Telegraf’s format.</p>

<p>Writes metrics to Parquet files via Telegraf’s Parquet output plugin, preparing them for ingestion into Apache Hudi’s lakehouse architecture.</p>

Integration details

Suricata

<p>The Suricata plugin captures and reports internal performance metrics from the Suricata IDS/IPS engine, which includes a wide range of statistics such as traffic volume, memory usage, uptime, and counters for flows and alerts. This plugin listens for JSON-formatted log outputs from Suricata, allowing it to parse and format the data for integration with Telegraf. It operates as a service input plugin, meaning it actively waits for metrics or events from Suricata rather than collecting metrics at predefined intervals. The plugin supports configurations for different metrics versions allowing for enhanced flexibility and detailed data gathering.</p>

Apache Hudi

<p>This configuration leverages Telegraf’s Parquet plugin to serialize metrics into columnar Parquet files suitable for downstream ingestion by Apache Hudi. The plugin writes metrics grouped by metric name into files in a specified directory, buffering writes for efficiency and optionally rotating files on timers. It considers schema compatibility—metrics with incompatible schemas are dropped—ensuring consistency. Apache Hudi can then consume these Parquet files via tools like DeltaStreamer or Spark jobs, enabling transactional ingestion, time-travel queries, and upserts on your time series data.</p>

Configuration

Suricata

Apache Hudi

Input and output integration examples

Suricata

<ol> <li> <p><strong>Network Traffic Analysis</strong>: Utilize the Suricata plugin to track detailed metrics about network intrusion attempts and performance, aiding in real-time threat detection and response. By visualizing captured alerts and flow statistics, security teams can quickly pinpoint vulnerabilities and mitigate risks.</p> </li> <li> <p><strong>Performance Monitoring Dashboard</strong>: Create a dashboard using the Suricata Telegraf plugin metrics to monitor the health and performance of the IDS/IPS engine. This use case provides an overview of memory usage, captured packets, and alert statistics, allowing teams to maintain optimal operating conditions.</p> </li> <li> <p><strong>Automated Security Reporting</strong>: Leverage the plugin to generate regular reports on alert statistics and traffic patterns, helping security analysts to identify long-term trends and prepare strategic defense initiatives. Automated reports also ensure that the security posture of the network is continually assessed.</p> </li> <li> <p><strong>Real-time Alert Handling</strong>: Integrate Suricata’s alert metrics within a broader incident response automation framework. By incorporating the inputs from the Suricata plugin, organizations can develop smart triggers for alerting and automated response workflows that enhance reaction times to potential threats.</p> </li> </ol>

Apache Hudi

<ol> <li> <p><strong>Transactional Lakehouse Metrics</strong>: Buffer and write Web service metrics as Parquet files for DeltaStreamer to ingest into Hudi, enabling upserts, ACID compliance, and time-travel on historical performance data.</p> </li> <li> <p><strong>Edge Device Batch Analytics</strong>: Telegraf running on IoT gateways writes metrics to Parquet locally, where periodic Spark jobs ingest them into Hudi for long-term analytics and traceability.</p> </li> <li> <p><strong>Schema-Enforced Abnormal Metric Handling</strong>: Use Parquet plugin’s strict schema-dropping behavior to prevent malformed or unexpected metric changes. Hudi ingestion then guarantees consistent schema and data quality in downstream datasets.</p> </li> <li> <p><strong>Data Platform Integration</strong>: Store Telegraf metrics as Parquet files in an S3/ADLS landing zone. Hudi’s Spark-based ingestion pipeline then loads them into a unified, queryable lakehouse with business events and logs.</p> </li> </ol>

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.