AWS Data Firehose and MariaDB Integration

Input and output integration overview

<p>This plugin listens for metrics sent via HTTP from AWS Data Firehose in supported data formats, providing real-time data ingestion capabilities.</p>

<p>This plugin writes metrics from Telegraf directly into MariaDB using parameterized SQL INSERT statements, offering a flexible way to store metrics in structured, relational tables.</p>

Integration details

AWS Data Firehose

<p>The AWS Data Firehose Telegraf plugin is designed to receive metrics from AWS Data Firehose via HTTP. This plugin listens for incoming data in various formats and processes it according to the request-response schema outlined in the official AWS documentation. Unlike standard input plugins that operate on a fixed interval, this service plugin initializes a listener that remains active, waiting for incoming metrics. This allows for real-time data ingestion from AWS Data Firehose, making it suitable for scenarios where immediate data processing is required. Key features include the ability to specify service addresses, paths, and support for TLS connections for secure data transmission. Additionally, the plugin accommodates optional authentication keys and custom tags, enhancing its flexibility in various use cases involving data streaming and processing.</p>

MariaDB

<p>The SQL output plugin in Telegraf enables direct writing of metrics into SQL-compatible databases like MariaDB by executing parameterized SQL statements. With support for the MySQL driver, the plugin seamlessly integrates with MariaDB for reliable, structured metric storage. This setup is ideal for users who prefer SQL-based analytics or want to store metrics alongside business data for unified querying. MariaDB is a community-developed, enterprise-grade fork of MySQL that emphasizes performance, security, and openness. The plugin supports inserting time series metrics into custom schemas, enabling flexible analytics and integrations with BI tools like Metabase or Grafana using SQL connectors.</p>

Configuration

AWS Data Firehose

MariaDB

Input and output integration examples

AWS Data Firehose

<ol> <li> <p><strong>Real-Time Data Analytics</strong>: Using the AWS Data Firehose plugin, organizations can stream data in real-time from various sources, such as application logs or IoT devices, directly into analytics platforms. This allows data teams to analyze incoming data as it is generated, enabling rapid insights and operational adjustments based on fresh metrics.</p> </li> <li> <p><strong>Profile Access Patterns for Optimization</strong>: By collecting data about how clients interact with applications through AWS Data Firehose, businesses can gain valuable insights into user behavior. This can drive content personalization strategies or optimize server architecture for better performance based on traffic patterns.</p> </li> <li> <p><strong>Automated Alerting Mechanism</strong>: Integrating AWS Data Firehose with alerting systems via this plugin allows teams to set up automated alerts based on specific metrics collected. For example, if a particular threshold is reached in the input data, alerts can trigger operations teams to investigate potential issues before they escalate.</p> </li> </ol>

MariaDB

<ol> <li> <p><strong>Business Intelligence Integration</strong>: Store application performance metrics directly into MariaDB and connect it to BI tools like Metabase or Apache Superset. This setup allows blending of operational data with business KPIs for unified dashboards, enhancing visibility across departments.</p> </li> <li> <p><strong>Compliance Reporting with Historical Metrics</strong>: Use this plugin to log metrics into MariaDB for audit and compliance use cases. The relational model enables precise querying of past performance indicators with timestamped entries, supporting regulatory documentation.</p> </li> <li> <p><strong>Custom Alerting Based on SQL Logic</strong>: Insert metrics into MariaDB and use custom SQL queries to define alert thresholds or conditions. Combined with cron jobs or scheduled scripts, this enables advanced alerting workflows not possible with traditional metric platforms.</p> </li> <li> <p><strong>IoT Sensor Metrics Storage</strong>: Collect sensor data from IoT devices via Telegraf and store it in MariaDB using a normalized schema. This approach is cost-effective and integrates well with existing SQL-based systems for real-time or historical analysis.</p> </li> </ol>

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.