NSQ and DuckDB Integration

Input and output integration overview

<p>The NSQ Telegraf plugin reads metrics from the NSQD messaging system, allowing for real-time data processing and monitoring.</p>

<p>This plugin enables Telegraf to write structured metrics into DuckDB using SQLite-compatible SQL connections, supporting lightweight local analytics and offline metric analysis.</p>

Integration details

NSQ

<p>The NSQ plugin interfaces with NSQ, a real-time messaging platform, enabling the reading of messages from NSQD. This plugin is categorized as a service plugin, meaning it actively listens for metrics and events rather than polling them at regular intervals. With an emphasis on reliability, it prevents data loss by tracking undelivered messages until they are acknowledged by outputs. The plugin allows for configurations such as specifying NSQLookupd endpoints, topics, and channels, and it supports multiple data formats for flexibility in data handling.</p>

DuckDB

<p>Use the Telegraf SQL plugin to write metrics into a local DuckDB database. DuckDB is an in-process OLAP database designed for efficient analytical queries on columnar data. Although it does not provide a traditional client-server interface, DuckDB can be accessed via SQLite-compatible drivers in embedded mode. This allows Telegraf to store time series metrics in DuckDB using SQL, enabling powerful analytics workflows using familiar SQL syntax, Jupyter notebooks, or integration with data science tools like Python and R. DuckDB’s columnar storage and vectorized execution make it ideal for compact and high-performance metric archives.</p>

Configuration

NSQ

DuckDB

Input and output integration examples

NSQ

<ol> <li> <p><strong>Real-Time Analytics Dashboard</strong>: Integrate this plugin with a visualization tool to create a dashboard that displays real-time metrics from various topics in NSQ. By subscribing to specific topics, users can monitor system health and application performance dynamically, allowing for immediate insights and timely responses to any anomalies.</p> </li> <li> <p><strong>Event-Driven Automation</strong>: Combine NSQ with a serverless architecture to trigger automated workflows based on incoming messages. This use case could involve processing data for machine learning models or responding to user actions in applications, thus streamlining operations and enhancing user experience through rapid processing.</p> </li> <li> <p><strong>Multi-Service Communication Hub</strong>: Use the NSQ plugin to act as a centralized messaging hub among different microservices in a distributed architecture. By enabling services to communicate through NSQ, developers can ensure reliable message delivery while maintaining decoupled service interactions, significantly improving scalability and resilience.</p> </li> <li> <p><strong>Metrics Aggregation for Enhanced Monitoring</strong>: Implement the NSQ plugin to aggregate metrics from multiple sources before sending them to an analytics tool. This setup enables businesses to consolidate data from various applications and services, creating a unified view for better decision-making and strategic planning.</p> </li> </ol>

DuckDB

<ol> <li> <p><strong>Embedded Metric Warehousing for Notebooks</strong>: Write metrics to a local DuckDB file from Telegraf and analyze them in Jupyter notebooks using Python or R. This workflow supports reproducible analytics, ideal for data science experiments or offline troubleshooting.</p> </li> <li> <p><strong>Batch Time-Series Processing on the Edge</strong>: Use Telegraf with DuckDB on edge devices to log metrics locally in SQL format. The compact storage and fast analytical capabilities of DuckDB make it ideal for batch processing and low-bandwidth environments.</p> </li> <li> <p><strong>Exploratory Querying of Historical Metrics</strong>: Accumulate system metrics over time in DuckDB and perform exploratory data analysis (EDA) using SQL joins, window functions, and aggregates. This enables insights that go beyond what typical time-series dashboards provide.</p> </li> <li> <p><strong>Self-Contained Metric Snapshots</strong>: Use DuckDB as a portable metrics archive by shipping <code>.duckdb</code> files between systems. Telegraf can collect and store data in this format, and analysts can later load and query it using the DuckDB CLI or integrations with tools like Tableau and Apache Arrow.</p> </li> </ol>

Feedback

Thank you for being part of our community! If you have any general feedback or found any bugs on these pages, we welcome and encourage your input. Please submit your feedback in the InfluxDB community Slack.