sensor-tools

Overview

sensor-tools is a fast, stream-oriented command-line toolkit written in C/C++ for working with sensor data files. It handles the full post-collection workflow — format conversion, filtering, error detection, statistical analysis, and real-time monitoring — so researchers can move quickly from raw readings to actionable insight.

Why It Matters

Biodiversity monitoring networks can generate thousands of sensor readings per day across dozens of nodes. Manually inspecting this data for errors, gaps, or anomalies is impractical. sensor-tools automates the heavy lifting — filtering out known hardware errors (e.g. DS18B20 power-on reset values), identifying temporal gaps in sampling, and producing summary statistics — letting ecologists spend their time on analysis rather than data wrangling.

Commands

transform

Convert between JSON and CSV formats, filter by date range, remove error readings, and stream-process large files.

# JSON to CSV
sensor-data transform -F csv input.out output.csv

# Filter by date range
sensor-data transform --min-date 2026-01-01 --max-date 2026-01-31 input.out

# Remove error readings
sensor-data transform --remove-errors input.out output.out

# Recursive directory processing
sensor-data transform -r -e .out /path/to/logs/ output.csv

count

Count readings with optional grouping by column value or time period.

# Count with filters
sensor-data count --remove-errors input.out

# Count by sensor
sensor-data count --by-column sensor_id input.out

# Count by day
sensor-data count --by-day input.out

distinct

List unique values in a column, optionally with frequency counts.

# Unique sensors with counts
sensor-data distinct sensor -c input.out

stats

Calculate descriptive statistics including min, max, mean, median, standard deviation, quartiles, outlier detection, and sampling-rate analysis.

# Statistics for all numeric columns
sensor-data stats -c all input.out

# Clean statistics (exclude errors and empty values)
sensor-data stats --clean input.out

latest

Find the most recent timestamp for each sensor — useful for checking which nodes are still reporting.

sensor-data latest -r -e .out /path/to/logs/

list-errors & summarise-errors

Detect and summarise known sensor error patterns such as DS18B20 communication failures (value = 85) and disconnections (value = −127).

sensor-data summarise-errors -r -e .out /path/to/logs/

sensor-mon

Interactive ncurses-based sensor monitoring with live graphs.

sensor-plot

Historical data graphing tool for visualising sensor readings over time.

Key Features

• Stream processing — handles large files without loading everything into memory
• Recursive directory processing with configurable depth limits
• Automatic input format detection (JSON / CSV)
• Flexible date filtering (ISO 8601, UK format, Unix timestamps)
• Built-in sensor error detection for DS18B20 hardware faults
• --clean shorthand combines error removal, deduplication, and null filtering
• Follow mode (-f) for live count and stats updates from streaming input
• Bash autocompletion
• Debian packaging for easy deployment on Raspberry Pi nodes

Installation

From Source

make
sudo make install

Debian Package

dpkg-buildpackage -us -uc -b
sudo dpkg -i ../sensor-tools_*.deb

Source Code

sensor-tools is open source under the GPL-3.0 licence. View on GitHub.