IoT Data Collection and Storage

IoT systems generate data continuously. A single factory with 500 sensors reporting every 10 seconds produces 3 million data points per hour. Managing that volume — collecting it reliably, storing it efficiently, and making it accessible for analysis — is one of the most important challenges in IoT system design.

The Data Lifecycle in IoT

[Sensor reads data]
       |
       v
[Device buffers data locally]
       |
       v
[Data transmitted to gateway or cloud]
       |
       v
[Data ingested and validated]
       |
       v
[Data stored in appropriate database]
       |
       v
[Data queried for dashboards, alerts, and analysis]
       |
       v
[Old data archived or deleted per retention policy]

Data Collection Methods

Polling

The server or gateway regularly asks each device: "What is your current reading?" The device responds with its latest value. The server controls the collection interval.

Best for: devices that store their latest reading in memory, where the server needs data at fixed intervals (e.g., every 5 minutes).

Drawback: if the server is busy or offline, data is missed during that window. Polling also adds network load because the server must contact every device on schedule.

Push (Event-Driven)

The device sends data to the server whenever something happens or on a fixed schedule it controls. The server does not request it — the device decides when to transmit.

Best for: battery-powered devices that send data when they wake up, sensors that detect events (motion detected, door opened), and situations where low latency matters.

Batch Upload

A device collects multiple readings locally and sends them all at once in a single transmission. This reduces the number of network connections, which saves battery life.

Example: A remote weather station collects temperature readings every minute but only transmits a batch of 60 readings once per hour.

Streaming

The device maintains a persistent connection and sends a continuous, real-time stream of data. Used when monitoring must happen with no delay — live video feeds, industrial vibration monitoring, or real-time ECG from a medical device.

Data Types in IoT

Storage Options for IoT Data

Time-Series Databases

Time-series databases are purpose-built for IoT. Nearly all sensor data is time-stamped — each reading records both the value and exactly when it was measured. Time-series databases store and compress these time-stamped values extremely efficiently and make it fast to query ranges like "show me all temperature readings from the last 7 days."

Popular choices:

• InfluxDB: The most widely used open-source time-series database for IoT and monitoring. It includes a built-in query language and data retention policies.
• TimescaleDB: Runs on top of PostgreSQL and adds time-series optimizations. Teams that already use PostgreSQL adopt this easily.
• AWS Timestream: Amazon's managed time-series database service. No server to manage — data scales automatically.

Relational Databases

Relational databases like MySQL or PostgreSQL store structured data in tables with fixed columns. They work well for storing device metadata — device names, locations, owners, configuration settings — but are not optimized for the high-frequency numeric readings that sensors produce.

Best for: device registry, user accounts, configuration data — the structured non-sensor parts of an IoT system.

NoSQL Databases

NoSQL databases like MongoDB or DynamoDB store flexible documents — each data record can have a different structure. This suits IoT data that varies by device type. A temperature sensor's record looks different from a GPS tracker's record, and NoSQL handles that without requiring a fixed schema.

Best for: mixed device types, IoT data with variable structure, or rapid prototyping where the data schema changes frequently.

Object Storage

Large IoT files — video, images, audio recordings, firmware update files — go into object storage. Services like Amazon S3, Google Cloud Storage, or Azure Blob Storage hold these files cheaply and durably. A camera sends video clips to object storage; the IoT platform stores only a reference (URL) to each clip in the time-series database.

Local Edge Storage

Devices or local gateways often buffer data locally before sending it to the cloud. This protects against data loss during network outages. An edge device might store 24 hours of sensor readings on a microSD card and upload them when connectivity is restored.

Data Retention and Management

IoT systems produce enormous volumes of data. Storing everything forever is expensive. A structured data retention policy defines:

• How long raw data stays in the hot store (fast, expensive storage) — typically 7–30 days
• How long aggregated data (hourly or daily averages) stays in warm storage — typically 6–12 months
• When data gets archived to cold storage (very cheap, slow to access) or deleted

Day 1–30:    Raw readings every 30 seconds   [Hot storage]
Day 31–365:  Hourly averages only            [Warm storage]
Day 366+:    Daily averages or delete        [Cold storage / Archive]

Data Compression and Aggregation

Before data moves from the device to the cloud, it can be compressed or aggregated to reduce the volume transmitted. Instead of sending 3,600 individual temperature readings every hour, the device sends one average, one minimum, and one maximum reading — three numbers instead of 3,600. Aggregation dramatically cuts data transmission costs and storage requirements.

A Practical Architecture: Environmental Monitoring Network

[ 50 Air Quality Sensors ]
       |
       | (MQTT over LoRaWAN, every 10 min)
       v
[ LoRaWAN Gateway ]
       |
       | (Forwards to cloud via HTTPS)
       v
[ Cloud MQTT Broker ]
       |
       +---> [ InfluxDB ] <-- stores numeric time-series readings
       |
       +---> [ MongoDB ] <-- stores device metadata and config
       |
       +---> [ S3 Bucket ] <-- stores daily report files
       |
       v
[ Grafana Dashboard ] <-- queries InfluxDB to display live charts

Summary

IoT data collection uses four main patterns: polling, push, batch upload, and streaming. Each fits different device types and use cases. Time-series databases handle sensor readings most efficiently. Relational and NoSQL databases store device configuration and metadata. Object storage holds files. Retention policies and aggregation keep storage costs manageable as data volumes grow.