Data Scraper

The Data Scraper is the core data collection engine within the Mirox-Agent, actively retrieving real-time information from all monitored equipment at your plant. It connects to your loggers, inverters, meters and battery systems through a library of vendor-specific adapters, normalizes everything into one consistent metric vocabulary, and forwards the result to the rest of the platform — while running a growing set of edge analytics (performance ratio, curtailment tracking, clear-sky baselines, forecasting and network monitoring) and a continuous component health watchdog directly at the plant.

Purpose and Role

The Data Scraper serves a single, focused purpose: actively collect raw measurements from equipment and forward them for processing. It acts as the bridge between diverse manufacturer equipment and the unified Mirox platform, translating proprietary data formats into standardized metrics.

Core Responsibilities:

• Connect to data loggers and monitoring devices through vendor-specific adapters
• Retrieve raw measurements on configurable schedules
• Transform manufacturer-specific data into standardized metric format
• Automatically discover and track installation components
• Monitor component activity and operational status
• Continuously watch component health and connection status, and raise park events for outages, measurement conflicts and connection issues
• Run edge analytics (expected power, performance ratio, curtailment, clear-sky and forecasts)
• Inspect the plant's local network and audit device access
• Forward metrics to the Time-Series Database and Digital Twin service
• Report component health and operational status to the IoT Cloud

This separation of concerns keeps the Data Scraper lightweight, focused, and independently deployable.

Architecture Overview

The Data Scraper operates as an asynchronous, event-driven service where multiple data collection tasks run concurrently:

Key Architectural Principles:

• No database of its own: No analysis state survives restarts; undelivered data is buffered on local disk until it can be sent
• Adapter-Based: A dedicated, vendor-specific adapter speaks each device's protocol
• Self-Healing: Automatic error recovery with exponential backoff
• Concurrent: Each data source collected independently
• Edge-Deployed: Runs on or near the plant, close to the equipment it reads

The Adapter System

Concept

An adapter is a vendor-specific connector module that knows how to communicate with one particular family of device. Each adapter is hand-built and reverse-engineered against that device's own web, API, or database interface — there is no single "speaks-any-protocol" engine. The adapter system is the core extensibility mechanism: supporting a new device means adding a new adapter, which we do on request (see below).

Each adapter is a self-contained module responsible for:

1. Connection Management - Establishing and maintaining communication
2. Data Retrieval - Fetching measurements using the appropriate protocol
3. Data Transformation - Converting to standardized metric format

All adapters inherit from a base class that provides health monitoring, automatic retry logic, exponential backoff on failures, metric validation, and status reporting to the IoT platform.

Health Management

Each adapter implements an automatic state machine:

• INITIALIZING: Starting up and establishing initial connections
• HEALTHY: Operating normally with successful data collections
• UNHEALTHY: Experiencing errors but attempting to continue
• RECONNECTING: Performing recovery actions after repeated failures
• FROZEN: The device is returning stale data — the same values repeat, or no new reading has arrived within the expected window
• PAUSED: Temporarily paused by user command; resumes automatically when the pause expires

The system automatically transitions between states, reports status to the platform, and attempts recovery without manual intervention. The FROZEN state is what lets the platform tell a genuinely-down logger apart from one that is merely repeating a stuck value.

Supported Devices

The Data Scraper ships with around twenty vendor-specific adapters, each built for a particular device family. The list below reflects what is supported today and grows whenever a new device is integrated.

Creating New Adapters

New adapters can be developed to support additional devices or protocols. The modular design and base class functionality significantly reduce development time.

Compatibility with Legacy Equipment: We can create adapters for older devices that were never specifically designed for data export. As long as the device provides its data in any accessible way—whether through a REST API, web interface, database, file system, or any other mechanism—we can extract and integrate that data into the platform.

Unrestricted Data Collection: Our adapters are not limited to the pre-defined data-export formats that data loggers typically provide. We can collect any data that the device makes available, going beyond the standard set of metrics a manufacturer's logger might expose. If a device has additional diagnostic information, advanced parameters, or hidden data points accessible through its interface, we can retrieve and standardize them.

Onboarding Through the Platform

For a subset of devices, you can bring a logger online from the platform without hand-writing any configuration. An onboarding wizard asks the agent to dry-run-connect to the device and streams the live probe results back to you, so you see immediately whether the connection works before committing it. There are two flavours:

• Zero-config onboarding — the adapter already owns the device's full reading set, so the wizard just shows a read-only live preview and you save. Available today for seventeen device families: Janitza meters, Huawei SmartLogger, Phoenix Contact controllers, Fronius inverters, Solar-Log data loggers, SMA Sunny Central, SMA Power Manager, Sungrow logger, Zebotec, Becker PV-Control (Prometheus), FREQCON battery storage, Wattkraft Parkcontrol, NR Electric battery converters, Linyang/Xieneng container batteries SEG HighPROTEC protection relays, INTILION scalecube batteries and the WAGO PFC200 alarm controller. Several of these — for example Janitza, Fronius and Wattkraft — need no credentials at all; SMA Power Manager and Sungrow logger onboard with device credentials and additionally report their own device inventory (component list, serials, firmware) automatically instead of asking you to type it. The WAGO alarm controller is the one exception to the fast path: nothing on the wire says which program a WAGO is running, so it is never auto-detected — it is added through Add Logger, with the adapter and its integrator profile picked by hand.
• Interactive mapping for generic loggers — some loggers expose arbitrary raw values the platform can't interpret on its own. For these the wizard asks the agent to enumerate every raw value the device exposes (group, name, unit, live sample), the operator maps each one to a known metric, and a mapping-driven dry run previews the exact metrics that would be produced before saving. QReader and Bluelog are onboarded this way.

Metric Standardization

All collected data is transformed into a standardized metric format defined by the platform's metric taxonomy. This ensures consistency across all data sources and enables unified processing downstream.

Metric Structure

Each metric follows a standardized structure compatible with modern time-series databases:

Components:

• Name: Standardized metric identifier from predefined taxonomy
• Value: Numeric measurement in base SI units
• Labels: Key-value pairs for component identification and grouping
• Timestamp: Optional preservation of original device timestamp

Standard Labels:

• Source adapter type and instance number
• Human-readable names
• Component identifiers (inverter ID, string number, etc.)
• Physical location or grouping information

Unit Conventions

All metrics use base SI units regardless of what the manufacturer's device reports:

• Power: Watts (W)
• Energy: Watt-hours (Wh)
• Voltage: Volts (V)
• Current: Amperes (A)
• Temperature: Celsius (°C)
• Irradiance: Watts per square meter (W/m²)

Adapters automatically convert from manufacturer-specific units (kW, MWh, etc.) to these standards during the transformation phase.

Metric Categories

The platform defines 451 standardized metric types organized into 11 families:

Label expansions (per-string, per-phase, per-inverter, and so on) multiply these into far more individual time series at a real plant. For the complete metric taxonomy and definitions, see Metric Collection.

Data Collection Flow

Polling Strategies

Adapters support two polling modes:

Interval-Based (default): Executes every N seconds after the previous collection completes. Simple and responsive to varying collection durations.

Static Time-Based: Executes at fixed intervals from midnight with optional offset (e.g., at 00:01, 05:01, 10:01 for 5-minute intervals with 1-minute offset). Useful for alignment with external systems.

Processing Pipeline

After collection, metrics pass through several processing stages:

Metric Preparation: Source labels are added, timestamps applied, and structure validated.

Filtering: Configured filters can modify values, validate ranges, or skip metrics based on rules.

Calculations: Automated calculators derive additional metrics:

• Solar radiation power integrated to irradiation energy
• String voltage × current calculated to power
• Power values integrated to energy over time

Component Discovery: As metrics flow through, the Data Scraper automatically discovers and identifies installation components. This is a crucial feature—since the Data Scraper is the layer that actively collects data, it inherently knows which components exist and are providing data. The system automatically discovers:

• Inverters (from inverter power metrics)
• String combiner boxes / GAKs (from GAK metrics)
• Individual strings (from string voltage/current metrics)
• Irradiation sensors (from radiation metrics)
• Grid connection points (from grid energy metrics)

Discovered components are synchronized to the IoT platform for inventory management, creating a real-time, self-maintaining equipment registry without manual configuration.

Production Detection: The system monitors plant operational state:

• Detects when production begins based on irradiance and power
• Identifies unexpected shutdowns during production hours
• Reports state transitions for alerting

Metric Grouping: Metrics are batched by time series to optimize database insertion performance.

Data Export

Processed metrics are forwarded to two destinations:

Time-Series Database: Metrics are pushed in batches with rate limiting and retry logic for long-term storage and historical querying.

Digital Twin Webhook: A separate background task continuously forwards the latest metric values to the Digital Twin service (a completely separate microservice) for real-time analysis. The Data Scraper has no knowledge of what the Digital Twin does with the data—it simply provides the metrics. For information about Digital Twin processing, see Digital Twin.

Stateless Operation

The Data Scraper keeps no database of its own and holds no analysis state between restarts:

• Can be stopped and restarted without data loss
• Multiple instances can run independently for different parks
• Each polling cycle is independent of previous cycles
• Crash-resilient with no risk of corrupting persistent state
• Measurements and status reports that cannot be delivered — for example during an internet outage — are buffered on local disk and delivered automatically once connectivity returns, so nothing is lost (the "Local Retention / Automatic Resumption" resilience described in Mirox-Agent Overview)

The only persistent state is the local delivery buffer of not-yet-transmitted data, plus what lives externally:

• Configuration files (version-controlled)
• Time-Series Database (external system)
• IoT Cloud component registry (external system)

This design ensures operational simplicity, reliability, and easy horizontal scaling.

Separation of Concerns

The Data Scraper has a narrow, focused responsibility that enables clear separation from other platform components:

Data Scraper:

• Collects raw measurements from equipment
• Transforms data to standard format
• Discovers and tracks components
• Monitors component activity status
• Watches live component health and connection status, opening and closing park events
• Forwards metrics to other services

Digital Twin: Validates against physics models and detects anomalies and losses

Time-Series Database: Stores historical data, provides query interface

IoT Cloud: Maintains component registry, tracks device status, manages equipment inventory

This separation enables independent development, testing, deployment, and scaling of each component while ensuring each service focuses on its core competency.

Advanced Features

Automatic Health Monitoring

Each adapter implements a state machine that tracks operational health with automatic reporting to the platform and exposure via the metrics API for operational monitoring.

Automatic Component Discovery

The Data Scraper's position as the data collection layer gives it a unique advantage: it inherently knows which components exist at an installation because it directly interacts with the metrics they produce. As metrics flow through the system, components are automatically discovered from metric labels and registered with the IoT platform.

Discovery Process:

1. Metrics arrive with identifying labels (inverter ID, string number, sensor location, etc.)
2. The Data Scraper extracts component information from these labels
3. New components are automatically registered with the IoT Cloud
4. Component metadata (type, identifier, location) is synchronized
5. The platform maintains an up-to-date equipment inventory without manual entry

This self-discovery mechanism ensures the platform always knows what equipment exists at the installation, eliminating the need for manual configuration and reducing deployment time.

Production State Detection

The service monitors plant operational state and detects production starts, unexpected shutdowns during production hours, and state transitions for alerting and analysis, reporting only when the state actually changes. It also watches for overproduction — output above the clear-sky model for a sustained period — which can flag a logger that is returning frozen values and, in that case, fall back to the clear-sky model so the data stream stays sensible. This provides real-time operational awareness beyond just raw measurements.

Calculated Metrics

Several calculators automatically derive metrics from raw measurements—solar radiation integrated to irradiation energy, string power calculated from voltage and current, and power values integrated to energy over time. These calculations happen transparently, enriching the data stream without requiring explicit configuration.

Edge Analytics

Beyond raw collection, the Data Scraper runs a set of analytics directly at the plant, computed from the live metric feed and exported as chartable time series alongside the raw data.

Expected Power and Performance Ratio

The agent continuously computes the expected power for each plant and compares actual production against it as a performance ratio (PR) — a normalized measure of how well the plant converts available sunlight into electricity. Rather than trusting a single input, several independent irradiance sources are monitored in parallel, and each source gets its own PR series — so a drifting on-site sensor can never silently distort your baseline:

• On-site pyranometer — the plant's own irradiance sensors, available near-real-time
• Satellite — satellite-derived irradiance for the site's exact location
• Weather model — modelled irradiance from meteorological data

Conversion to the module plane (POA). All sources are converted to plane-of-array (POA) irradiance — the irradiance actually hitting the modules — using the plant's component hierarchy (the park tree): every string carries its own orientation, an azimuth degree and a tilt, either configured by the operator or detected by the analysis engine. Horizontal irradiance is decomposed into its direct and diffuse parts and transposed onto each distinct module plane with industry-standard solar-position physics, so module fields facing different directions each get their own correct expectation. String expectations are summed into combiner-box and inverter curves and capped at each inverter's AC nameplate.

Anomaly filtering. Before a moment in time may count toward the trustworthy ("clean") PR, a battery of filters masks out every situation in which a low ratio would not mean a fault:

One PR per source. The performance ratio is then calculated separately for each irradiance source — published both raw (every computable value) and clean (only the filtered, trustworthy values) — at park level and per component (inverter, combiner box, string). The trailing average of the clean PR also self-calibrates the expected-power model: the expectation automatically adapts to each plant's real-world efficiency, and this same expectation is what the component health watchdog holds live production against.

Live Curtailment Tracking

When a plant produces less than it could, the agent attributes the foregone production to its cause: curtailment by the marketer (a deliberate market-driven cap) versus curtailment by the grid operator. This distinction matters for loss accounting and contractual reporting. Curtailment is tracked per minute and emitted as both instantaneous power and cumulative energy. See Loss Detection for how curtailment fits into overall loss attribution.

Clear-Sky Baseline and Forecasting

• Clear-sky baseline: a theoretical ideal-conditions PV curve maintained as a long-term record, giving you a stable reference to compare real output against.
• Day-ahead forecast: a short-horizon PV production forecast derived from weather data, so you can anticipate the next day's output. These are weather-physics based, not statistical guesses.

Historical Backfill

When a plant is first connected or after a gap, the agent can backfill historical data — replaying raw readings and re-deriving the analytics above for a requested window, then handing the result to the live pipelines so charts are complete from day one rather than starting empty.

Component Health Watchdog

Beyond collecting data, the agent continuously judges whether every producing component — inverters, combiner boxes (GAKs), strings and irradiance sensors — is actually healthy, and turns its findings into park events you see on the platform. Two questions are asked around the clock: is the component still connected, and is it producing what it should?

Connection Monitoring

Connectivity is tracked as a strict causal chain — park link, network devices, data loggers, components — and the watchdog differentiates the ways a connection can fail, each with its own event:

One alarm per root cause. Each layer only raises events while every layer above it is healthy: a park-wide outage raises exactly one park-level event and silently holds everything below, a dead switch raises a device event, and only when the network path and the logger are provably fine does a silent component become a component fault. All transitions are debounced against reboots and brief blips, and every event closes automatically on recovery, stating the outage duration.

Power and Production Checks

In parallel, the watchdog compares each component's measured power against the expected-power model in short evaluation windows. A zero or shortfall only counts as evidence when an environment gate proves conditions warrant production — sufficient sustained irradiance, no snow, frost or fog, no actively binding curtailment, and a healthy network path — so a dark winter morning can never page anyone.

Before an alarm opens, the finding is cross-examined against independent witnesses: the component's children one level down the hierarchy, its own lifetime energy counter, and a nightly parent-versus-children energy audit. The outcome decides what kind of event you get:

• Production outage — the component really stopped producing under conditions where it should produce
• Measurement conflict — the component is producing, but its own reading is wrong (a data fault to investigate, never counted as lost production)
• No communication — the component stopped reporting entirely; production is unknown and deliberately not counted as loss
• Reduced output — the component is producing, but far below its modelled expectation (judged both live and in a nightly whole-day audit)

Strings and irradiance sensors get staged investigations instead of instant verdicts: a silent string first opens a low-priority investigation and is only confirmed as an outage or defect on physics-grade proof — for example zero output under an overcast sky while its siblings produce — while recurring zeros that follow the sun's path are recognized and recorded as shading, not a defect. A sensor whose irradiance channel goes dark while its other channels stay alive is reported as soiling (clean the dome) rather than a hardware defect.

Inverter Vital Signs

The checks above ask whether an inverter still produces. A second set of checks asks whether a producing inverter is quietly heading for a defect — by reading the signals the agent already collects from every inverter but never used to judge it: temperature, insulation resistance, per-phase AC currents and voltages, and the DC side (voltage and power).

A service track of its own. These findings never arrive as an outage: the inverter is producing, and nothing is lost yet. They open as service items, deliberately kept apart from the outage events above — a producing inverter's routine health reports can never close a service finding, and an open service finding can never hold back a real outage alarm. They also never move a production figure: an inverter carrying a service finding keeps counting as producing wherever the platform counts producing components, because it is producing.

Each finding is an event of its own. There is no generic "maintenance required" entry any more — every finding is reported under its own descriptive name (Elevated temperature, Insulation weakening, Phase imbalance, Efficiency declining), and an inverter carries at most one open event per finding. Several findings can therefore stand side by side on the same unit, each with its own reasoning, its own history and its own recovery, instead of being merged into a single entry that says only "something needs attention".

Watching is visible; the verdict waits. A fresh suspicion no longer disappears into the agent's own bookkeeping. As soon as a check has enough evidence to take an interest, it opens a low-priority observation event — Observing: elevated temperature, Observing: phase imbalance, and so on, carrying the same icon as the confirmed finding in a lighter shade — so the plant's activity list shows exactly what the system is watching, and since when. What such an event deliberately does not do is change a component's health: observations never appear on the component-health views — the analysis and production pages, the component chips, the station components column and its counts — because those state a verdict only where the platform is sure. An observation is a lead, not a verdict.

One chain from first suspicion to finding. When the confirming evidence arrives, the observation is closed and the confirmed finding opens in its place at normal priority, linked back to the observation it grew out of. The two read as a single history — watched from this day, confirmed on that one — and the closing of the observation announces no recovery, because nothing recovered: the finding simply moved up a stage. Only from the confirmation onwards does it appear on the component-health views. If the corroboration never arrives, the observation just closes again when the condition goes away, and nothing was ever claimed.

Service findings name the actual inverter component and carry the same plain-language reasoning as every other event, with the concrete numbers behind them — the measured excess, the peer comparison, the number of days observed. For now they are deliberately silent: they appear in the plant's event list but sit below every notification threshold, while the new checks are calibrated against live fleet data. Inverter Events sets out what opens, confirms and closes each one.

Cause Evidence on Existing Events

Two of the newly read signals raise no events of their own — they strengthen the events that already exist:

AC side or DC side? When the watchdog opens an inverter production outage, it also reads the DC voltage across exactly those daylight windows in which the unit delivered nothing. A dark DC bus means the cause sits on the DC side (strings, DC switchgear); a live one means the array is intact and the fault sits in the unit itself or on its AC side. The verdict — AC-side, DC-side or undetermined — goes into the outage event's reasoning and decides which service call is worth making at all. The witness is vendor-dependent: some devices — Huawei among them — never report a dark bus, so there it can only exonerate the DC side, never accuse it.

A grid-side derate is not a defect. When a single inverter holds its AC voltage at or above 1.10 times its nominal voltage during an underperformance window, it is limiting its output on purpose, following the grid code's volt-watt rule — it is obeying the grid, not failing. Such windows are held neutral instead of being scored as a shortfall, so a stiff grid connection can never accumulate into a false defect.

How Long Until Events Are Detected?

These detectors do not watch continuously. They run once per night — plus once shortly after the agent starts — and rebuild their evidence from the stored history on every pass. A condition that has been standing for weeks is therefore found at the first evaluation after its lane is armed, not weeks later. What takes time is the evidence itself: every check demands a minimum history and a persistence streak before it says anything at all. The first column below is the point at which the observation opens — low priority, visible in the plant's activity list, deliberately absent from the component-health views. The second is what it takes for that observation to be replaced by the official finding, which is what those views then show.

A freshly onboarded plant therefore sees its first temperature observations after about three weeks and its first efficiency observations after about two months, while the DC witness and the grid exoneration work from day one.

Two limits are worth knowing:

• A peer group needs at least six comparable inverters. All of these rules are peer-relative — each unit is judged against its own siblings, never against an absolute threshold, because absolute limits scale with array size and vendor. The consequence is deliberate: if an entire peer group degrades in the same way at the same time, that common-mode drift is not detectable by this method.
• Coverage is per inverter and data-driven. A check arms only where the vendor's logger actually delivers the metric — no configuration switch turns it on. Coverage therefore differs from plant to plant and from vendor to vendor.

Events That Explain Themselves

Every event carries a plain-language reasoning: what was measured, under what conditions, which independent evidence corroborated or contradicted it, and what would resolve it. Events close automatically on proven recovery, are reclassified in place when the evidence changes its mind (never a close-and-reopen flap), and respect your decisions — an event you close stays closed until the fault fully re-earns a fresh one. A single park-level "components down" summary event carries the actual operator alarm: it opens when at least one inverter or combiner box has been down for a sustained period, lists the affected components, and notifies only on first open and on final recovery.

Network Monitoring

The Data Scraper includes a built-in local-network inspector that runs alongside data collection and maps the plant's on-site network. It discovers devices on the configured network ranges by sweeping for reachable hosts, reading the address table, identifying vendors from hardware addresses, and probing devices for their identity. Discovered devices are classified against a large library of known network-equipment profiles, and an AI device-identification step helps recognize device families that simple rules miss.

Once devices are known, the inspector polls them for reachability and health (response time, interface and resource status) and reports the results to the platform. You can trigger or stop a discovery scan, re-check an individual device, and review the discovered network — see Local Network Inspector.

Proxy Auditing

For plants reachable through the Mirox Browser Proxy, the agent audits human access to local device interfaces. It groups each person's activity into sessions, redacts sensitive query data before anything is stored, and can produce an AI-generated summary of what a session did. This feeds the platform's access audit trail — see Access Audit Log.

Performance Characteristics

Typical Performance:

• Polling frequencies: 1-300 seconds per adapter (configurable)
• Concurrent adapters: 20+ running simultaneously
• Throughput: 10,000+ metrics per minute sustained
• Latency: Sub-100ms from collection to database insertion
• Resource usage: 5-15% CPU, 100-500 MB memory on edge hardware

The asynchronous architecture ensures high concurrency without blocking, enabling efficient collection from many sources simultaneously.

Related Features

• Digital Twin — the physics-based analysis engine that consumes the metrics the Data Scraper collects
• Mirox-Agent Overview — how the Data Scraper fits within the wider edge agent
• Deployment Options — on-site versus cloud deployment trade-offs for the agent
• Metric Collection — the full standardized metric taxonomy
• Local Network Inspector — the on-site network monitoring surface
• Loss Detection — how curtailment and other losses are attributed
• Events — how the watchdog's park events reach you