AI Diffusion Model Unveils Hidden Rooftop Solar

New AI diffusion model instantly reveals hidden rooftop solar

The conditional denoising diffusion probabilistic model (cDDPM) can reconstruct the hourly output of behind‑the‑meter (BTM) PV systems from just 30‑minute smart‑meter data, giving distribution system operators (DSOs) a clear picture of otherwise invisible solar generation. By generating multiple plausible PV profiles and attaching statistical confidence intervals, the method turns low‑resolution consumption data into a probabilistic map of rooftop solar.

The research team at Spain’s Universitat Politècnica de Catalunya built the model after a Spanish DSO asked for a way to estimate voltage volatility without knowing which customers host PV panels. The cDDPM therefore acts as a “service enabler” – once the DSO knows where solar is installed, it can run any number of grid‑management tools, from voltage‑control to demand‑response scheduling.

How the conditional diffusion model works

cDDPM learns the distribution of daily PV generation by conditioning on three easy‑to‑obtain inputs: net consumption from the smart meter, local solar irradiance, and location‑specific features (e.g., latitude, roof orientation). During inference the model starts from random noise and iteratively denoises it, guided by a 1‑D U‑Net neural network that captures temporal patterns across the day. The result is a set of synthetic PV curves that respect the physics of solar production while matching the observed net load.

Because the model is conditional, adding more relevant information – such as yesterday’s consumption – tightens the uncertainty bands. The authors showed that pairing irradiance with both current and previous‑day net consumption yields the best reconstruction accuracy.

Sources: the original study in Energy and AI (pv magazine), the detailed methodology in the UPC pre‑print ScienceDirect, and the accompanying technical report UPCommons.

Proven accuracy on real‑world data sets

The model was trained and tested on two public residential data sets:

• Australia – 257 homes with a full year of half‑hourly smart‑meter and PV output data.
• Netherlands – a separate set used to check cross‑region generalisation.

Across both data sets cDDPM outperformed classic physics‑based disaggregation, proxy‑based methods, ensemble learners, quantile regression, and even conditional variational auto‑encoders. Deterministic metrics improved, with a lower RMSE compared with the best deterministic benchmark, and probabilistic metrics such as Prediction Interval Coverage Probability (PICP) met the desired coverage while keeping the interval width narrow (low PINAW). Calibration plots confirmed that the uncertainty bands were reliable, especially during sunny periods when PV output dominates the net load.

See the benchmark results in the authors’ conference paper KAUST repository and the IEEE PES working‑group summary IEEE PES.

Why uncertainty quantification matters for DSOs

Grid operators need more than a single point estimate of hidden solar; they must know how confident the estimate is to avoid over‑ or under‑reacting. cDDPM’s probabilistic output lets a DSO:

1. Plan voltage control with a known risk margin, reducing the chance of over‑voltage events.
2. Allocate flexibility resources (e.g., demand‑response or storage) only where the solar forecast is tight enough.
3. Design tariffs that fairly charge customers whose net load is heavily masked by self‑consumption.

The ability to attach a 95 % confidence interval to each reconstructed PV curve is a step forward compared with deterministic disaggregation, which can mislead operators during cloud‑burst events.

Extending the approach to batteries and EVs

The authors note that adding battery data to the training set would let cDDPM separate PV generation from storage charging/discharging. In homes without batteries, conditioning on irradiance alone already yields competitive performance. For electric‑vehicle (EV) charging, the impact on disaggregation is modest because most residential charging occurs after sunset, when PV output is near zero. However, as smart‑charging aligns EV load with midday solar, the model would need extra variables such as electricity price signals or user‑defined charging windows to keep its uncertainty low.

Related insights on BTM batteries and flexibility are discussed in Ember’s EU outlook Ember.

What it means for Israel

Israel’s distribution grids are already grappling with a rapid rise in rooftop solar – a significant share of single‑family homes now host panels. Because most installations are behind the customer meter, the Israeli Electricity Authority and grid operator NOGA lack precise visibility, which hampers voltage management and accurate load forecasting.

If a DSO in Israel adopted cDDPM, it could identify the hidden 10 kW‑peak system typical of a central‑Israel home. Using the verified Israeli figures:

• Annual generation ≈ 17,000 kWh (central region yield 1,700 kWh/kW · 10 kW).
• At the residential feed‑in tariff of ₪0.48 /kWh, that solar output translates to ₪8,160 per year of avoided electricity purchases for the homeowner.
• Recognising hidden capacity across a feeder could reduce midday load substantially, easing voltage stress and potentially deferring costly network upgrades.

A simple pay‑back illustration shows the homeowner’s economics: a 10 kW system costs about ₪31,500 (₪3,150 /kW) and earns ₪8,160 per year, yielding a ≈ 3.9‑year payback even without subsidies – the same calculation used in the Israeli “worked example.” When the DSO can reliably flag these systems, it can design more accurate net‑metering tariffs, target incentives where they matter most, and better coordinate with emerging battery or EV flexibility.

For readers who want to run their own numbers, our solar ROI calculator (/calculator) and market‑data page (/data) let you plug in the typical Israeli inputs.

Outlook: scaling BTM visibility worldwide

The cDDPM framework demonstrates that deep‑generative AI can turn coarse smart‑meter data into a high‑resolution picture of distributed solar. As more utilities digitise their metering infrastructure and as batteries and EVs become commonplace, the same conditional diffusion approach can be extended to jointly disaggregate multiple flexible resources. Researchers are already exploring multi‑resource conditioning (PV + battery + EV) and incorporating price signals to keep uncertainty low in increasingly complex demand‑side landscapes.

In the next few years we can expect:

• Standardised uncertainty reporting for BTM DERs, endorsed by bodies such as IEEE PES and the European Union’s distribution‑system guidelines.
• Open‑source toolkits that let DSOs plug in local irradiance and consumption data without needing proprietary datasets.
• Regulatory incentives that reward utilities for improving visibility, because better forecasts directly translate into lower curtailment costs and deferred grid reinforcement.

For Israel, the message is clear: adopting probabilistic disaggregation like cDDPM will sharpen the picture of rooftop solar, help meet the 30 % renewable electricity target for 2030, and give households a clearer path to self‑sufficiency.

---

FAQ

• What is cDDPM? It is a conditional denoising diffusion probabilistic model that generates many plausible PV output curves from low‑resolution smart‑meter data, providing both point estimates and uncertainty bands.
• Why can’t DSOs just use the smart‑meter data as‑is? Smart meters record net consumption, which mixes PV generation with load. Without disaggregation the DSO cannot tell how much of the net reduction is due to solar versus reduced demand.
• How accurate is the method? In Australian and Dutch test sets it achieved a lower RMSE compared with the best deterministic benchmark and provided 95 % coverage of its prediction intervals.
• Will it work for homes with batteries? Yes, but the model needs training data that include battery charge/discharge patterns; otherwise irradiance‑only conditioning still gives decent results.
• Can Israeli utilities adopt this today? The methodology only requires existing half‑hourly smart‑meter data, local irradiance forecasts and basic location attributes – all of which are already available to Israeli DSOs.

---

Key Facts

• cDDPM reconstructs hidden PV output from 30‑minute smart‑meter data with 95 % confidence intervals.
• In Australian tests it achieved a lower RMSE compared with the best deterministic benchmark.
• A typical 10 kW Israeli rooftop system yields ~17,000 kWh/year, worth ₪8,160 at the residential tariff.
• The model can be extended to include batteries and EVs by adding extra conditioning variables.

Sources & further reading

• Conditional diffusion modeling for probabilistic behind-the-meter PV...
• Conditional diffusion modeling for probabilistic... - UPCommons
• Conditional denoising diffusion probabilistic model based ante-hoc...
• Conditional Denoising Diffusion Probabilistic Models for Data... - arXiv
• A Conditional Denoising Diffusion Probabilistic Model for Point...