Forecasting unmetered solar energy with AI aims to improve visibility and forecasting of unmetered solar generation. This will allow UK Power Networks to better anticipate network power flows. This should deliver lower flexibility procurement costs, reduced renewable curtailment and better-informed long-term network planning, resulting in lower costs for end users. The project will develop a machine learning (ML) algorithm to infer the capacity of unmetered solar generation installed behind substations. Capacity estimates feed into a solar forecast algorithm to produce forecasts of unmetered solar generation at primary substations. This forms an important input to the forecasting team’s system modelling. This capacity is currently hidden from UK Power Networks’ forecasting capabilities, and having improved estimates will allow better understanding of rooftop solar capacity growth to help calibrate strategic reinforcement planning.
Benefits
The total forecasted benefits including social return on investment for UK Power Networks are £901k over RIIO-ED2 and £2,024k over ED3.
The quantifiable financial benefit arises from increased efficiency in flexibility markets and the resultant lowering of costs to support the distribution network where there is high renewable penetration. The quantifiable social return on investment arises from the reduction of curtailment of renewable generation.
Quantified Financial Benefit
The financial benefits for this project are forecasted to be £370k across RIIO-ED2, and £823k through ED3. The details and assumptions of these calculations is below.
Reduction in flexibility procurement
The UK Power Networks DSO’s forecasting team uses forecasts of renewable generation across the UK Power Networks regions to procure flexibility services. Through having access to more accurate forecasts, the DSO will be able to estimate likely constraints more accurately. This will allow the DSO to procure and dispatch flexibility services on fewer occasions, and at lower volumes while maintaining the same risk appetite.
Open Climate Fix’s solar forecast work with NESO reduced the demand error by 7%. Given that this project expands on this work by also improving the capacity estimation, this would likely be a conservative estimate. Based on this, UK Power Networks should be able to purchase flexibility more precisely by this amount. However, solar is not always the driver of flexibility requirements, and there is already good solar forecasting for metered capacity. We have therefore estimated that 15% of the improvement in error is likely to be attributed to the new solution, providing 1.05%
UK Power Networks’ Flexibility Dispatches Dataset shows that oversupply flexibility services cost over £500,000 over a four-month period (at time of writing), or around £5.4m per year. Flexibility is expected to grow as renewable generation increases on the existing infrastructure, so we have assumed an annual growth rate from 2025 to 2028. If we assume a modest efficiency gain of 1.05% on flexibility procurement, this delivers the cumulative savings outlined above.
Quantified Societal Benefits
Social benefits for this project are forecasted to total £531k across RIIO-ED2, and £1,201k through ED3 This corresponds to a social return that is 12.37 times the original investment. The details and assumptions for these calculations are below.
Reduced Curtailment
Improved forecasting will enable the DSO to more smartly manage distributed energy resources (DERs), including distributed energy resources management system (DERMS) settings and dynamic trim limits. These will enable the DSO to reduce the amount of curtailment that it enforces on generators providing a financial benefit to generators and a carbon reduction saving.
The benefits of reduced curtailment have been modelled by taking the total reduction in curtailment due to dynamic outage management, and then applying a 1.05% scaling factor to account for the proportion of this reduction that will likely be attributable to the new service. This is then multiplied by the carbon benefit and energy price (£85/MWh).
Carbon savings are worth £70K and £261k in RIIO-ED2 and ED3 respectively.
Non quantified benefits:
Tailored forecasts
To enable whole system coordination, the service should be tailored to UK Power Networks. This requires regional aggregations of renewable generation at the primary and secondary level that are probabilistically coherent with each other and with demand forecasts. This means probability distributions must be aligned or correlated, not simply aggregated. Forecasts for individual weather ensemble members can be used to align with UK Power Networks’ internal forecasts. The control room will have access to a user interface (UI) which can be deployed for granular real-time visibility.
Behind the meter renewable capacity
There is a significant amount of renewable generation behind the meter. A significant proportion of this embedded renewable capacity is unmetered or invisible to UK Power Networks. At the national level the uncertainty in the estimated total capacity of installed solar photovoltaic (PV) is the most significant source of error when estimating national or regional solar PV generation. To quote a recent paper from Sheffield Solar, “We find that the capacity error, at ±5%, dominates the yield calculation error, at < ±1% and leads to an overall error in GB solar PV output estimates of ±5.1%” [https://www.sciencedirect.com/science/article/pii/S1364032121012636]. This uncertainty nationally of up to 10% would tend to increase as you move to smaller aggregation levels as the averaging of noise across multiple regions is reduced. Therefore, at grid supply point (GSP) or primary level, the error could be double that figure. The project includes research and development to improve the energy accounting of renewables in the UK Power Networks’ regions. The research uses half-hourly substation load figures and the modelling of expected renewable generation given known weather conditions to infer the behind the meter renewable capacity in each primary and secondary substation area. This should have a significant impact on the accuracy of the end forecast as well as giving UK Power Networks a much-improved capacity map.
NESO – better grid connections & grid awareness
The outputs of this project (the capacity map) could be used as part of improving the overall energy accounting of renewable capacity in the UK Power Networks’ region. These forecasts and capacity estimates could aid the NESO’s grid awareness, allowing better estimates of renewable generation, better forecasts and ultimately better decisions on grid connections.
Learnings
Outcomes
To date, the project has successfully demonstrated that machine learning techniques can be used to materially improve visibility of behind‑the‑meter solar generation across the distribution network. The project has delivered a validated and scalable methodology capable of inferring embedded solar photovoltaic capacity at primary substation level using a combination of historical demand data and meteorological inputs.
A key outcome is the transition from research and prototyping into a live operational capability. A production data platform, API and prototype user interface have been deployed, enabling UK Power Networks to access embedded solar capacity estimates and associated forecasts programmatically and visually. This represents a significant step towards operational adoption and real‑world decision support.
The project has also established a practical approach to generating primary substation‑level solar forecasts by scaling GSP forecasts using inferred embedded capacity. Back‑testing has been completed from December 2024 onwards, providing UK Power Networks with a robust dataset to support ongoing assessment of forecast performance and operational value.
The quantitative data, which will validate the efficacy of these capabilities, are not yet available. The activities required to provide this validation will begin in the final stages of the project. It should be noted that the full impact of the service will not be known unless and until it is deployed into production and is sufficiently utilised in flexibility forecasting. This is planned to take place beyond the lifetime of the project. However, the impact and associated benefits will continue to be tracked.
Lessons Learnt
The delivery of this project has generated several valuable lessons that will inform future innovation activities, particularly those involving advanced data analytics and machine learning applied to operational energy data.
The need to rescope at the beginning of the project has provided two valuable lessons learnt:
Circumstances can change over time, therefore ensuring that processes around elements such as procurement are as efficient as possible is critical to ensure that projects are not delayed, and learnings can be achieved swiftly
Working more closely with existing vendors to ensure we are aware of their product roadmaps could limit time and effort being spent on projects where there is already likely to be a market offering in the near term
Several additional lessons have also been learnt which may be useful to future projects utilising energy data in the development of machine learning models:
Understanding data conventions and operational context is critical. Initial analysis identified several non‑intuitive characteristics within primary transformer datasets, including negative power values arising from measurement conventions rather than data quality issues. Early collaboration between data scientists and operational subject‑matter experts proved essential in correctly interpreting these behaviours and ensuring that data preparation activities did not inadvertently distort meaningful signals.
Model selection must align with the end use case, not just raw accuracy. While some machine learning approaches demonstrated strong short‑term forecasting performance, they lacked sufficient sensitivity to weather‑driven features required for robust embedded generation estimation. The adoption of a TFT model demonstrated the value of selecting architectures that balance predictive accuracy with interpretability and feature attribution, particularly when scenario‑based analysis is required.
Scalability and computational efficiency should be considered from the outset. During model evaluation, certain architectures exhibited high computational and training overheads, limiting their suitability for large‑scale deployment. Early assessment of scalability constraints helped guide the project towards solutions capable of supporting network‑wide application across hundreds of substations.
Progressive validation builds confidence in innovative methodologies. Employing multiple, complementary validation techniques, including year‑on‑year consistency checks, synthetic data perturbation, aggregation comparisons and manual inspection, proved effective in building confidence in novel estimation approaches where direct ground truth data is limited. This layered validation strategy is recommended for future innovation projects involving inferred or model‑derived outputs.
