Balancing costs have been increasing significantly over the past three years and are forecast to increase even further out to 2030. There are many factors which influence increasing balancing costs. Identifying what system conditions lead to higher-cost outcomes, and which of these have the most significant effect, is vital to improve control room decisions and to ensure the ESO’s balancing cost reduction strategy is fit for purpose. This project will deliver a method to quantify the probability that certain conditions will lead to high balancing costs, and a more detailed causal and statistical analysis will then be completed for the most impactful factors identified. If successful, the methodology will be used to produce a prototype tool that can identify the probability of high balancing costs outcomes and inform the control room on how to best mitigate these conditions.
Benefits
This project will quantify the major behaviours and conditions that influence balancing costs and will create clarity for the ESO control room in their decision making. Balancing costs are currently at around £3 billion per year and are forecast to continue rising. Understanding how different factors impact balancing costs more specifically will be a key benefit from this project, for example if there was just one interval in which the ESO were to reduce the impact of interconnector swings it could save millions in balancing costs. Identification of these key factors that influence balancing costs will also provide input into the ESO balancing cost strategy, allowing for prioritisation of actions that will have the greatest impacts on these costs.
Learnings
Outcomes
The Causal Analysis of Balancing Costs project successfully delivered on its primary objectives, with some scope adjustments in the final phase.
The Outcomes of the Project
The project delivered the following outcomes:
• A documented and replicable methodology to identify and quantify the key drivers of balancing costs using structured statistical and causal analysis.
This provides NESO with a repeatable framework that can be updated as system conditions evolve and applied to other cost categories.
• Identification and systematic analysis of multiple influential factors, including wind generation, demand levels, spot pricing, constraint flows and limits, outages, interconnector fluctuations, and wholesale market conditions.
These drivers were quantified and ranked, providing clearer evidence of which operational conditions most strongly influence balancing cost outcomes.
• Prioritisation of renewable generation (in particular wind forecast levels) as a dominant structural driver.
The analysis confirmed that high wind penetration significantly reduces wholesale prices (e.g. from ~£94/MWh to ~£12/MWh during the study period) but can increase balancing costs under constrained network conditions due to curtailment and congestion. This clarified the interaction between market outcomes and real-time system costs.
• Development of a probabilistic framework capable of estimating how the likelihood of high balancing cost outcomes changes under different system conditions.
This enables forward-looking assessment of high-cost risk rather than purely retrospective analysis.
• Delivery of an open-source analytical codebase supporting data processing, modelling and visualisation.
While the final integrated forecasting tool was not completed within the project timeframe due to resourcing constraints, the analytical components and implementation roadmap were produced, enabling internal continuation by NESO’s AI Centre of Excellence.
• Earlier completion of the project following formal change control, with remaining budget associated with the undelivered tool component recovered.
This demonstrates appropriate governance while preserving the core analytical outputs.
The project’s findings are informing NESO’s balancing strategy and are being considered alongside related initiatives, including dispatch transparency and wind forecasting improvements.
Lessons Learnt
Technical Learnings
The project provided quantified evidence on how renewable generation and system conditions influence balancing costs. It confirmed, with data-driven analysis, that renewable penetration, transmission constraints, interconnector behaviour, outage rates, and market conditions all have measurable impacts under specific system states.
A key learning was the importance of analysing interactions between variables rather than considering them in isolation. For example, wind forecast levels interact strongly with transmission constraints and wholesale prices, particularly under high-penetration regimes. This reinforced the need to assess system variables in combination when evaluating cost risk.
The project also demonstrated the value of moving beyond correlation-based analysis. Applying structured causal modelling techniques provided clearer insight into which factors genuinely drive balancing costs and under what conditions their effects are amplified.
Combining distributional profiling, nonlinear analysis and uncertainty testing enabled identification of threshold regimes and high-cost risk states that would not have been apparent through simpler analysis. This approach strengthened confidence in the robustness of the findings.
The feasibility work on the probabilistic framework showed that integrating multiple data sources — including wind forecasts, demand forecasts, constraint flows and market indicators — into a unified analytical model is achievable using open-source tools. Historical back-testing increased confidence that the framework captures system states associated with elevated balancing costs.
Process and Collaboration Learnings
Close collaboration between NESO and the academic partner was important in maintaining methodological rigour while ensuring relevance to operational needs. Regular engagement helped steer the analysis and allowed early identification of issues.
The involvement of an experienced NESO data scientist within the project team proved particularly valuable. Internal expertise supported effective data selection, interpretation of modelling outputs, and alignment with operational context. This highlighted the importance of maintaining in-house capability when working with external research partners.
Reordering elements of the analytical programme to test the deep-dive methodology before applying it more broadly improved overall robustness without altering scope. This demonstrated the value of flexibility in innovation project delivery.
The removal of the final tool deliverable through formal change control highlighted the importance of contingency planning for key-person risks. Transitioning development to internal teams ensured that the analytical foundation and learning were retained beyond the formal NIA project.
Organisational Learning
Inter-project Synergy:
By design, this project aligned with other NESO innovation projects (e.g., those on solar nowcasting (NIA2_NGESO002) and dispatch transparency methodology (NIA2_NESO092)). A key learning is the value of structured cross-project dialogue. For example, the Balancing Costs team consulted with the Market and Interconnectors Costing project (Managing Large Interconnector Moves – NIA2_NESO119) to avoid duplicating work and to ensure consistent definitions and assumptions.
This coordination helped ensure that the outputs of this project could be combined more effectively with related initiatives, such as using wind forecasting improvements from other projects to further reduce balancing costs. The experience demonstrates that early and ongoing collaboration across innovation projects strengthens consistency, reduces duplication, and increases the practical value of findings.
