Learnings

Outcomes

The outcomes of the project to date are:  

• The national solar generation forecast developed in the course of this project is 2.8 times better than the existing ESO PV forecast (for forecasts up to two hours ahead). The best national PV forecasts developed to date have a mean absolute error (MAE) of 233, compared to the ESO's existing national solar PV forecasts which have an MAE of 650 MW. 
• A prototype of the PV Nowcasting service has been developed based on the needs of key stakeholders and end users of the forecasts. A demonstration session has been held in June 2022 and a feedback session is scheduled for July 2022. 
• The first version of a fully operational PV Nowcasting service was delivered to the control room users in December 2022. As part of the further work extension, the project is now investigating the addition of further features and improving the accuracy of the forecasting algorithm by at least 20%. 

Lessons Learnt

In the course of W1 and WP2, the project identified the following lessons:  
  

• Never underestimate the importance of cleaning up and checking data in advance  
   Several approaches to loading data were tried, from on-the-fly to pre-preparing, and instituted automatic and visual tests of the data to ensure the project was always lining up the various data sources correctly. 
• Having infrastructure as code allows the main production service to run uninterrupted  
  Having code to easily instantiate infrastructure is very useful to the efficient management of environments to ensure the project could bring the algorithm into productive use. The Terraform software tool was used which makes spinning up (and down) environments very easy and repeatable. Being able to spin up new environments allowed the project to test new features in development environments while allowing the main production to keep on running uninterrupted.  
• Using Microservices to “start simple and iterate” accelerates development  
  Using a microservice architecture allowed the project to upgrade individual components as we see benefit in improving them, independently of changing other components’ behaviour. This has been very useful when building out the prototype service, as it has allowed the project team to start with a simple architecture - even a trivial forecast model - and iteratively improve the functionality in the components. For example, first starting out with one PV provider of data has allowed the project to get a prototype working, and in WP3 we will expand to onboard an additional PV provider. 
• Data processing may take longer than expected  
  While it was initially planned to extend our dataset back to 2016 for all data sources during WP2, it turned out that data processing takes much longer than expected. This does not have a direct impact on project deliverables but is something to consider in further ML research. 
• Data validation is important  
  For both ML training and inference, using clear and simple data validation builds trust in the data. This helps build a reliable production system and keeps software bugs at a minimum. 
• Engaging specialist UX/UI skills is important  
  By acknowledging that UX and UI design is a specialised area and incorporating those skills, a UI has been developed which will be easier to use and convey information effectively. This will be validated over WP3 through working with the end users.  
• Building our own hardware demonstrates value for money but may pose other challenges for a small team  
  Two computers have been built during the project with a total of six GPUs and it is estimated that using on-premises hardware instead of the cloud for data-heavy & GPU heavy machine learning R&D can significantly reduce the direct costs. However, the time it would require for a small team to put together all the components is significant (approx. 25 days for one person in total). While the total costs would still be lower, appropriate resource planning should be considered if planning hardware upgrades in the future.  

In the course of WP3 and WP1 (extension), the project identified the following lessons:  

• Merging the code right away when performing frontend testing is of upmost importance 
  Merging the code after frontend testing  proved to be time-consuming and it is something important to consider when performing tests.
• Large Machine Learning models are harder to productionise  
  Large Machine Learning models proved to be difficult to productionise and the size of the model makes it difficult to use. Going forward we need to investigate further how to deploy large models.
• Machine Learning training always takes longer than expected 
  Even with an already made model, making datapipes to work correctly takes time. It is important to always have enough time allocated when planning ML training activities.
• Security and authentication is hard 
  Ensuring the robust authentication/security measures are in place is harder than we envisaged. It may be easier to implement packages already built or contract third party providers to support the process.
• Separate National Forecast Model 
  PVLive estimate of National Solar generation does not equal the sum of PV Live’s GSP generation estimate. This motivates us to build a separate National forecast, compared to adding up our GSP forecast.  
• Investment is needed to take open-source contributions to the next level 
  Time and resources are needed to engage with open-source contributors and develop an active community. We may want to consider hiring an additional resource to support this activity.