Space weather – Enhanced weather forecasting systems for the Met Office

Overview:

Space weather describes conditions in space that can interact with the Earth’s upper atmosphere and magnetic field and disrupt our infrastructure and technology including power grids, radio communications and satellite operations including GPS.

The Met Office owns the UK space weather risk on behalf of the Department for Business, Energy and Industrial Strategy (BEIS) and as part of the National Risk Assessment mitigation strategy, delivers space weather services to government, critical national infrastructure (CNI) providers, international partners such as ESA, PECASUS, KNMI and the public.

Challenge:

CACI Information Intelligence Group were asked to undertake this project to enhance the existing Met Office space weather forecast systems and the services it delivers to customers. These enhancements include implementing new scientific forecasting models, incorporating new data sources and a full migration of the system into the AWS cloud while maintaining continuous operations.

Space Weather is on the UK’s national risk registry as a high impact, high likelihood event and as such this system needs to be secure and have 24/7/365 high availability.

Key Issues:

• A significant amount of data in different formats, to differing levels of quality, from NASA, NOAA and BGS that were handled as disparate external sources, were costly to maintain and could not be easily updated.
• Complex scientific models that were developed by different domain experts over a period of time, written in varying technologies, that were difficult to run as a component of a production service.
• Consumers that were interested less in complex data outputs of models, and more in what the results meant for them in their own domain, such as power or communications.

Approach:

CACI follow a disciplined Agile methodology agreed with the Met Office teams with whom we work. For this project, we needed to rapidly stand up a new data science environment and undertake cloud migration, our engineering teams followed the Scrum framework, but also have experience using Kanban and SAFe in other situations.

The Space Weather team consisted of 8 core people (data engineers, software engineers) and 3 rapid response resources (business analysts and software engineers).

In a complex situation, with just a high-level brief to work from, we adopted a highly condensed form of ‘discovery / alpha / beta’ in agreement with the customer. With access to existing data sources, models and staff, all members of the team initially ran rapid discovery activities against the three key challenges and refining requirements to give a prioritized backlog of user stories and tasks.

In a series of sprints, we proposed and implemented appropriate solutions in each of these areas. We adjusted the delivery approach as we went to fit the customer’s needs: streamlining our sprint planning meetings by having interim backlog grooming sessions and by regularly standing up demonstrations of the work we have developed. Successfully using agile principles and evolving agile techniques in this way meant that development velocity is high despite complex requirements and geographically distributed teams.

We also agreed best in class tooling with the customer, for engineering a cloud-based data pipeline and models, including MongoDB, Java, Spring, Apache Camel, AWS (Lambda, SQS, SNS, S3, API Gateway, Fargate, CloudWatch, EC2), for front end development (Angular, HighchartsJS). Using this approach, the team have recently designed and implemented an improved platform for building and deploying scientific models into operations, delivering an enterprise-ready service in close collaboration with a wide range of scientists, academics, and organisations.

What CACI provided:
• A production-scale data pipeline capable of being configured to ingest a wide variety of data formats. This includes the original sources external to the Met Office, and also a number of internal sources including complex scientific models, the ‘supercomputer’ results and forecaster analyses.
• A set of robust scientific models running as a service on AWS, such as the OVATION Aurora nowcast and forecast.
• Front end applications that allow the customer to perform qualitative analysis and predict space weather events, providing alerts, warnings, and forecasts to a diverse range of customers to allow them to take mitigating actions relevant to their domain.
• A capability for transitioning complex scientific models into an operational environment, in close collaboration with Space Weather scientists and other expert users.

Outcome:

• A single, cloud-hosted data pipeline to handle 50+ large, disparate, real-time data sets from a wide variety of sources, making a robust and extensible service to reliably and efficiently feed a productionised set of cloud-hosted models, feed an automated alerting system and multiple clients directly.
• This service is now consumed 24/7/365 by the Met Office Space Weather Operations Centre and other consumers, allowing Met Office users to make informed operational decisions using specific graphs displaying geospatial and Space Weather data e.g. predictions of Coronal Mass Ejections and geomagnetic activity, allowing a range of consumers to more readily interpret space weather e.g. interruptions to power grid, GPS and (for MOD) over the horizon communications.

Contact Us

If you have any questions or want to learn more, get in touch today.