REF Hackathon participants outside the Met Office in Exeter, UK.
Over 90 participants from around the world gathered to co-develop the Rapid Evaluation Framework at a dedicated 4-day hybrid hackathon hosted by the Met Office in Exeter, UK.
The Rapid Evaluation Framework (REF) is envisioned as a community developed, complete end-to-end framework providing a systematic and rapid performance assessment of the next generation of models participating in the Fast Track set of experiments, supporting the next IPCC Assessment Report 7 (AR7) cycle.
Over the past year, the CMIP Model Benchmarking Task Team have organised several community engagement events as part of the REF development process. Such engagement ensures that input from modelling centres, observational dataset providers and model evaluation software developers is integral to the design of the REF, which enables the REF to respond to the needs of, and provide benefit to, the environmental science community.
Last month, the REF delivery team conducted a hybrid Hackathon event, hosted by the Met Office in Exeter, to further engage the community in co-development of the REF. The event, which took place on the 10th–13th March 2025, was launched with a warm welcome from the Director of the Met Office Hadley Centre, Professor Rowan Sutton.
Four separate workstreams were run in parallel, targeting different aspects of the REF development; Metrics development, Infrastructure, Observation data products, and Documentation. The Model Benchmarking Task Team co-leads and the REF development team members shared updates on the progress of the REF across each workstream and engaged with experts and community members in multiple open sessions throughout the week. Software development sessions and hands-on training for running the REF were also provided by members of the development team.
Just over 90 participants registered to participate in the hackathon, including both scientific and technical representatives of modelling centres, observation dataset providers as well as members of WCRP activities whose communities see potential use of the REF as it expands. These included members of the CORDEX (regional) as well as high resolution modelling communities. Holding the event at a modelling centre, involved not only in the CMIP Assessment Fast Track but also across a wide variety of projects within the WCRP modelling multiverse, provided invaluable and timely insights and suggestions as the REF development evolved during the Hackathon.
Ed Blockley, Model Benchmarking Task Team member hosting the hackathon remarked,
“Interaction with CMIP is important for the Met Office and for UK climate science. I was very happy to welcome so many partners and collaborators to the Met Office for the REF Hackathon and to be involved with such an engaging event”.
The four day event not only served to showcase the REF development activities to the community but also provided important scientific, technical and operational feedback to the REF development team. This was particularly important for ensuring the REF is positioned at the forefront of piloting advancements in complex citations. Helpful input on complex citations was provided by representatives from the IPCC’s Task Group on Data Support for Climate Change Assessments. Important suggestions for strengthening the observation-modelling interface were made including handling observational uncertainties. Many of the points raised were also made in the recently published report by the Climate Modelling User Group.
Overall the Hackathon not only served to significantly advance REF development and identify gaps in its planned structure, but also provided valuable input from the community on how to grow the REF more broadly for model evaluation. Suggestions for future benchmarking development featured including a wider array of simulations, observational datasets, and metrics.
Further community involvement in co-development of the REF has been made possible through an open access public GitHub repository, which has a list of open issues that anyone in the community can contribute to: GitHub – Climate-REF/climate-ref: Rapid Evaluation Framework for CMIP simulations
Helene Hewitt, co-chair of the CMIP Panel welcomed the progress made by those participating in person and online,
“The Rapid Evaluation Framework is a potential game changer for users of CMIP data. The CMIP panel wants to extend its thanks to the model benchmarking task team, hackathon participants, and the funding bodies especially ESA and DoE for their contributions.”
A set of Frequently Asked Questions, is now available on the dedicated Rapid Evaluation Framework Page. Resources from the hackathon are available on the dedicated event page.
