Data-assimilation of crowdsourced weather stations for urban heat and water studies in Birmingham as a testbed

Data-assimilation of crowdsourced weather stations for urban heat and water studies in Birmingham as a testbed

DARE pilot project with participants:  Professor Lee Chapman, University of Birmingham;  Sytse Koopmans; Gert-Jan Steeneveld, Wageningen University & Research (WUR)

The research will be a verification by ground truth observations of the Birmingham Urban Climate Lab (BUCL) observational network.

The worktask aims are:

  • Setting up basic model run with WRF with meteorological boundaries from ECMWF
  • Remapping land use from CORINE land cover inventory (100 m) to USGS classification used by WRF
  • Calculate land use fraction by Landsat 8 satellite
  • Derived urban morphology indicators with the NUDAPT tool

For the project we completed the model infrastructure without data assimilation and a basic run without was conducted accordingly. The main effort done in the last year was to create a geographical dataset that satisfy the 200 meter resolution. Before applying data assimilation it is important to describe the land use and urban characteristics as well as possible.

  • The default available land use datasets were too course for a 200m run. We have remapped the 100m Corine land use dataset to a format which can be read by WRF.
  • The urban morphology has been improved by applying the National Urban Database and Access Portal Tool (NUDAPT) (Ching et al, 2009*). With this step we progressed the representation of urban morphology in the WRF model compared to previous studies.

NUDAPT has been tested against more elementary and basic representations in WRF in a study for a Chinese city. The performance of NUDAPT looks promising.

Image 1 – Mean building height Birmingham


Image 2: Land use fraction Birmingham

*Ching J, Brown M, McPherson T, Burian S, Chen F, Cionco R, Hanna A, Hultgren T, Sailor D, Taha H, Williams D. 2009. National Urban Database and Access Portal Tool, NUDAPT. Bulletin of the American Meteorological Society 90( 8): 1157– 1168.

Investigation of the ability of the renewed UK operational weather radar network to provide accurate real-time rainfall estimates for improved flood warnings.

Investigation of the ability of the renewed UK operational weather radar network to provide accurate real-time rainfall estimates for improved flood warnings.

by Dr RobThompson and Prof Anthony Illingworth, Dept of Meteorology, University of Reading

The UK operational radar network has the potential to deliver real-time rainfall estimates every five minutes with a resolution of 1km2 over most of the populated areas of the UK. If these rain rates were accurate then such data would have a major impact on the ability to predict short term ‘flash’ flooding events so that mitigating action can be taken. However, at present, for flood warnings, the accuracy is deemed insufficient, so the radar rainfall estimates from each radar are continuously adjusted using recent observations by ground-based rain gauges.

The UK radar network has recently been renewed and upgraded to dual polarisation, resulting in much improved data quality. In this study we will compare the radar signal obtained every five minutes from  the operational Dean Hill radar with the rain rate at the ground some 20km from the radar and just 400m below the radar beam and use this to validate and improve the retrieval algorithms that convert the radar return signal, or ‘reflectivity’ into a rain rate at the ground.

In the preparatory work  for this DARE pilot project we have been comparing the radar reflectivity observed every five minutes with the scanning Dean Hill radar 20km distant from Chilbolton where we have five different high resolution rain gauges. The radar pulse samples a volume 300m by 300m by 600m and is at a height of 400m above the gauges (see image).  One of these gauges measures the size distribution of the rain drops and so once we know the sizes and number of the drops we can calculate the radar reflectivity we would expect from the radar. Over the past two years we find close agreement, but there appears to be a slow drift in the radar calibration of about 60%. In collaboration with the Met Office we are trying to find the source of this drift.  However, once we correct for this drift, we find that with the new radar and its improved data quality there is a close correspondence between the rain from the radar and that observed at the ground.  This performance appears to be much better than previously obtained before the radars were upgraded.


Image. Multiple rain gauges

Operational meteorology for the public good, or for profit?

Operational meteorology for the public good, or for profit?

Most countries have a national weather service, funded by the government. A key role is to provide a public weather service, publishing forecasts to help make everyday business and personal decisions, as well as providing weather warnings for hazardous events. Large sums of money are invested in research and development of forecasting systems; supercomputing resources and in observing networks. For example in 2018-19 the UK Met Office invested £55 million in satellite programmes. International cooperation between weather services means that weather data obtained by one country are usually distributed to others through the World Meteorological Organisation (WMO) Global Transmission Service (GTS) in near real time, and for the common good. Is this all about to change?

In the future there is likely to be an increasing need for smart, local forecasts for the safety of autonomous vehicles (e.g. to allow the vehicle to respond to rain, snow, ice etc). Such vehicles also provide an observing platform able to take local measurements of weather that could be used to improve forecasts.  But who owns the data (the driver, the car owner, the car manufacturer…) and can it be distributed for the common good? Can the data be trusted? What about privacy concerns?

IBM weather infographic

Across the observation sector, access to space is getting less expensive. For example, depending on the specifications, a nanosatellite can be built and placed in orbit for 0.5 million euros.  Furthermore, industry is beginning to run its own numerical weather prediction models (e.g., IBM weather).  This means that there are a growing number of companies investing in earth observation and numerical weather prediction,  and wanting financial returns on their investments.

Do we need a new paradigm for weather prediction?

My Met Office Placement

My Met Office Placement

by Laura Mansfield


This summer, I spent 10 weeks on a placement at the Met Office, in Exeter. This was part of the Mathematics of Planet Earth training programme and started with a week of lectures and lab sessions thanks to Professor Rupert Klein from the Freie Universität Berline, with guest lectures from UK academics and Met Office staff. The theme was “Multiscale analysis of atmosphere-ocean flows and related numerical issues” with topics covering scales in geophysical flows and asymptotic analysis. I learned a lot about how to approach geophysical problems and found the lectures to be the perfect balance of mathematics and physical intuition.

I spent the remaining 9 weeks in the Informatics Lab, who are a team of technologists, scientists and designers who work to innovate, explore and demonstrate new ideas, particularly to make data useful. I explored how probabilistic programming languages could be used in climate and weather modelling, which also gave me the chance to learn how to build simple climate models in Python. I presented some of this work at a seminar in the Met Office and wrote a few blog posts on my progress (see those here: an introduction to probabilistic programming , an application with differential equation modelling and simple climate modelling)

The working environment in the lab was very different from a PhD office. I found that colleagues have a genuine interest in what other team members are working on and partner or team work to solve problems was common. While I was there, I also gained some tips on how to work better, including better coding practises, how to distribute tasks across computing resources and how to visualise data more effectively. I will definitely be taking a lot of this back to my PhD with me!

Outside of work, I also enjoyed life in Devon. We generally had great weather and Exeter is a lovely city to spend time in. I also took a few trips down to the beach, to surrounding villages and to Dartmoor. Plus, I can’t really complain about the views from the Informatics Lab.

My view from the Informatics Lab on a sunny day

Thanks to Rachel Prudden and everyone at the Informatics Lab who took me in for 2 months and to Mathematics of Planet Earth and the Met Office for making this happen.

wCROWN: Workshop on Crowdsourced data in Numerical Weather Prediction

wCROWN: Workshop on Crowdsourced data in Numerical Weather Prediction

by Sarah Dance

On 4-5 December 2018, the Danish Meteorological Institute (DMI) is hosted a workshop on crowdsourced data in numerical weather prediction (NWP), attended by Joanne Waller and Sarah Dance from the DARE project.  DMI  hosted this workshop with two aims, 1) Gather experts on crowdsourced data focused on NWP, to start a network of people working on the subject and 2) producing a white paper directing the research community towards best practices and guidelines on the subject.

Presenters from the University of Washington (Seattle), University of Reading and several operational weather centres including the Met Office (UK), German Weather Service (DWD), Meteo France, ECMWF, KNMI and EUMETNET gave us status reports on their research into using crowdsourced data, opportunistic data and citizen science. We discussed the issues arising in the use of such data and agreed to write a workshop report together to feed into EUMETNET activities. We also enjoyed a fascinating tour of the DMI  operational forecasters centre.

Machine learning and data assimilation

Machine learning and data assimilation

by Rossella Arcucci

Imagine a world where it is possible to accurately predict the weather, climate, storms, tsunami and other computational intensive problems in real time from your laptop or even mobile phone – if one has access to a supercomputer then to be able to predict at unprecedented scales/detail. This is the long term aim of our work on Data Assimilation with Machine Learning at the Data Science Institute (Imperial College London, UK) and as such, we believe, it will be a key component of future Numerical Forecasting systems.

We proved that the integration of machine learning with Data assimilation can increase the reliability of prediction, reducing errors by including information with an actual physical meaning from observed data. The resulting cohesion of machine learning and data assimilation is then blended in a future generation of fast and more accurate predictive models. This integration is based on the idea of using machine learning to learn the past experiences of an  assimilation process. This follows the principle of Bayesian approach.

Edward Norton Lorenz stated “small causes can have larger effects”, the so called butterfly effect. Imagine a world where it is possible to catch “small causes” in real time and predict effects in real time as well. To know to act! A world where science works with continuously learning from observation.

Figure 1. Comparison of the Lorenz system trajectories obtained by the use of Data Assimilation (DA) and by the integration of machine learning with Data assimilation (DA+NN)

Investigating alternative optimisation methods for variational data assimilation

Investigating alternative optimisation methods for variational data assimilation

by Maha Kaouri

Supported by the DARE project, I and a few others from the University of Reading recently attended the weeklong workshop on sensitivity analysis and data assimilation in meteorology and oceanography (a.k.a. the Adjoint workshop) in Aveiro, Portugal.

The week consisted of 60 talks on a variety of selected topic areas including sensitivity analysis and general theoretical data assimilation. I presented the latest results from my PhD research in this topic area and discussed the benefits of using globally convergent methods in variational data assimilation (VarDA) problems. Variational data assimilation combines two sources of information, a mathematical model and real data (e.g. satellite observations).

The overall aim of my research is to investigate the latest mathematical advances in optimisation to understand whether the solution of VarDA problems could be improved or obtained more efficiently through the use of alternative optimisation methods, whilst keeping computational cost and calculation time to a minimum. A possible application of the alternative methods would be to estimate the initial conditions for a weather forecast where the dynamical equations in this case include the physics of the earth system. Weather forecasting has a short time window (the forecast will no longer be useful after the weather event occurs) and so it is important to investigate alternative methods that provide an optimal solution in the given time.

The VarDA problem is known in numerical optimisation as a nonlinear least-squares problem which is solved using an iterative method – a method which takes an initial guess of the solution and then generates a sequence of better guesses at each step of the algorithm. The problem is solved in VarDA as a series of linear least-squares (simpler) problems using a method equivalent to the Gauss-Newton optimisation method. The Gauss-Newton method is not globally convergent in the sense that the method does not guarantee convergence to a stationary point given any initial guess. This is the motivation behind the investigation of newly developed, advanced numerical optimisation methods such as globally convergent methods which use safeguards to guarantee convergence from an arbitrary starting point. The use of such methods could enable us to obtain an improvement on the estimate of the initial conditions of a weather forecast within the limited time and computational cost available.

The conference brought together many key figures in weather forecasting as well as those new to the field such as myself, providing us with the opportunity to learn from each other during the talks and poster session. I had the advantage of presenting my talk on the first day, allowing me to spend the rest of the week receiving feedback from the attendees who were eager to discuss ideas and make suggestions for future work. The friendly atmosphere of the workshop made it easier as an early-career researcher to freely and comfortably converse with those more senior during the breaks.

I would like to thank the DARE project for funding my attendance at the workshop and the organising committee for hosting such an insightful event.

Accounting for Unresolved Scales Error with the Schmidt-Kalman Filter at the Adjoint Workshop

Accounting for Unresolved Scales Error with the Schmidt-Kalman Filter at the Adjoint Workshop

by Zak Bell

This summer I was fortunate enough to receive funding from the DARE training fund to attend the 11th workshop on sensitivity analysis and data assimilation in meteorology and oceanography. This workshop, also known as the adjoint workshop, provides academics and students with an occasion to present their research of the inclusion of Earth observations into mathematical models. Due to the friendly environment of the workshop, I was presented with an excellent opportunity to condense a portion of my research into a poster and discuss it with other attendees at the workshop.

Data assimilation is essentially a way to link theoretical models of the world to the actual world. This is achieved by finding the most likely state of a model through observations of it. A state for numerical weather prediction will typically be comprised of variables such as wind, moisture and temperature at a specific time. One way to assimilate observations is through the Kalman Filter. The Kalman Filter assimilates one observation at a time and through consideration of the errors of our models, computations and observations we can determine the most probable state of our model and use this state to better model or forecast the real world.

It goes without saying that a better understanding of the errors involved in the observations would lead to a better forecast. Therefore, research into observation errors is a large and ongoing area of interest. My research is on observation error due to unresolved scales in data assimilation which can be broadly described as the difference between what an observation actually observes and a numerical model’s representation of that observation. For example, an observation taken in a sheltered street of a city will have a different value than a numerical model of that city unable to individually represent the spatial scales of each street. To utilize such observations within data assimilation, the unresolved spatial scales must be accounted for in some way.  The method I chose to create a poster for was the Schmidt-Kalman Filter which was originally developed for navigation purposes but has since been the subject of a few studies within the meteorology community on unresolved scales error.

The Schmidt-Kalman Filter accounts for the state- and time-dependence of the error due to unresolved scales through use of the statistics of the unresolved scales. However, to save on computational expense, the unresolved state values will be disregarded. My poster presented a mathematical analysis of a simple example for the Schmidt-Kalman Filter and highlighted its ability to compensate for unresolved scales error. The Schmidt-Kalman filter performs better than a Kalman Filter for just the resolved scales but worse than a Kalman Filter that resolves all scales which is to be expected. Using the feedback from the other attendees and ideas obtained from other presentations at the workshop I will continue to investigate the properties of the Schmidt-Kalman Filter as well as its suitability for urban weather prediction.

Producing the best weather forecasts by using all available sources of information

Producing the best weather forecasts by using all available sources of information

Jemima M. Tabeart is an PhD student at the University of Reading in the Mathematics of Planet Earth Centre for Doctoral Training, she has received funding from the DARE  training fund to attend Data Assimilation tutorials at the  Workshop on Sensitivity Analysis and Data Assimilation in Meteorology and Oceanography, 1-6 July 2018, Aveiro, Portugal. Here she writes about  her research work.

In order to produce the best weather forecast possible, we want to make use of all available sources of information. This means combining observations of the world around us at the current time with a computer model that can fill in the gaps where we have no observations, by using known laws of physics to evolve observations from the past. This combination process is called data assimilation, and our two data sources (the model and observations) are weighted by our confidence in how accurate they are. This means that knowledge about errors in our observations is really important for getting good weather forecasts. This is especially true where we expect errors between different observations to be related, or correlated.

Caption: An image of the satellite MetOp-B which hosts IASI (Infrared Atmospheric Sounding Interferometer) – an instrument that I have been using as an example to test new mathematical techniques to allow correlated errors to be used inexpensively in the Met Office system.  Credit: ESA AOES Medialab MetOp-B image.

Why do such errors occur? No observation will be perfect: there might be biases (e.g. a thermometer that measures everything 0.5℃ too hot), we might not be measuring variables that are used in a numerical model, and converting observations introduces an error (this is the case with satellite observations), and we might be using high density observations that can detect phenomena that our model cannot (e.g. intense localised rainstorms might not show up if our model can only represent objects larger than 5km). Including additional observation error correlations means we can use observation data more intelligently and even extract extra information, leading to improvements in forecasts.

However, these observation error correlations cannot be calculated directly – we instead have to estimate them. Including these estimates in our computations is very expensive, so we need to find ways of including this useful error information in a way that is cheap enough to produce new forecasts every 6 hours! I research mathematical techniques to adapt error information estimates for use in real-world systems.

Caption: Error correlation information for IASI instrument. Dark colours indicate stronger relationships between errors for different channels of the instrument – often strong relationships occur between variables that measure similar things. We want to keep this structure, but change the values in a way that makes sure our computer system still runs quickly.

At the workshop I’ll be presenting new work that tests some of these methods using the Met Office system. Although we can improve the time required for our computations, using different error correlation information alters other parts of the system too! As we don’t know “true” values, it’s hard to know whether these changes are good, bad or just different. I’m looking forward to talking with scientists from other organisations who understand this data and can provide insight into what these differences mean. Additionally, as these methods are already being used to produce forecasts at meteorological centres internationally, discussions about the decision process and impact of different methods are bound to be illuminating!

Science and Society …

Science and Society …

by Roland Potthast

How is science helping us to improve life in the city? How does it help us to predict high-impact weather events? Water and wind today influence all of our lives: by its impact on travel, by strong dependence of renewable energy on clouds and winds, and by the dependence of agriculture and many other parts of or modern society on the atmosphere, precipitation, rivers, clouds and wind!

ASCII��� ���JKJK'«\#�ù������@����©k������  �����Ù� ��iž����Wd�. ����Se�͖�)|ÿÿ íÿÿëÁÿÿÖ<�?��Ï��˜Cÿÿ™¶���"�"���"�"�"�"���"�"�"�"�����������"���"�"�ˆ�"���"���"�"���������������"�"�"�"�������������������������"�ˆ�"�"�������������‘�!"�‘�������"�"�"�"����������‘�‘�‘�‘�‘�������"�"�"�"� ������’™ �’™ �‘�‘�‘�‘�� �����"�� �"��‘�‘�’™ �’™ �‘�‘�‘�1"�Q3�‘�‘� �0��"�!"�‘�’™ �’™ �‘� �‘�‘�1"�A"�‘�‘�0�0� �‘�’™ �’™ �’™ �’™ �‘�‘�‘�0�1"�0�A"�`���‘�1"�‘�‘�’™ �’™ �’™ �’™ �’™ �‘�‘�‘�0�0�‘�Q3�A"�‘�‘�‘�‘�‘�‘�‘�‘�‘�"�ˆ� �0���‘�‘�‘���‘�‘�‘�‘�‘�‘�‘�‘�0�0�‘�‘�‘�‘�‘� �0�P3�‘�‘�‘�€���‘�‘�‘�0�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�"�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�‘�e¡���Zb�›�÷ÿ��Mi�»~����“’�������������������������������������Š����s�������������-f����ï™�������������ҟ�­ÿ��èb�ë�²ÿ��ëc�ë�²ÿ��ëc�ë�²ÿ��ëc�ë�²ÿ��ëc�������������fù�\Ù�z���D�����������������FAFA�š�´�0�w�Ò���FAFAe�������û(‘_������û(‘_|�O›�óû(‘_p�-�õû(‘_������û(‘_������û(‘_f�ÇÏ�Ûû(‘_p�dŠ�ëû(‘_x�¡�ïû(‘_€�ém�æû(‘_ˆ�)ü�Ïû(‘_������������w�ä�òû(‘_������������������������������������������������������������������������������������FAFA��æZ v���×����f�Þӂ���T����‚�Fl|���(��������  ������������������3����������������ZP����������y����������������������w�7w�ÿFAFA��������������������������������������®®®®F��  �����æ��É����� »���… ��d ��&�����������������������������������������������������������������������?��–9�������������������É���������������/���������������������������������������@����������������æ�����e������������� ���(�����������������9��  ��‘�����������������������������������i���s���!����������|������������2�����Ù��Å��à...

Basic Science. First, basic science aims to understand the processes which are relevant to a particular event or phenomenon. For example, there is cloud microphysics, which describes how clouds are built. It combines physical and chemical insight with meteorological knowledge, and in all of it models are formulated and tested based on the language of mathematics. There are students and professors, smaller and larger teams, which carry out experiments, formulate systems of equations, carry out simulations, test their theories and then publish them in both science journals and for the general public.

Applied Science. Usually, basic science tries to achieve understanding, and applied science tries to work on influencing applications, on simulations and predictions, on using the science to actually make a difference. That might not be easy. For example: when you want to calculate a weather forecast, you need to be fast. If it takes you a week to calculate a forecast for a week, the weather has already moved on.

Modern Supercomputing. Today, in modern supercomputing centres, we have 15 minutes to determine the state of the atmosphere on a global scale, and then less than one hour to calculate forecasts for three days. This is done every 6 hours. If it comes to fast high-impact events like heavy thunderstorms with strong precipitation, we do it even every three hours with a forecast lead time of one day (24 hours). That means we have about 30 minutes to carry out the task. We want to get forecasts for the next 6-12 hours every hour, and forecasts for the next 2 hours every 5 minutes! To do this, it needs very advanced science, new high-resolution measurements, new algorithms, new computers: it needs new ideas all the time. And it needs teams of dedicated scientists, who carry out the programming and make sure that forecasts are reliable and state-of-the-art!

Public Service and Companies. Providing a service for the public every three hours, or every hour, or every 5 minutes with reliable forecasts cannot be carried out by a university. If each and every airplane departure depends on such a service, you need a very reliable team of people dedicated to this task. Think about this: if the impact is very important for the whole society, it should not be done by a company. The state needs institutions like a national weather service, equipped with a modern supercomputer to do this. At the same time, the scientists who program the forecasts need the intense interaction with researchers which are not bound by the constraints of immediate delivery. And all of us need different companies to work on services and products based on the insight, algorithms and methods which scientists develop day by day …

Your Role. There are scientists working on pure science. There are scientists working on applied science. There are those working in the supercomputing centres, in the public services, in the many companies active around the globe. Where is your own role? What attracts you most: the basic science, trying to understand the phenomena? Or the applied science, with algorithms and programs? Or the science working in the framework of a supercomputing centre, making it work efficiently for real large-scale applications? Or the scientist making sure the service is delivered to a wide range of people, to many countries and many industries? Science is touching all layers of our society! It is fascinating and exciting, and there is space for each and every character, for many different people!

Find your own role and place, and then enjoy it and contribute!