by Amos Lawless, December 2023
Variational data assimilation is widely used in many operational weather forecasting centres, such as the Met Office and the European Centre for Medium-range Weather Forecasts, as a means of obtaining initial conditions for numerical weather forecasts. In this method, the data assimilation problem is treated like a big data fitting problem where we try to find the model representation of the atmosphere that most closely matches all the available data. Mathematically this is treated as an optimization problem. We look for the atmospheric state that minimizes a function measuring the mismatch between the model and observations. However, the problem is that we have millions of observations to fit and billions of model variables to find, so doing this in practice is not so easy! We also need to solve the problem quickly, so that we can issue a forecast before the weather actually happens.
Conditioning and why it matters
Besides the huge size of the problem, something that hinders our ability to solve the problem efficiently is what mathematicians call the conditioning of the problem. The conditioning measures how sensitive the solution to the problem is to small errors in the data. If the solution is very sensitive, then the problem is known as ill-conditioned. Since we know that our measurements of the atmosphere (e.g. from satellites, weather balloons and land stations) always contain some uncertainties, ill-conditioning in the data assimilation problem can amplify these uncertainties, leading to very large errors in our estimate of the atmospheric state. To understand and overcome this we need to use some advanced mathematics.
For several years at the University of Reading, we have worked to analyse what makes the data assimilation problem ill-conditioned (see, for example, [1-3]). But having understood this, what can we do about it? The answer is called preconditioning. Preconditioning is a mathematical technique that aims to transform an ill-conditioned problem into one that is better conditioned. In other words, we change the problem to one that is easier to solve, but in a way that we can then retrieve the solution to the original problem. This is a common technique in other areas of mathematics, but the particular form of the variational data assimilation means that standard preconditioning techniques do not always work.
A new network
Recently we have obtained some funding from the Isaac Newton Institute to set up a small network to look at this problem. Thus, the network on Preconditioning Variational Data Assimilation Problems (PVDAP) was born. PVDAP involves mathematicians from the Universities of Reading, Strathclyde, TU Eindhoven and Potsdam, as well as the CERFACS research institute. The aim of the network is to share expertise, identify key challenges and work together to begin to tackle some of the core problems in this area. On 4-5 December the first in-person meeting of the network took place at the University of Reading, where we spent two full days discussing current approaches for preconditioning the variational data assimilation and potential future avenues of research. You can find more information about the network and our activities on the network website
https://new-og.is.strath.ac.uk/science/mathematicsstatistics/pdvap/
[1] Haben, S., Lawless, A.S. and Nichols, N.K. (2011), Conditioning of incremental variational data assimilation, with application to the Met Office system. Tellus A, 63, 782-792.
[2] Tabeart, J.M., Dance, S.L., Haben, S.A., Lawless, A.S., Nichols, N.K. and Waller J.A. (2018), The conditioning of least squares problems in variational data assimilation. Numer. Linear Algebra Appl., 25, e2165.
[3] Shataer, S., Lawless, A.S. and Nichols, N.K. (2023), The Conditioning of Hybrid Variational Data Assimilation. Numer. Linear Algebra Appl., doi.org/10.1002/nla.2534.