EUMETNET, is a grouping of 31 European National Meteorological Services which is provides a framework for collaboration between its members in the meteorological and hydrological fields. You can find out more about EUMETNET and its missions here.
ON 12-13th of March 2019 EUMETNET held a Crowd Sourcing Workshop at the Met Office in Exeter, UK, which a number of us from DARE attended remotely. The workshop was attended by a large number of the EUMETNET members both in person and remotely. The topics covered included:
description of the Met Office observation network (WOW) and their move to a cloud based data system;
the existing crowd sourcing platforms across the Europe (see the list below);
summaries of other recent crowd sourcing meetings/workshops;
data related issues such as data format, quality control, storage, data sources and legal issues.
From the various talks it was clear that there is a great opportunity for crowdsourced data to contribute to meteorological forecast accuracy, in particular to nowcasting and more timely warnings since observations had higher spatial resolution in the established crowd sourcing platforms. However, the collected data types and their quality control varied greatly between the various crowdsourcing platforms discussed. For example, e.g. AEMET, concentrate on collecting singular atmospheric observations which are characterised by being local, rare, of significant intensity and with the capacity to cause high social impact. While, the Met Office Weather Observation Website (WOW) system accepts all types of observations from various types of sources.
Distinction between crowd sourced and opportunity data, such as smartphone pressure measurements, car temperature data, aircraft data was also made. Leading to an important discussion on collection of such data which often requires collaboration with industry e.g. car manufacture, mobile network provider etc. Should companies release such opportunity data for the mutual benefit through improved forecasts and warning systems? A number of companies already release their data, however, these are exceptions rather than the norm currently. There is no coherent agreement on how this would be uniformly achieved in practice which remains an open question for the debate.
List of various crowd sourcing platforms discussed at the workshop:
by Dr Natalie Douglas, University of Surrey and Dr Alison Fowler, University of Reading and NCEO
ISDA2019, the 7th International Symposium for Data Assimilation 2019, was hosted in Kobe, Japan this year from the 21st to the 24th January at the RIKEN Center for Computational Science – home of the K-computer. Attended by over 100 research scientists, the conference boasted a guest list of inspiring speakers and poster presenters from all over the globe. Topics of current relevance invoking enthusiastic discussion included Big Data Assimilation, Uncertainty Quantification, Satellite and Coupled DA, Multi-Scale Processes and DA in Broader Applications to name a few.
“I thoroughly recommend attending ISDA to anyone working in Data Assimilation. This was my first conference abroad, it was hugely informative and extremely well organised. Not only that, I had enormous amounts of fun getting to know and even making good friends with a lot of the key players in my field.” – Dr Natalie Douglas from the University of Surrey, UK.
“The ISDA provided a fascinating overview of the latest developments in Data Assimilation from around the world. It included a diverse range of applications from supernova astrophysics to my more familiar area of meteorology. I found the chance to spend a week with other scientists hugely beneficial to my work. After the symposium I enjoyed an extended visit to RIKEN to continue discussions on the efficient use of high-volume observations in their home-developed rapid-update-forecasting system. The aim of this state-of-the-art system is to provide advanced warnings of the most extreme rainfall events that can evolve in the matter of minutes. Each year in Japan, such events result in a multitude of deaths and wider devastation, and so such a system is sorely needed. Bringing this knowledge back to the UK may prove greatly beneficial as we prepare for the effects of a changing climate.” – Dr Alison Fowler from the University of Reading, UK and NCEO.
On 4-5 December 2018, the Danish Meteorological Institute (DMI) is hosted a workshop on crowdsourced data in numerical weather prediction (NWP), attended by Joanne Waller and Sarah Dance from the DARE project. DMI hosted this workshop with two aims, 1) Gather experts on crowdsourced data focused on NWP, to start a network of people working on the subject and 2) producing a white paper directing the research community towards best practices and guidelines on the subject.
Presenters from the University of Washington (Seattle), University of Reading and several operational weather centres including the Met Office (UK), German Weather Service (DWD), Meteo France, ECMWF, KNMI and EUMETNET gave us status reports on their research into using crowdsourced data, opportunistic data and citizen science. We discussed the issues arising in the use of such data and agreed to write a workshop report together to feed into EUMETNET activities. We also enjoyed a fascinating tour of the DMI operational forecasters centre.
Last week, on 22-23 October 2018, around 230 scientists from the three ocean and climate related clusters of excellence in northern Germany met in Berlin in the joint conference on Ocean – Climate – Sustainability Research Frontiers. The participants brought in lively discussions within the context of scientific and societal action towards ocean and climate research. Apart from the discussions more oriented toward the basic climate science and technical aspects, from a personal standpoint (perhaps because of its distance from my own work), I found a number of presentations from “The Future Ocean” cluster in Kiel, which include scholars from politics, social science, philosophy and international law most interesting. Some of these presentations offered a window on the connection between climate change and global and local politics in countries (e.g.; as tropical islands in the Indian ocean, who generally rely on external aid) most affected by increasing sea levels and coastal erosion. In common, this class of talks indicated a need for improving the communication of climate and natural risk science to society. Actually, a huge component of the unpredictability in future climate projections comes from the societal component.
However, as analyzed in one talk in the conference, it seems that, ultimately, public opinion is mostly driven by what is shown on TV, and TV, public offer is in turn mostly driven by the economic powers. Thus, as described the writer Jose Luis Sampedro more than 6 years ago, “public opinion” (defined in Wikipedia as consisting of the “desires, wants, and thinking of the majority of the people”), is in reality the “opinion of the media” or the “opinion of the economic powers”. This clearly connects to the results of Brazil elections just yesterday and the new presidency, and so to the derived very uncertain future of the Amazon management. Apart from the risks to biodiversity, a further deforestation of the Amazon rainforest would make it impossible to cut carbon pollution and the aspirational target of no more than 1.5ºC global warming above pre-industrial temperatures set in the Paris climate agreement. Brazilian people (and they are not alone) seem either oblivious to the problem or convinced that they are not affected by it (even, as from a friend’s personal communication last week, it appears that some people in Brazil sadly believe climate change is an European hoax to take control over their rainforest). Generally rising sea levels and increased storm surge risks, as well as the extra energy accumulated in the Earth system in general (and ocean in particular, boosting atmospheric convection and associated flood risks), will surely lead to a further demand of online, continuously updated, risk information to face emergency situations in the future city. One can wish the best for Brazil and the Amazon, which is the best for the world. In any case, let’s hope that Copacabana is not swallowed in the sea before Rio is transformed into a resilient city.
I attended the Workshop on Sensitivity Analysis and Data Assimilation in Meteorology and Oceanography, also known as Adjoint Workshop, which took place in Aveiro, Portugal between 1st and 6th July 2018. This opportunity was given to me due to funding for early career researchers from the Engineering and Physical Sciences Research Council (EPSRC) Data Assimilation for the Resilient City (DARE) project in the UK. All recipients of this fund that were participating for the first time in the workshop were invited to attend the pre-workshop day of tutorials, presenting sensitivity analysis and data assimilation fundamentals geared to the early career researchers. I would like to thank to EPSRC DARE award committee and the organizers of the Adjoint Workshop for finding me worthy of this award.
Currently I’m a post graduate student at the Brazilian National Institute for Space Research (INPE) and have been visiting the Global Modeling and Assimilation Office (GMAO) of the American National Aeronautics and Space Administration (NASA) for almost one year as part of my PhD comparing two approaches to obtain what is known as the observation impact measure. This measure is obtained as a direct application of sensitivity in data assimilation and basically is a measure of how much each observation helps to improve the short-range forecasts. In Meteorology, specifically in numerical weather prediction, these observations are represented by the global observing system, which includes observations obtained from a number of in situ (e.g., radiosondes, and surface observations) and remote sensed observations (e.g., satellite sensors). During my visit, I’ve been working under the supervision of Ricardo Todling from NASA/GMAO comparing results from two strategies for assessing the impact of observations on forecasts using data assimilation system available at NASA/GMAO: one based on the traditional adjoint technique, another based on ensembles. Preliminary results from this comparison were presented during the Adjoint Workshop.
The Adjoint Workshop provided a perfect environment for early career researchers interact with experts in the field from all around the world. The attendance at the workshop has helped me engage healthy discussions about my work and data assimilation in general. The full programme with abstracts and presentations is available at the workshop web site: https://www.morgan.edu/adjoint_workshop
Thanks to everyone who contributed to this workshop.
Supported by the DARE project, I and a few others from the University of Reading recently attended the weeklong workshop on sensitivity analysis and data assimilation in meteorology and oceanography (a.k.a. the Adjoint workshop) in Aveiro, Portugal.
The week consisted of 60 talks on a variety of selected topic areas including sensitivity analysis and general theoretical data assimilation. I presented the latest results from my PhD research in this topic area and discussed the benefits of using globally convergent methods in variational data assimilation (VarDA) problems. Variational data assimilation combines two sources of information, a mathematical model and real data (e.g. satellite observations).
The overall aim of my research is to investigate the latest mathematical advances in optimisation to understand whether the solution of VarDA problems could be improved or obtained more efficiently through the use of alternative optimisation methods, whilst keeping computational cost and calculation time to a minimum. A possible application of the alternative methods would be to estimate the initial conditions for a weather forecast where the dynamical equations in this case include the physics of the earth system. Weather forecasting has a short time window (the forecast will no longer be useful after the weather event occurs) and so it is important to investigate alternative methods that provide an optimal solution in the given time.
The VarDA problem is known in numerical optimisation as a nonlinear least-squares problem which is solved using an iterative method – a method which takes an initial guess of the solution and then generates a sequence of better guesses at each step of the algorithm. The problem is solved in VarDA as a series of linear least-squares (simpler) problems using a method equivalent to the Gauss-Newton optimisation method. The Gauss-Newton method is not globally convergent in the sense that the method does not guarantee convergence to a stationary point given any initial guess. This is the motivation behind the investigation of newly developed, advanced numerical optimisation methods such as globally convergent methods which use safeguards to guarantee convergence from an arbitrary starting point. The use of such methods could enable us to obtain an improvement on the estimate of the initial conditions of a weather forecast within the limited time and computational cost available.
The conference brought together many key figures in weather forecasting as well as those new to the field such as myself, providing us with the opportunity to learn from each other during the talks and poster session. I had the advantage of presenting my talk on the first day, allowing me to spend the rest of the week receiving feedback from the attendees who were eager to discuss ideas and make suggestions for future work. The friendly atmosphere of the workshop made it easier as an early-career researcher to freely and comfortably converse with those more senior during the breaks.
I would like to thank the DARE project for funding my attendance at the workshop and the organising committee for hosting such an insightful event.
Luca Cantarello is an PhD student at the University of Leeds. He received funding from the DARE training fund to attend Data Assimilation tutorials at the Workshop on Sensitivity Analysis and Data Assimilation in Meteorology and Oceanography, 1-6 July 2018, Aveiro, Portugal. Here he writes about his experience.
Since I started my PhD project at the University of Leeds as a NERC DTP student a few months ago, I have been reflecting on the importance of not feeling too alone in doing science, exactly like in the everyday life. The risk of feeling isolated while doing research can very much apply to all PhD students, but it may be particularly relevant to cases like mine, as very few people are dealing with Data Assimilation in my university.
In this sense, joining the last week’s 11th Adjoint workshop on sensitivity analysis and Data Assimilation in Meteorology and Oceanography in Aveiro has been an excellent opportunity and I am very grateful to the University of Reading and the DARE project for having helped me to take part in it, I received funding from the DARE project which enabled me to attend.
In Aveiro I could enjoy the company and the support of a vast community of scientists, all willing to share their findings and discuss problems and needs with their peers. In the room there was an impressive synergy among many researchers who had attended the same workshop several times in the past, despite it has been held only every second or third year.
The photograph is of the hotel where the adjoint workshop was held.
The workshop has been an important training opportunity for me as I am still in the process of learning, but also an occasion to revive my motivation with new stimuli and ideas before getting to the heart of my PhD in the coming two years.
During the poster session I took part in, I got useful feedback and comments about my project (supervised by Onno Bokhove and Steve Tobias at the University of Leeds and by Gordon Inverarity at the Met Office), in which I am trying to understand how satellite observations at different spatial scales impact on a Data Assimilation scheme. I will bring back to Leeds all the hints and the suggestions I have collected, hoping to attend the next adjoint meeting in a few years and being able to tell people the progress I have achieved in the meantime.
Jemima M. Tabeart is an PhD student at the University of Reading in the Mathematics of Planet Earth Centre for Doctoral Training, shehas received funding from the DARE training fund to attend Data Assimilation tutorials at the Workshop on Sensitivity Analysis and Data Assimilation in Meteorology and Oceanography, 1-6 July 2018, Aveiro, Portugal. Here she writes about her research work.
In order to produce the best weather forecast possible, we want to make use of all available sources of information. This means combining observations of the world around us at the current time with a computer model that can fill in the gaps where we have no observations, by using known laws of physics to evolve observations from the past. This combination process is called data assimilation, and our two data sources (the model and observations) are weighted by our confidence in how accurate they are. This means that knowledge about errors in our observations is really important for getting good weather forecasts. This is especially true where we expect errors between different observations to be related, or correlated.
Caption: An image of the satellite MetOp-B which hosts IASI (Infrared Atmospheric Sounding Interferometer) – an instrument that I have been using as an example to test new mathematical techniques to allow correlated errors to be used inexpensively in the Met Office system. Credit: ESA AOES Medialab MetOp-B image.
Why do such errors occur? No observation will be perfect: there might be biases (e.g. a thermometer that measures everything 0.5℃ too hot), we might not be measuring variables that are used in a numerical model, and converting observations introduces an error (this is the case with satellite observations), and we might be using high density observations that can detect phenomena that our model cannot (e.g. intense localised rainstorms might not show up if our model can only represent objects larger than 5km). Including additional observation error correlations means we can use observation data more intelligently and even extract extra information, leading to improvements in forecasts.
However, these observation error correlations cannot be calculated directly – we instead have to estimate them. Including these estimates in our computations is very expensive, so we need to find ways of including this useful error information in a way that is cheap enough to produce new forecasts every 6 hours! I research mathematical techniques to adapt error information estimates for use in real-world systems.
Caption: Error correlation information for IASI instrument. Dark colours indicate stronger relationships between errors for different channels of the instrument – often strong relationships occur between variables that measure similar things. We want to keep this structure, but change the values in a way that makes sure our computer system still runs quickly.
At the workshop I’ll be presenting new work that tests some of these methods using the Met Office system. Although we can improve the time required for our computations, using different error correlation information alters other parts of the system too! As we don’t know “true” values, it’s hard to know whether these changes are good, bad or just different. I’m looking forward to talking with scientists from other organisations who understand this data and can provide insight into what these differences mean. Additionally, as these methods are already being used to produce forecasts at meteorological centres internationally, discussions about the decision process and impact of different methods are bound to be illuminating!
Takuya Kurihana has received funding from the DARE training fund to attend Data Assimilation tutorials at the Workshop on Sensitivity Analysis and Data Assimilation in Meteorology and Oceanography, 1-6 July 2018, Aveiro, Portugal. Here he writes about himself and his research.
What if we could more accurately predict what atmospheric phenomenon will happen in the next minute, hour and day using the current limited information. This scientific question has inspired me to be being involved in the research activity since undergraduate student. I am Takuya Kurihana, a Meteorology MS student in the University of Tsukuba under the supervision of Dr. Hiroshi L. Tanaka, and an incoming Computer Science PhD student in the University of Chicago. My current research focuses on 1. How to improve the accuracy of weather forecasting: “Predictability”, and 2. How to make use of a massive amount of dense meteorological dataset for data assimilation. With developing new application for purpose 2, I am now researching the impact of using a large amount of atmospheric observation as much as we can towards the daily scale weather forecast.
Regarding to the improvement predictability, as a previous article explained by Zak Bell, Making the Most of Uncertain Urban Observations , data assimilation plays an imperative role in numerical weather prediction because the longer we run a numerical weather forecasting model, the larger the error of forecast grows up. This is because the uncertainty. Even if we use the most precise model, this tendency would not change more or less. But, applying the data assimilation methods can minimize the error by installing observation into the optimization process Fig.1 is an example experiment about an advantage of data assimilation. Therefore, we have to gather a variety of denser observation data from both horizontally and vertically wider range of points in real operation. Other than land observation (Figure 2) , sondes, and buoys, recent satellite observation (Figure 3 and Figure 4) [2, 3], which provide us much richer and denser information, have been utilized in operational data assimilation processes.
The spatially condensed satellite data, however, causes one problem in the current data assimilation methods. The issue is that too much dense data will rather deteriorate the quality of assimilation products based on previous researches. Simply put, we have to leave out large proportion of these data: “Thinning”, even while the technology of meteorological satellites is advancing. Moreover, there are several resource limitations to prepare the forecasting since we are not afford to compute endlessly, and the performance and size of computer are constrained. In order to make use of larger proportion of these data while not reducing assimilation quality, the spatial interpolation, so called super-observation (SO) procedure are developed. As one SO system, I proposed a new algorithm which could deal with a massive amount of satellite big data efficiently and speedily within a cloud-resolving model (Nonhydrostatic ICosahedral Atmospheric Model; NICAM) grid coordination. The algorithm primary targets to reduce “Do/For Loop” iteration process to find the nearest model grid location, which can also skip the computation by a complex observation operator.
Which is better Thinning or SO? Although this would be controversial discussion among meteorologists, I would like to give one example in the Workshop on Sensitivity Analysis and Data Assimilation in Portugal. While the new application should be tested in further numerical experiments through my master research project, I ponder that we should consider a more efficient usage of these meteorological “Big Data” in the near future. Through the attendance at the workshop, I would like to discuss my application and its effect on the data assimilation, as well as receive fruitful advice from cutting edge researchers.
Figure 1. These timeseries of trajectories imply a small difference between two initial conditions finally ends up completely varied behaviors. Blue is No data assimilation from 200 time steps, and Red is data assimilation Lorenz63 Trajectory. Demo above by Takuya Kurihana.
Figure. 2 This map shows the sparse location of land observation points
 S-A. Boukabara, K. Garrett, K. V. Kumar, “Potential Gaps in the Satellite Observing System Coverage: Assessment of Impact on NOAA’s Numerical Weather Prediction Overall Skills”. (2016). Mon. Wea. Rev., 144, 2547–2563, https://doi.org/10.1175/ MWR-D-16-0013.1.
The Dare team went on a field trip last month! It was a well planned and executed trip – as you would expect from a group of mathematicians. It was also a very interesting trip for us since most of us have only ever used data (e.g. for improving forecasts) not collected it. Even better Tewkesbury area has become a sort of benchmark for testing new data assimilation methods, ideas, tools, observations, etc, and so many of us have worked with LisFlood numerical model (developed by a team led by Prof. Paul Bates at the University of Bristol) over the Tewkesbury domain. We have seen the river runs in the model outputs, watched the rivers Avon and Severn go out of banks in our plots, and investigated various SAR images of the area but we have never been to the area. We generally do not need to visit the area when working with the models, however, now that there was a chance to do so, it was no surprise that many of us were keen to go. And we did go like ‘d’ A-team:
However we had a more important reason for visiting too – we were going to the Tewkesbury area to collect metadata from a number of river cameras located near Tewkesbury town. These river cameras are high definition webcams owned and serviced by Farson Digital Ltd in various location over the UK. We had recently discovered that six of such cameras are within the LisFlood model domain and have captured the November 2012 floods in the area. With the permission from the Farson Digital Ltd, we have obtained hourly daylight images of the floods from 21st November 2012 to 5th of December 2012. Hence, the aim our trip was to obtain accurate (with errors of no more than few centimeters) positional information (i.e. latitude, longitude, height) of the cameras themselves as well as the positional information of a number of markers in the images for each of the cameras. We need this information to extract as accurately as possible water extents and water depth from these images using image processing tools (which we are currently working on).
To take these measurements we had borrowed some tools from the Department of Geography at the University of Reading. We used a differential GPS tool (GNSS) to very accurately (on order of few centimeters) measured the position of a given point in 3D space, that is its latitude, longitude, and height above the sea level, however, it had to be used on the ground (e.g. could not measure remote or high points such as building corners where some cameras were mounted) and not be too close to buildings or large trees. To measure remote and high points we used Total Station, which allowed us to shoot a laser beam to the desired point to measure its 3D position in space.
We had planned to visit all six cameras within the space of the two days 16th and17th of April, however, despite our best plans and fantastic organisation skills we were too ambitious with our time and we had to drop the camera furthest from our base – the Bewdley camera (see map with camera positions in figure 2). Thus, on our first day, we took measurements from Wyre Piddle, Evesham, and Digglis Lock cameras, spotting ourselves live on the Farson Digital Ltd site.
We returned to our base – the Tewkesbury Park Hotel, to be joined by the Ensemble team from the Lancaster University. Ensemble Project is lead by Prof. Gordon Blair, and as Dare is funded by the EPSRC Senior Fellowship in Digital Technology for Living with Environmental Change. It was very interesting to meet the Ensemble project team and learn more in-depth about their work, future interests, and scope for the collaboration.
On our second day, the Dare team visited the Tewkesbury camera while the Ensemble team learned more about the purpouse of the data collection and the Novermber 2012 floods in the area. Then we all jointly measured a large number of points at the Strensham Lock. In 2012 we all would have been totally sumberged in water in this picture since the flood waters completely swallowed the island on which the house is standing flooding the building along with it.
Our grand finale was the meeting with the director of the Farson Digital Ltd Glyn Howells as well as a number of stakeholders who have commissioned the cameras we visited. It was very interesting for us to learn how the network of the river cameras was born from the need to know and understand the current state of the river for a variety of river users – fishermen, campers, boaters, etc. Also, how these cameras have become invaluable assets to many stakeholders for various reasons – greatly reducing the number of river condition related phone enquiries, monitoring river bank and bridge conditions, and so on.
Now a month later we have downloaded and processed the data we collected from these stations. In figure 7 we have plotted the data points we took at the Tewkesbury site, owned both by the Environmental Agency and Tewkesbury Marina (both of which we greatly thank for their support and assistance before and during our trip, especially to Steve Edgar from EA and Simon Amos and Bruno from Tewkesbury Marina). In the figure, the red dots are the camera positions – pre-2016 and current camera positions, and the black dots are all the other measurements we took using both the TotalStation and GNSS tools, which are plotted against the Environmental Agency lidar data with 1m horizontal resolution.
We are currently working on extracting the water extent from these images which we then will use to produce water depth observations. Our final aim is to see how much forecast improvement such rich source of observations offer, in particular, before the rising limb of the flood.
We are very thankful to Glyn Howells and the various stakeholders for permitting us to use of the images, allowing us to take the necessary measurements, assisting us on the sites, and joining at the workshop!