{"id":895,"date":"2019-07-09T14:22:07","date_gmt":"2019-07-09T13:22:07","guid":{"rendered":"http:\/\/35.193.178.118\/?page_id=217"},"modified":"2019-11-08T11:49:37","modified_gmt":"2019-11-08T11:49:37","slug":"getting-started-2","status":"publish","type":"page","link":"https:\/\/research.reading.ac.uk\/fiduceo\/fcdrs\/harmonisation\/getting-started-2\/","title":{"rendered":"Getting Started"},"content":{"rendered":"\r\n

Harmonisation<\/h1>\r\n\r\n\r\n\r\n

By Ralf Quast and Sam Hunt<\/p>\r\n\r\n\r\n\r\n

 <\/p>\r\n\r\n\r\n\r\n

Getting started on the FIDUCEO method <\/h2>\r\n\r\n\r\n\r\n

An explicit methodology to conduct such a harmonised calibration of a series of sensors has not existed before FIDUCEO and was explored and developed by following the errors-in-variables (EIV) idea, which takes into account the uncertainty and error correlation in all sensor telemetry measurements and yields optimised calibration coefficients and an associated calibration error covariance matrix for all sensors (e.g. Newey 2001).<\/p>\r\n\r\n\r\n\r\n

Figure 4 broadly illustrates the methodology which was developed by the FIDUCEO project to conduct harmonisation within four steps. Step one is to collect dual-sensor match-ups where two sensors observe approximately the same Earth target at approximately the same time (e.g. simultaneous nadir overpasses). Ideally, these match-up data include all possible reference-to-sensor and sensor-to-sensor pairs. Step two is an assessment of the error covariance of all measurements included with the match-up data and an assessment of the expected match-up radiance differences and their associated uncertainties. Step three consists in a joint optimisation of sensor calibration models. The last step is the evaluation of the calibration coefficient error covariance matrix. The complete methodology is described in detail by Giering et al.<\/em> (2019)<\/a> and explained at a higher level in the following sections.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Figure 4: Methodology to conduct harmonisation<\/p>\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Newey, W.K. Flexible simulated moment estimation of nonlinear errors-in-variables models.\u00a0Rev. Econ. Stat. <\/em>2001<\/strong>,\u00a083<\/em>, 616\u2013627. [CrossRef<\/a>]
Giering, R.; Quast, R.; Mittaz, J.P.D.; Hunt, S.E.; Harris, P.M.; Woolliams, E.R.; Merchant, C.J. A Novel Framework to Harmonise Satellite Data Series for Climate Applications.\u00a0Remote Sens.<\/em>\u00a02019<\/strong>,\u00a011<\/em>, 1002. [CrossRef<\/a>]
Harris, P.M.; Hunt, S.E.; Quast, R.; Giering, R.; Mittaz, J.P.D.; Woolliams, E.R.; Dilo, A.; Cox, M. Solving large structured non-linear least-squares problems with an application in Earth observation. In preparation<\/em>.<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n\r\n\r\n\r\n

Step one: how to collect match-up data?<\/h2>\r\n\r\n\r\n\r\n

In FIDUCEO, we call a match-up a \u201cpoint\u201d measurement that is matched by another \u201cpoint\u201d measurement sufficiently close in space and time \u2013 excluding the trivial case of neighbouring measurements from the same acquisition. In other words, we require match-up pixels that cover the same place on Earth acquired at almost the same time and with similar viewing geometries (Figure 5).<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Figure 5: Concept of a dual-sensor match-up.<\/p>\r\n\r\n\r\n\r\n

In most cases, we extend this definition to also include the neighbouring pixels of a sensor acquisition, covering a symmetrical area of several pixels around the match-up point, which allows us to check homogeneity and detect cloud shadows or other problematic characteristics. Match-up data will include calibrated measurements for the reference sensor and raw telemetry data for as many as possible other (i.e. non-reference) sensors. It is important to have a set of match-up data which represents the relevant aspects of Earth sufficiently well.<\/p>\r\n\r\n\r\n\r\n

You will want to compare acquisitions of different spaceborne sensors which are spatially and temporally close so that you may assume a similar atmospheric condition. You will also want to compare satellite sensor data with in situ<\/em> measurements, like AERONET data, in order to have measurements close to the measured entity. In some cases, we have also considered indirect match-ups between two sensors, with a geo-stationary sensor or in situ<\/em> measurements used to identify stable Earth targets.<\/p>\r\n\r\n\r\n\r\n

FIDUCEO has developed a variable multi-sensor match-up software (MMS) that allows us to run various match-up tasks using many different satellite- and ground-based instruments. The framework comprises an open design with various plugin points to allow for easy algorithmic updates or adding new sensor data.<\/p>\r\n\r\n\r\n\r\n\r\n\r\n\r\n
To browse and download the code for the multi-sensor match-up software take a look at FIDUCEO\u2019s MMS repositor on GitHub<\/a>.

T. Block, S. Embacher, C. J. Merchant, and C. Donlon, \u201cHigh Performance Software Framework for the Calculation of Satellite-to-Satellite Data Matchups (MMS version 1.2),\u201d Geosci. Model Dev. Discuss., vol. 11, no. 6, pp. 1\u201315, Jun. 2017.<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n\r\n\r\n\r\n

Step two: how to assess the match-up error covariance structure?<\/h2>\r\n\r\n\r\n\r\n

After having collected a representative set of match-up data, you will have to assess the radiance difference \\textcolor{orange}{L^i} \u2013 \\textcolor{red}{L^j} = \\textcolor{purple}{K^{ij}}<\/span> expected due to different sensor spectral response functions and the uncertainty of this expected difference. You will also have to assess the uncertainty of the \\textcolor{orange}{L^i} \u2013 \\textcolor{red}{L^j} = \\textcolor{purple}{K^{ij}}<\/span> due to matchup and representativeness errors. Doing this requires expert knowledge on the satellite instruments involved and on radiative transfer modelling.<\/p>\r\n\r\n\r\n\r\n

To explain how to assess the error covariance structure of the satellite measurements, either calibrated radiance or sensor telemetry, let us have a closer look at the mathematical representation of a matchup dataset. We arrange each dataset in form of a matrix, where the columns correspond to different variables and the rows correspond to different times of measurement of these variables.<\/p>\r\n\r\n\r\n\r\n

For reference-to-sensor match-up datasets, the first column contains the calibrated radiance of your reference sensor (yellow column in Figure 6) while the other columns contain the telemetry data of a non-reference sensor (red columns). Each column represents a different variable.<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Figure 6: Mathematical representation of a reference-to-sensor match-up dataset<\/p>\r\n\r\n\r\n\r\n

For sensor-to-sensor match-up datasets you have several columns of sensor telemetry data instead of a single column of calibrated radiance (Figure 7).<\/p>\r\n\r\n\r\n\r\n

\"\"<\/figure>\r\n\r\n\r\n\r\n

Figure 7: Mathematical representation of a sensor-to-sensor match-up dataset<\/p>\r\n\r\n\r\n\r\n

Writing matchup data in form of a matrix is helpful when we consider how the errors of the measurements are correlated. This is essential, because considering the error covariance of all measurements is the main difference when comparing FIDUCEO harmonisation to less rigorous calibration methods.<\/p>\r\n\r\n\r\n\r\n

In FIDUCEO, we have minimised the error correlation between different<\/em> variables (i.e. inter-column error correlation) by design of our calibration models thus we need to consider error correlation among the measurements of the same<\/em> variable over time only (i.e. intra-column error correlation). Then the error covariance matrix \\boldsymbol{S}(\\textcolor{orange}{\\boldsymbol{q}})<\/span> of any column \\textcolor{orange}{\\boldsymbol{q}}<\/span> in a matchup dataset is of one of three genuine types or a combination thereof (Figure 8).<\/p>\r\n\r\n\r\n\r\n

    \r\n
  1. Independent random \u2013 the errors of the measurements are not correlated because the measurements themselves are all independent. We have error variances only.<\/li>\r\n
  2. Common random – the errors of the measurements are fully correlated because all measurements are susceptible to the same error.<\/li>\r\n
  3. Structured random \u2013 the measurements in a column are the result of some averaging procedure, where the original measurements (which are averaged) are all independent.<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n
    \"\"<\/figure>\r\n\r\n\r\n\r\n

    Figure 8: Schematic illustration of error covariance matrices for genuine error correlation types.<\/p>\r\n\r\n\r\n\r\n

    FIDUCEO has specified a match-up dataset format to represent these error covariance structures and has developed software tools to aid users in preparing correctly formatted data. The tools are included with the harmonisation software on GitHub<\/a>.<\/p>\r\n\r\n\r\n\r\n\r\n\r\n\r\n
    To browse and download the code for the harmonisation software take a look at FIDUCEO\u2019s Harmonisation repository on GitHub<\/a>.<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n\r\n\r\n\r\n

    Step three: how to optimise the sensor calibration models?<\/h2>\r\n\r\n\r\n\r\n

    Knowing the error covariances of all your matchup data, the mathematical formulation of the harmonisation problem is relatively straightforward. Let us assume we optimise the calibration model we have used for the AVHRR series of sensors, which is<\/p>\r\n\r\n\r\n\r\n

    \\textcolor{blue}{L}(\\textcolor{red}{\\boldsymbol{x}},\\textcolor{blue}{\\boldsymbol{\\alpha}}) = \\textcolor{blue}{\\alpha_1} + (\\epsilon + \\textcolor{blue}{\\alpha_2}) \\textcolor{red}{L_\\mathrm{ICT}} \\frac{\\textcolor{red}{C_\\mathrm{E}} \u2013 \\textcolor{red}{\\bar{C}_\\mathrm{S}}}{\\textcolor{red}{\\bar{C}_\\mathrm{ICT}} \u2013 \\textcolor{red}{\\bar{C}_\\mathrm{S}}} + \\textcolor{blue}{\\alpha_3} (\\textcolor{red}{C_\\mathrm{E}} \u2013 \\textcolor{red}{\\bar{C}_\\mathrm{S}}) (\\textcolor{red}{\\bar{C}_\\mathrm{ICT}} \u2013 \\textcolor{red}{\\bar{C}_\\mathrm{S}}) + \\textcolor{blue}{\\alpha_4} \\textcolor{red}{T_\\mathrm{env}} \\tag{Eq.4}<\/span><\/p>\r\n\r\n\r\n\r\n

    where<\/p>\r\n\r\n\r\n\r\n