Fast regression considering random errors in all variables<\/h2>\r\n\r\n\r\nLet\u2019s revisit the example from Sam and Jon\u2019s blog:\r\n\r\n\r\n\r\nTake a straight-line equation Y=A+BX <\/span> \u00a0and generate a set of random points (X,Y) <\/span> \u00a0that all lie on the straight line with intercept A=0\u00a0and slope B=1 <\/span> . Then add random errors to both X <\/span> and Y <\/span> values and fit a straight line to the data. The animated figure below illustrates what happens: ODR takes account of all uncertainties in both X <\/span> and Y <\/span> by iteratively varying the straight line and the X <\/span> \u00a0value of each data point slightly (indicated by the jiggling blue line and the wobbling yellow triangles) until the vertical distance of the data points from the straight line is \u201cin balance\u201d with the horizontal displacement of the data points from their \u201cmeasured\u201d position. The eventual straight line has an intercept of A=0 <\/span> and a slope B=1 <\/span> with small errors that are consistent with the random errors added to X <\/span> and Y <\/span> . (Our animation just illustrates what is varied during the fitting, but does not show how the fitting converges.)\r\n\r\n\r\n![\"\"](\"https:\/\/research.reading.ac.uk\/fiduceo\/wp-content\/uploads\/sites\/129\/2019\/07\/wobbling-new-dancing-fig-1.gif\")
<\/figure>\r\n\r\n\r\nBy intention, we discern between jiggling (the straight blue line) and wobbling (the yellow triangles) because the conceptual difference between ODR and simple linear regression is that the former jiggles and<\/em> wobbles, whereas the latter merely jiggles, which is why it leads to biased solutions.\r\n\r\n\r\n\r\nThe animated figure above shows 60 data points, and there is already quite a bit of wobbling apparent\u00a0. Two parameters are jiggled for the straight blue line, whereas 60 parameters are wobbled for the data. So, there is more wobbling than jiggling.\r\n\r\n\r\n\r\nThe harmonisation problem that FIDUCEO is tackling considers some 100 million data points, and ODR must wobble some 100 million yellow triangles, leading to far more wobbling than jiggling. Here ‘jiggling’ specifically refers to the variation of calibration parameters, whereas ‘wobbling’ refers to the variation of satellite state measurements. The solution of the FIDUCEO harmonisation problem by ODR must vary (jiggle) about 60 calibration parameters and (wobble) about 100 million satellite state parameters. In this sense, there is plenty more wobbling than jiggling.\r\n\r\n\r\n\r\nIs there a way to do what ODR does, but without wobbling – just by jiggling, like simple linear regression, but without leading to biased solutions?\r\n\r\n\r\n\r\nWe think there is! The general formalism of uncertainty propagation we explained in our November blog<\/a> makes it possible to transform the uncertainty of the X <\/span> values into an uncertainty of Y <\/span> values. The animated figure below illustrates how it works. The straight blue line and the yellow triangles are the same as in the figure above. But here the yellow triangles are not wobbled. Instead, the wobbling of triangles along the X <\/span> axis is transformed into a stretching and squeezing of shady yellow uncertainty bars along the Y <\/span> axis. The uncertainty is a function of the slope of the straight blue line: the steeper the slope, the larger the uncertainty. The straight blue line is now jiggled until the vertical distance of the yellow triangles from the straight blue line is \u201cin balance\u201d with the shady yellow uncertainty. (Again, our animation just illustrates what is varied during the fitting, but does not show how the fitting converges.)\r\n\r\n\r\n![\"\"](\"https:\/\/research.reading.ac.uk\/fiduceo\/wp-content\/uploads\/sites\/129\/2019\/07\/wobbling-new-dancing-fig-2.gif\")
<\/figure>\r\n\r\n\r\nLet\u2019s name our new method \u201cfast ODR\u201d. Then, how do the results of fast ODR compare to those of the ODR method in Sam and Jon\u2019s blog? We have repeated Sam and Jon\u2019s experiment 10,000 times, each time randomly generating thousand (X,Y) <\/span> points and fitting a straight line by means of fast ODR. The figure below depicts the distribution of the error of the fitted slope B versus the error of the fitted intercept A <\/span> . We deliberately draw a joint error distribution here, because the errors of the fitted A <\/span> s and B <\/span> s are correlated. Like Sam and Jon\u2019s ODR fitting, our fast ODR yields an unbiased solution.\r\n\r\n\r\n![\"\"](\"https:\/\/research.reading.ac.uk\/fiduceo\/wp-content\/uploads\/sites\/129\/2019\/07\/wobbling-new-dancing-fig-3.png\")
<\/figure>\r\n\r\n\r\nBut what happens, if \u2013 using the fitted A <\/span> s and B <\/span> s \u2013 we predict the value of Y <\/span> at, let\u2019s say, X=2 <\/span> , a point outside of the fitting domain? Sam and Jon showed that simple linear regression wrongly predicts Y<2 in this case, whereas ODR correctly Y=2 (with a small uncertainty). The figure below illustrates the distribution of the errors of the Y <\/span> s obtained from fast ODR fitting, which are in very good agreement with the errors of Sam and Jon\u2019s ODR fitting. Both fast and standard ODR give equivalent results!\r\n\r\n\r\n![\"\"](\"https:\/\/research.reading.ac.uk\/fiduceo\/wp-content\/uploads\/sites\/129\/2019\/07\/wobbling-new-dancing-fig-4.png\")
<\/figure>\r\n\r\n\r\nBy means of the same algorithmic differentiation techniques we demonstrated in our November blog<\/a>, fast ODR can be generalised to apply to the measurement equations of the FIDUCEO harmonisation problem. Our \u201cfast errors-in-variables\u201d implementation solves the (simulated) harmonisation problem for the whole series of AVHRR sensors within less than 400 iterations, in less than 20 minutes on a single workstation computer. Yet, our implementation does not take account of correlated errors.\r\n\r\n\r\nWhat is error correlation, how does it arise, and how can it be avoided?<\/h2>\r\n\r\n\r\nOur first animated figure (that illustrates how ODR works) visualises an error structure that is random: each wobble of an X <\/span> is independent of all other wobbles of all other X <\/span>s. If the errors of X <\/span> were correlated, each wobble would be correlated with some other wobbles, looking like \u201cchoreographed dancing\u201d rather than random wobbling. Is considering correlated errors more difficult than considering random errors? Absolutely. The problem is to realise the correct \u201cdance\u201d. Correctly describing the dance of correlated errors in the data for all match-ups requires the computation of some ten quadrillion (10,000,000,000,000,000) numbers. Many of them are simply zero \u2013 which unfortunately does neither simplify nor speed-up the problem solving!\r\n\r\n\r\n\r\nSo how does error correlation arise? Error correlation between e.g. different image pixels of calibrated radiance arise, if the calibration of different pixels is computed from the same basis data, like the temperature of the internal calibration target or the digital count number recorded for the dark target. Satellite instruments often measure these basic quantities once per row of image pixels, but not per individual pixel. Now, if a match-up data set contains image pixels of the same row, the errors of the calibrated radiance of these pixels are correlated. These type of error correlations in a data set can be avoided only, if the data set does not include multiple image pixels of the same row. Whether such a \u201cclearing\u201d of data sets to avoid error correlation is feasible depends on the number of match-up pixels needed to achieve the accuracy and stability requirements on a calibration.\r\n\r\n\r\n\r\nAnother sort of error correlation arises, if images of calibrated radiance are computed from basis data, which are averaged over many image rows: the temperature of the internal calibration target or the digital count number from the dark target are often averaged over several image rows to reduce the uncertainty of the computed calibrated radiance. In fact, the calibrated radiance images look smoother with averaging of these basis data than without, suggesting less uncertainty. But is the uncertainty truly less? No – it isn\u2019t! Instead, the uncertainty of an individual datum is smoothly distributed across many data. In consequence, these calibrated radiance images, though appearing smooth, are not more certain than those without averaging, but with an error structure like choreographed \u00a0\u00a0\u201cdancing and singing\u201d that is much more problematic to take account of. Avoiding this entertainment is, however, easy: don\u2019t average your basis data, and the error structure of your derived data will be an easy-to-operate uncorrelated wobbling.\r\n\r\n\r\n\r\nWhether wobbling (graciously) or dancing, FIDUCEO has mastered the former (by means of two independent errors-in-variables implementations) and is approaching the latter. Ultimately, only one question matters: Is the error structure of our climate data records adequately characterised, or not?\r\n\r\n\r\nFurther Reading<\/h2>\r\n\r\n\r\nYork, Derek et al. (2004). \u201cUnified equations for the slope, intercept, and standard errors of the best straight line.\u201d American Journal of Physics, 72, 367. DOI: 10.1119\/1.1632486<\/a>\r\n\r\n\r\n\r\nCarroll, R.J. and Ruppert, D. (1996). “The use and misuse of orthogonal regression in linear errors-in-variables models.\u201d The American Statistician, 50, 1.\u00a0DOI: 10.1080\/00031305.1996.10473533<\/a>","protected":false},"excerpt":{"rendered":"Illustrating regression methods with animated GIF images and explaining how error correlations arise.<\/p>\n","protected":false},"author":219,"featured_media":497,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"__cvm_playback_settings":[],"__cvm_video_id":"","footnotes":""},"categories":[15],"tags":[],"coauthors":[6],"class_list":["post-1249","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"yoast_head":"\n
Is gracious wobbling the new dancing? - Fiduceo<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Is gracious wobbling the new dancing? - Fiduceo\" \/>\n<meta property=\"og:description\" content=\"Illustrating regression methods with animated GIF images and explaining how error correlations arise.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/\" \/>\n<meta property=\"og:site_name\" content=\"Fiduceo\" \/>\n<meta property=\"article:published_time\" content=\"2017-08-17T10:38:39+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-07-29T10:40:19+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/research.reading.ac.uk\/fiduceo\/wp-content\/uploads\/sites\/129\/2019\/07\/matrix.png\" \/>\n\t<meta property=\"og:image:width\" content=\"357\" \/>\n\t<meta property=\"og:image:height\" content=\"357\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Alex Daykin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Alex Daykin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/\",\"url\":\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/\",\"name\":\"Is gracious wobbling the new dancing? - Fiduceo\",\"isPartOf\":{\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/#website\"},\"datePublished\":\"2017-08-17T10:38:39+00:00\",\"dateModified\":\"2019-07-29T10:40:19+00:00\",\"author\":{\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/532e1ce773445fb76d77785ea76f5be8\"},\"breadcrumb\":{\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/research.reading.ac.uk\/fiduceo\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Is gracious wobbling the new dancing?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/#website\",\"url\":\"https:\/\/research.reading.ac.uk\/fiduceo\/\",\"name\":\"Fiduceo\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/research.reading.ac.uk\/fiduceo\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/532e1ce773445fb76d77785ea76f5be8\",\"name\":\"Alex Daykin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/image\/26159086a629d325cff50dd27004d579\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/48d6c4db18e4195997d9ec88cad612357d655264b37576280d9aceae3fa7aac4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/48d6c4db18e4195997d9ec88cad612357d655264b37576280d9aceae3fa7aac4?s=96&d=mm&r=g\",\"caption\":\"Alex Daykin\"},\"url\":\"https:\/\/research.reading.ac.uk\/fiduceo\/author\/a-j-daykinreading-ac-uk\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Is gracious wobbling the new dancing? - Fiduceo","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/","og_locale":"en_GB","og_type":"article","og_title":"Is gracious wobbling the new dancing? - Fiduceo","og_description":"Illustrating regression methods with animated GIF images and explaining how error correlations arise.","og_url":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/","og_site_name":"Fiduceo","article_published_time":"2017-08-17T10:38:39+00:00","article_modified_time":"2019-07-29T10:40:19+00:00","og_image":[{"width":357,"height":357,"url":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-content\/uploads\/sites\/129\/2019\/07\/matrix.png","type":"image\/png"}],"author":"Alex Daykin","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Alex Daykin","Estimated reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/","url":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/","name":"Is gracious wobbling the new dancing? - Fiduceo","isPartOf":{"@id":"https:\/\/research.reading.ac.uk\/fiduceo\/#website"},"datePublished":"2017-08-17T10:38:39+00:00","dateModified":"2019-07-29T10:40:19+00:00","author":{"@id":"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/532e1ce773445fb76d77785ea76f5be8"},"breadcrumb":{"@id":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/research.reading.ac.uk\/fiduceo\/"},{"@type":"ListItem","position":2,"name":"Is gracious wobbling the new dancing?"}]},{"@type":"WebSite","@id":"https:\/\/research.reading.ac.uk\/fiduceo\/#website","url":"https:\/\/research.reading.ac.uk\/fiduceo\/","name":"Fiduceo","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/research.reading.ac.uk\/fiduceo\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/532e1ce773445fb76d77785ea76f5be8","name":"Alex Daykin","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/image\/26159086a629d325cff50dd27004d579","url":"https:\/\/secure.gravatar.com\/avatar\/48d6c4db18e4195997d9ec88cad612357d655264b37576280d9aceae3fa7aac4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/48d6c4db18e4195997d9ec88cad612357d655264b37576280d9aceae3fa7aac4?s=96&d=mm&r=g","caption":"Alex Daykin"},"url":"https:\/\/research.reading.ac.uk\/fiduceo\/author\/a-j-daykinreading-ac-uk\/"}]}},"cc_featured_image_caption":{"caption_text":false,"source_text":false,"source_url":false},"_links":{"self":[{"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/posts\/1249","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/users\/219"}],"replies":[{"embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/comments?post=1249"}],"version-history":[{"count":1,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/posts\/1249\/revisions"}],"predecessor-version":[{"id":1251,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/posts\/1249\/revisions\/1251"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/media\/497"}],"wp:attachment":[{"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/media?parent=1249"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/categories?post=1249"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/tags?post=1249"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/coauthors?post=1249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
<\/figure>\r\n\r\n\r\nBy intention, we discern between jiggling (the straight blue line) and wobbling (the yellow triangles) because the conceptual difference between ODR and simple linear regression is that the former jiggles and<\/em> wobbles, whereas the latter merely jiggles, which is why it leads to biased solutions.\r\n\r\n\r\n\r\nThe animated figure above shows 60 data points, and there is already quite a bit of wobbling apparent\u00a0. Two parameters are jiggled for the straight blue line, whereas 60 parameters are wobbled for the data. So, there is more wobbling than jiggling.\r\n\r\n\r\n\r\nThe harmonisation problem that FIDUCEO is tackling considers some 100 million data points, and ODR must wobble some 100 million yellow triangles, leading to far more wobbling than jiggling. Here ‘jiggling’ specifically refers to the variation of calibration parameters, whereas ‘wobbling’ refers to the variation of satellite state measurements. The solution of the FIDUCEO harmonisation problem by ODR must vary (jiggle) about 60 calibration parameters and (wobble) about 100 million satellite state parameters. In this sense, there is plenty more wobbling than jiggling.\r\n\r\n\r\n\r\nIs there a way to do what ODR does, but without wobbling – just by jiggling, like simple linear regression, but without leading to biased solutions?\r\n\r\n\r\n\r\nWe think there is! The general formalism of uncertainty propagation we explained in our November blog<\/a> makes it possible to transform the uncertainty of the X <\/span> values into an uncertainty of Y <\/span> values. The animated figure below illustrates how it works. The straight blue line and the yellow triangles are the same as in the figure above. But here the yellow triangles are not wobbled. Instead, the wobbling of triangles along the X <\/span> axis is transformed into a stretching and squeezing of shady yellow uncertainty bars along the Y <\/span> axis. The uncertainty is a function of the slope of the straight blue line: the steeper the slope, the larger the uncertainty. The straight blue line is now jiggled until the vertical distance of the yellow triangles from the straight blue line is \u201cin balance\u201d with the shady yellow uncertainty. (Again, our animation just illustrates what is varied during the fitting, but does not show how the fitting converges.)\r\n\r\n\r\n<\/figure>\r\n\r\n\r\nLet\u2019s name our new method \u201cfast ODR\u201d. Then, how do the results of fast ODR compare to those of the ODR method in Sam and Jon\u2019s blog? We have repeated Sam and Jon\u2019s experiment 10,000 times, each time randomly generating thousand (X,Y) <\/span> points and fitting a straight line by means of fast ODR. The figure below depicts the distribution of the error of the fitted slope B versus the error of the fitted intercept A <\/span> . We deliberately draw a joint error distribution here, because the errors of the fitted A <\/span> s and B <\/span> s are correlated. Like Sam and Jon\u2019s ODR fitting, our fast ODR yields an unbiased solution.\r\n\r\n\r\n<\/figure>\r\n\r\n\r\nBut what happens, if \u2013 using the fitted A <\/span> s and B <\/span> s \u2013 we predict the value of Y <\/span> at, let\u2019s say, X=2 <\/span> , a point outside of the fitting domain? Sam and Jon showed that simple linear regression wrongly predicts Y<2 in this case, whereas ODR correctly Y=2 (with a small uncertainty). The figure below illustrates the distribution of the errors of the Y <\/span> s obtained from fast ODR fitting, which are in very good agreement with the errors of Sam and Jon\u2019s ODR fitting. Both fast and standard ODR give equivalent results!\r\n\r\n\r\n<\/figure>\r\n\r\n\r\nBy means of the same algorithmic differentiation techniques we demonstrated in our November blog<\/a>, fast ODR can be generalised to apply to the measurement equations of the FIDUCEO harmonisation problem. Our \u201cfast errors-in-variables\u201d implementation solves the (simulated) harmonisation problem for the whole series of AVHRR sensors within less than 400 iterations, in less than 20 minutes on a single workstation computer. Yet, our implementation does not take account of correlated errors.\r\n\r\n\r\nWhat is error correlation, how does it arise, and how can it be avoided?<\/h2>\r\n\r\n\r\nOur first animated figure (that illustrates how ODR works) visualises an error structure that is random: each wobble of an X <\/span> is independent of all other wobbles of all other X <\/span>s. If the errors of X <\/span> were correlated, each wobble would be correlated with some other wobbles, looking like \u201cchoreographed dancing\u201d rather than random wobbling. Is considering correlated errors more difficult than considering random errors? Absolutely. The problem is to realise the correct \u201cdance\u201d. Correctly describing the dance of correlated errors in the data for all match-ups requires the computation of some ten quadrillion (10,000,000,000,000,000) numbers. Many of them are simply zero \u2013 which unfortunately does neither simplify nor speed-up the problem solving!\r\n\r\n\r\n\r\nSo how does error correlation arise? Error correlation between e.g. different image pixels of calibrated radiance arise, if the calibration of different pixels is computed from the same basis data, like the temperature of the internal calibration target or the digital count number recorded for the dark target. Satellite instruments often measure these basic quantities once per row of image pixels, but not per individual pixel. Now, if a match-up data set contains image pixels of the same row, the errors of the calibrated radiance of these pixels are correlated. These type of error correlations in a data set can be avoided only, if the data set does not include multiple image pixels of the same row. Whether such a \u201cclearing\u201d of data sets to avoid error correlation is feasible depends on the number of match-up pixels needed to achieve the accuracy and stability requirements on a calibration.\r\n\r\n\r\n\r\nAnother sort of error correlation arises, if images of calibrated radiance are computed from basis data, which are averaged over many image rows: the temperature of the internal calibration target or the digital count number from the dark target are often averaged over several image rows to reduce the uncertainty of the computed calibrated radiance. In fact, the calibrated radiance images look smoother with averaging of these basis data than without, suggesting less uncertainty. But is the uncertainty truly less? No – it isn\u2019t! Instead, the uncertainty of an individual datum is smoothly distributed across many data. In consequence, these calibrated radiance images, though appearing smooth, are not more certain than those without averaging, but with an error structure like choreographed \u00a0\u00a0\u201cdancing and singing\u201d that is much more problematic to take account of. Avoiding this entertainment is, however, easy: don\u2019t average your basis data, and the error structure of your derived data will be an easy-to-operate uncorrelated wobbling.\r\n\r\n\r\n\r\nWhether wobbling (graciously) or dancing, FIDUCEO has mastered the former (by means of two independent errors-in-variables implementations) and is approaching the latter. Ultimately, only one question matters: Is the error structure of our climate data records adequately characterised, or not?\r\n\r\n\r\nFurther Reading<\/h2>\r\n\r\n\r\nYork, Derek et al. (2004). \u201cUnified equations for the slope, intercept, and standard errors of the best straight line.\u201d American Journal of Physics, 72, 367. DOI: 10.1119\/1.1632486<\/a>\r\n\r\n\r\n\r\nCarroll, R.J. and Ruppert, D. (1996). “The use and misuse of orthogonal regression in linear errors-in-variables models.\u201d The American Statistician, 50, 1.\u00a0DOI: 10.1080\/00031305.1996.10473533<\/a>","protected":false},"excerpt":{"rendered":"Illustrating regression methods with animated GIF images and explaining how error correlations arise.<\/p>\n","protected":false},"author":219,"featured_media":497,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"__cvm_playback_settings":[],"__cvm_video_id":"","footnotes":""},"categories":[15],"tags":[],"coauthors":[6],"class_list":["post-1249","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"yoast_head":"\n
Is gracious wobbling the new dancing? - Fiduceo<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Is gracious wobbling the new dancing? - Fiduceo\" \/>\n<meta property=\"og:description\" content=\"Illustrating regression methods with animated GIF images and explaining how error correlations arise.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/\" \/>\n<meta property=\"og:site_name\" content=\"Fiduceo\" \/>\n<meta property=\"article:published_time\" content=\"2017-08-17T10:38:39+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-07-29T10:40:19+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/research.reading.ac.uk\/fiduceo\/wp-content\/uploads\/sites\/129\/2019\/07\/matrix.png\" \/>\n\t<meta property=\"og:image:width\" content=\"357\" \/>\n\t<meta property=\"og:image:height\" content=\"357\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Alex Daykin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Alex Daykin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/\",\"url\":\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/\",\"name\":\"Is gracious wobbling the new dancing? - Fiduceo\",\"isPartOf\":{\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/#website\"},\"datePublished\":\"2017-08-17T10:38:39+00:00\",\"dateModified\":\"2019-07-29T10:40:19+00:00\",\"author\":{\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/532e1ce773445fb76d77785ea76f5be8\"},\"breadcrumb\":{\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/research.reading.ac.uk\/fiduceo\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Is gracious wobbling the new dancing?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/#website\",\"url\":\"https:\/\/research.reading.ac.uk\/fiduceo\/\",\"name\":\"Fiduceo\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/research.reading.ac.uk\/fiduceo\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/532e1ce773445fb76d77785ea76f5be8\",\"name\":\"Alex Daykin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/image\/26159086a629d325cff50dd27004d579\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/48d6c4db18e4195997d9ec88cad612357d655264b37576280d9aceae3fa7aac4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/48d6c4db18e4195997d9ec88cad612357d655264b37576280d9aceae3fa7aac4?s=96&d=mm&r=g\",\"caption\":\"Alex Daykin\"},\"url\":\"https:\/\/research.reading.ac.uk\/fiduceo\/author\/a-j-daykinreading-ac-uk\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Is gracious wobbling the new dancing? - Fiduceo","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/","og_locale":"en_GB","og_type":"article","og_title":"Is gracious wobbling the new dancing? - Fiduceo","og_description":"Illustrating regression methods with animated GIF images and explaining how error correlations arise.","og_url":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/","og_site_name":"Fiduceo","article_published_time":"2017-08-17T10:38:39+00:00","article_modified_time":"2019-07-29T10:40:19+00:00","og_image":[{"width":357,"height":357,"url":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-content\/uploads\/sites\/129\/2019\/07\/matrix.png","type":"image\/png"}],"author":"Alex Daykin","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Alex Daykin","Estimated reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/","url":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/","name":"Is gracious wobbling the new dancing? - Fiduceo","isPartOf":{"@id":"https:\/\/research.reading.ac.uk\/fiduceo\/#website"},"datePublished":"2017-08-17T10:38:39+00:00","dateModified":"2019-07-29T10:40:19+00:00","author":{"@id":"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/532e1ce773445fb76d77785ea76f5be8"},"breadcrumb":{"@id":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/research.reading.ac.uk\/fiduceo\/is-gracious-wobbling-the-new-dancing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/research.reading.ac.uk\/fiduceo\/"},{"@type":"ListItem","position":2,"name":"Is gracious wobbling the new dancing?"}]},{"@type":"WebSite","@id":"https:\/\/research.reading.ac.uk\/fiduceo\/#website","url":"https:\/\/research.reading.ac.uk\/fiduceo\/","name":"Fiduceo","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/research.reading.ac.uk\/fiduceo\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/532e1ce773445fb76d77785ea76f5be8","name":"Alex Daykin","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/research.reading.ac.uk\/fiduceo\/#\/schema\/person\/image\/26159086a629d325cff50dd27004d579","url":"https:\/\/secure.gravatar.com\/avatar\/48d6c4db18e4195997d9ec88cad612357d655264b37576280d9aceae3fa7aac4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/48d6c4db18e4195997d9ec88cad612357d655264b37576280d9aceae3fa7aac4?s=96&d=mm&r=g","caption":"Alex Daykin"},"url":"https:\/\/research.reading.ac.uk\/fiduceo\/author\/a-j-daykinreading-ac-uk\/"}]}},"cc_featured_image_caption":{"caption_text":false,"source_text":false,"source_url":false},"_links":{"self":[{"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/posts\/1249","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/users\/219"}],"replies":[{"embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/comments?post=1249"}],"version-history":[{"count":1,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/posts\/1249\/revisions"}],"predecessor-version":[{"id":1251,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/posts\/1249\/revisions\/1251"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/media\/497"}],"wp:attachment":[{"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/media?parent=1249"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/categories?post=1249"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/tags?post=1249"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/research.reading.ac.uk\/fiduceo\/wp-json\/wp\/v2\/coauthors?post=1249"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Illustrating regression methods with animated GIF images and explaining how error correlations arise.<\/p>\n","protected":false},"author":219,"featured_media":497,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"__cvm_playback_settings":[],"__cvm_video_id":"","footnotes":""},"categories":[15],"tags":[],"coauthors":[6],"class_list":["post-1249","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"yoast_head":"\n