2c:null 2d:["$","title",null,{"children":"Human-readable image quality scores | Eric Portis | Observable"}] 32:T45a,viewof chart = vegaembed({ "$schema": "https://vega.github.io/schema/vega-lite/v3.json", "width": width, "height": 400, "autosize": { "type": "fit", "contains": "padding" }, "description": "A scatterplot showing the Mean Opinion Scores and SSIMULACRA scores for images in the TID2013 database.", "data": {"url": "https://gist.githubusercontent.com/eeeps/4d5bc3e13ea3f8a7105545a52139b56d/raw/2c03455b466c5e557337c24b5d7e46c0b5fc7ec2/ssimulacra_mos.tsv", "format": { "type": "tsv"} }, "transform": [ { "calculate": "'https://eric-cloudinary-res.cloudinary.com/image/upload/f_png/tid2013/' + datum.file", "as": "link" } ], "mark": "point", "encoding": { "x": {"field": "ssimulacra", "type": "quantitative", "axis": { "tickStep": 0.05 } }, "y": {"field": "mean_opinion_score", "type": "quantitative"}, "href": {"field": "link", "type": "nominal"}, "tooltip": [ {"field": "file", "type": "nominal"}, {"field": "ssimulacra", "type": "quantitative"}, {"field": "mean_opinion_score", "type": "quantitative"} ] } } )33:T4b0,md`## A few observations - SSIMULACRA correlates more-or-less linearly with human perception! - And it’s a pretty good correlation. - Even though the MOS scale goes from 0–9, no JPEG-distorted image in the dataset scores higher than ~6.3, or lower than 1. (The range of MOS scores across the full TID2013 dataset, which includes many other types of distortion, is 0.24–7.21). - One image ([i25](https://eric-cloudinary-res.cloudinary.com/image/upload/tid2013/i25_10_1.png), of an artificial test pattern) is something of an outlier. By cutting it out, we get a significantly better trendline and fit: ${ tex`\text{MOS} = -14.645 \times \text{SSIMULACRA} + 6.388` }, ${ tex`R^2 = 0.9422` } - There are some truly awful-looking images in the TID2013 dataset. The images which have Mean Opinion Scores of less than ~3 are subjectively worse than any image I have seen in practical use on the web, even when including use cases like Low-Quality Image Placeholders. If we cut those out (in addition to cutting i25), we arrive at a perhaps more relevant trendline, (though, expectedly, a somewhat worse fit): ${ tex`\text{MOS} = -15.408 \times \text{SSIMULACRA} + 6.4638` }, ${ tex`R^2 = 0.909` } `34:T5b6,md` ## Making it human-readable 🤩 So now that we— 1. have a metric that moves more-or-less linearly, in step with human preception and judgement (or rather, we see that we already had that with SSIMULACRA in the first pace!) 2. have (somewhat-arbitrarily) determined the range of qualities that we actually care about —how should we present this metric? [Star ratings](https://en.wikipedia.org/wiki/Star_(classification) are widely-understood. A four-star rating system also maps nicely to common English-language descriptions of quality:

${ stars(4) }: Great
${ stars(3) }: Good
${ stars(2) }: Okay
${ stars(1) }: Bad

If we map the trimmed [3.0–6.4638] range of MOS scores that we care about to a [0-4] rating scale of stars, we arrive at the following formula for a star rating, as a function of SSIMULACRA:

${ tex`\text{Stars} = 4 \times (-4.448 \times \text{SSIMULACRA} + 1)` }

This has a couple of nice properties: - SSIMULACRA = 0.1, which the \`ssimulacra\` CLI calls out as “annoying,” falls at about halfway on this new scale: ${ stars(2.221) } (2.221 stars). So: 0-2 stars, we’re definitely seeing artifacts. 3 or 4 stars? Maybe not so much. - At the high end, SSIMULACRA = 0.01, which the CLI calls an “imperceptible” level of distortion, is equivalent to ${ stars(3.822) } (3.822 stars). So, only nearly-four-star images look perfect. `35:T611,html`

CSS

`36:[] 30:{"id":"77e82af82c1c0ee5","slug":"correlating-ssimulacra-and-tid2013-mean-opinion-scores","title":"Human-readable image quality scores","thumbnail":"78d3116b461fadacac71d6c8815d368a1b92d5bef5b067420bf68477b069c2fc","access_level":"public","create_time":"$D2019-02-07T04:41:29.135Z","update_time":"$D2019-02-11T22:46:51.105Z","view_time":null,"first_public_time":"$D2019-02-07T05:10:42.002Z","fork":false,"stars":0,"starred":false,"owner":{"type":"team","id":"3301827868803b5f","login":"eeeps","name":"Eric Portis","avatar_url":"https://avatars-next.observableusercontent.com/d9d2bbfe6f3f798e9ec8f3a91e0c10b7f6890314fd52f40e190022e173d838a9","tier":"starter_2024"},"version":280,"publish_version":280,"role":null,"authored":false,"authors":[{"type":"individual","id":"641c0f2eb3a762f9","login":"eeeps","name":"Eric Portis","avatar_url":"https://avatars-next.observableusercontent.com/d9d2bbfe6f3f798e9ec8f3a91e0c10b7f6890314fd52f40e190022e173d838a9","tier":null}],"members":[{"type":"individual","id":"641c0f2eb3a762f9","login":"eeeps","name":"Eric Portis","avatar_url":"https://avatars-next.observableusercontent.com/d9d2bbfe6f3f798e9ec8f3a91e0c10b7f6890314fd52f40e190022e173d838a9","tier":null,"role":"editor"}],"comments":[],"custom_thumbnail":false,"trashed":false,"subscribed":null,"forks":0,"collection":null,"type":"notebook","body":{"title":"Human-readable image quality scores","cells":[{"id":0,"value":"md`# Human-readable image quality scores\n\nMy favorite image quality metric is [SSIMULACRA](https://github.com/cloudinary/ssimulacra).\n\nUnfortunately, SSIMULACRA scores can be hard to intuitively understand. When you run the program on the command line, it gives you a nice, concise description of its output, with a couple of points of reference:\n\n

\nReturns a value between 0 (images are identical) and 1 (images are very different). If the value is above 0.1 (or so), the distortion is likely to be perceptible / annoying. If the value is below 0.01 (or so), the distortion is likely to be imperceptible.\n

\n\nOk, so 0.01 is *imperceptible* (great!), and 0.1 is “annoying.” When making quaity/size tradeoff decisions, we’ll usually want to be somewhere between these two values. But I don’t have a good intuitive sense of what SSIMULACRA scores like, say, 0.06801920 really *mean* – I bet almost nobody does.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":4,"value":"md`\n## Correlating SSIMULACRA and TID2013 Mean Opinion Scores\n\nTo get to human-readable, we’ll first need some humans. The [Tempere Image Database’s](http://ponomarenko.info/tid2013.htm) “Mean Opinion Scores” represent experimentally-determined quality scores for a set of images (on a scale from 0 [worst] thru 9 [best]), based on real-world observations of those images, by real people.\n\nHere, I’ve taken the subset of TID2013 images that have been distorted by JPEG compression, and charted how their Mean Opinion Scores (MOS) relate to their SSIMULACRA scores.`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":17,"value":"$32","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":29,"value":"tex`\\text{MOS} = -14.703 \\times \\text{SSIMULACRA} + 6.3535\\newline\nR^2 = 0.914`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":38,"value":"$33","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":130,"value":"$34","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":68,"value":"md`## Example\n\nHere's what star ratings *look* like, for the JPEG-distorted images in the dataset:\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":83,"value":"viewof imageNumber = { \n const form = html``;\n const input = form.querySelector( 'input' );\n const output = form.querySelector( 'output' );\n const update = function() {\n const padded = `00${ input.value }`.slice(-2);\n output.textContent = padded;\n form.value = padded;\n }\n input.addEventListener( 'input', update );\n update();\n return form;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":187,"value":"viewof distortionNumber = { \n const form = html``;\n const input = form.querySelector( 'input' );\n const output = form.querySelector( 'output' );\n const update = function() {\n const level = 4 - input.value;\n output.textContent = level;\n form.value = level;\n }\n input.addEventListener( 'input', update );\n update();\n return form;\n}\n","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":81,"value":"md`\n${ stars( selectedStars ) }\n\n${ selectedStars } stars; ${ descriptionFromStars( selectedStars ) }\n\n

\n`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":14,"value":"md`---`","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":192,"value":"scores = {\n const response = await fetch('https://gist.githubusercontent.com/eeeps/4d5bc3e13ea3f8a7105545a52139b56d/raw/2c03455b466c5e557337c24b5d7e46c0b5fc7ec2/ssimulacra_mos.tsv');\n const tsv = await response.text();\n const arrays = tsv\n .split( '\\n' )\n .map( ( line ) => {\n return line.split( '\\t' );\n } );\n const headers = arrays.shift(); // modifies arrays\n return arrays.reduce( ( accumulator, currentValue ) => {\n accumulator[ currentValue[ 0 ] ] = {\n ssimulacra: currentValue[ 1 ],\n mos: currentValue[ 2 ],\n stars: starsFromSsimulacra( currentValue[ 1 ] )\n };\n return accumulator;\n }, {} )\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":91,"value":"selectedImage = { \n if ( distortionNumber === 0 ) {\n return `i${ imageNumber }`;\n } else {\n return `i${ imageNumber }_10_${ distortionNumber }`;\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":251,"value":"selectedStars = {\n if ( distortionNumber === 0 ) {\n return 4;\n } else {\n return Math.round(\n 100 * parseFloat( \n scores[ `${ selectedImage }.bmp` ].stars\n ) ) / 100;\n }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":276,"value":"starsFromSsimulacra = function( ssimulacra ) {\n const s = 4 * ( -4.448 * parseFloat( ssimulacra ) + 1 );\n if ( s < 0 ) { return 0; }\n else { return s; }\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":230,"value":"descriptionFromStars = function( numberOfStars ) {\n let n;\n if ( numberOfStars >= 4 ) { n = 3; }\n else if ( numberOfStars < 0 ) { n = 0; }\n else { n = Math.floor( numberOfStars ); }\n return [ \"bad\", \"OK\", \"good\", \"great\" ][ n ];\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":62,"value":"stars = function(ratingFromZeroToFour) {\n return html`${ratingFromZeroToFour} out of 4 stars`;\n}","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":12,"value":"vegaembed = require('vega-embed')","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"},{"id":61,"value":"$35","pinned":false,"hidden":false,"mode":"ojs","author":"$undefined","summary":"$undefined","output":"$undefined","database":"$undefined"}],"files":"$W36","stdlib":"1","titling":"auto"}} 31:"$30:body"