Repository navigation
Rendering Repeatability
Rendering algorithms used by PBRT and Mitsuba are stochastic because they sample light rays from an enormous space of possible light rays. This means the output renderings will always contain some noise. Usually we don't notice the noise: if a rendering looks "good" then a repeat rendering of the same scene should also look "good", and the two renderings should look the same.
To check the repeatability of renderings, we rendered all of the RenderToolbox3 Example Scenes twice and compared all the pairs of outputs. We ran the utility function TestAllExampleScenes() twice, to generate two complete sets of renderings. We used CompareAllExampleScenes() to analyze and summarize the two sets.
Each set of renderings contained 304 distinct renderings. Each distinct rendering represents a particular variant of a particular parent scene, as rendered by a particular renderer.
Generally, each rendering and its repeated counterpart looked the same by visual inspection. Using two simple comparison metrics, we found that pairs or repeated renderings generally contained small overall differences and small per-pixel differences.
CompareAllExampleScenes() produced a summary of comparisons between repeated renderings.
The summary plot sorts these renderings using the Relative Difference metric described below. Only the 26 renderings with the largest Relative Differences are shown. These were the 26 "least repeatable" renderings.
Summary of the 26 "least repeatable" scenes.
The plot on the left shows the overall correlation between repeated renderings as blue circles. This metric treats the renderings as large arrays of pixel components, and ignores the 3-dimensional structure of the renderings (height x width x wavelengths). The comparison statistic is the correlation coefficient between the large array from the first rendering, and the large array from the second rendering.
Generally, the correlation coefficients are close to 1. The least correlation was about 0.9, which occurred for the 17th variant of the CubanSphere scene, as rendered by PBRT. For some reason, this rendering was not as "repeatable" as others. This may be related to how PBRT sampled rays in regions of specular reflectances or sharp edges, both of which are part of the CubanSphere scene.
The plot on the right compares individual pixel components between repeated renderings. This metric is based on the difference image formed by subtracting pixel components in the second rendering from the corresponding components in the first. For each pixel component, we express the absolute difference as a fraction of the component value from the first rendering.
When component values in the fist rendering (the denominator) are very small, the relative difference may appear unfairly large. So we ignore differences where the components of the first image fall below an arbitrary threshold (1/5 of the max value).
Black circles show the mean relative difference between corresponding pixel components in the first and second renderings. These values are generally close to 0, indicating that corresponding pixel components generally agree across renderings.
Red crosses show the max relative difference between components. This indicates the single spectral component of the single pixel that showed the greatest difference across renderings. Often this component has an relative difference close to 50%. This difference may be tolerable for single pixel components, as long as components in the rest of the rendering have smaller differences. We might think of a speck of dust sitting on an otherwise pristine photograph.
The worst relative error occurred for a pixel component in the 1st variant of the Checkerboard scene as rendered by Mitsuba. This may relate to ray sampling at the sharp edges between checkers.
We found that agreement between renderings can vary quite a bit between comparison test runs.
We generated three sets of CubanSphere renderings on our test machine, one immediately after the other. We ran CompareAllExampleScenes() on each of the three pairs of rendering sets. For one pair of rendering sets, the worst rendering correlation was about 0.85. For another pair of rendering sets, the worst correlation was about 0.95.
So even with no changes made to the CubanSphere recipe, nor to the RenderToolbox3 code, we should expect significant variability across renderings and comparisons of renderings.
CubanSphere seems to be our least forgiving recipe, with respect to rendering repeatability and repeatability of comparisons. CubanSphere might be challenging for renderers because it contains sharp edges, a contrasting black background, and specular reflectances. Other recipes seemed to render more consistently.
CompareAllExampleScenes() also produced a detailed comparison figure for each pair of repeated renderings. These show the first and second renderings as converted to sRGB. They also show absolute difference images formed by subtracting the first rendering from the second, or the second from the first. The difference images are also converted to sRGB. All four sRGB images are scaled by maximum luminance.
Visual inspection of these comparison figures provides an intuitive "metric" for rendering repeatability. Here are three of the "least repeatable" renderings, selected from the summary above. Generally, the first and second renderings look the same, and the difference images contain a few noisy specks or strange artifacts.
The 17th variant of the CubanSphere scene as rendered by PBRT gave the worst overall correlation between first and second renderings.
Above, PBRT rendered the 17th variant of the CubanSphere scene, twice.
The 1st variant of the Checkerboard scene as rendered by Mitsuba gave the worst relative error for a single pixel component, between first and second renderings.
Above, Mitsuba rendered the 1st variant of the Checkerboard scene, twice.
This demonstration included 278 other renderings that don't appear in the summary figure above. These renderings were all "at least as repeatable" than the examples shown. They had higher rendering correlations and lower relative errors between pixel components.
For this demonstration, we Rendered all of the RenderToolbox3 example scenes twice, using the TestAllExampleScenes() utility function.
We did renderings on two machines:
- 10 September 2015
- an Ubuntu instance on Amazon EC2
- Matlab R2015a
- RenderToolbox3 version
- Mitsuba 0.5.0
- PBRT version
- 19 September 2015
- an OS X desktop machine
- Matlab R2015a
- RenderToolbox3 version
- Mitsuba 0.5.0
- PBRT version