In the latter part of 2008 developments to the Colour Recovery software application had to take second place to actually using it to process programmes such as the Dad's Army and Doctor Who episodes. It is undoubtedly also the case that the 'politics' surrounding the Colour Recovery Working Group had the effect of sapping my enthusiasm for further software work.
One positive effect of leaving the Working Group has been to renew my interest in thinking of ways to improve my process, although the opportunities for improvement are I fear relatively small. This page will record ongoing developments, some of which may be incorporated in released versions of the software but others may not.
23rd January 2009
Separation of luminance and chrominance is obviously a vital element of any Colour Recovery process, and mine does it by means of a 2-dimensional filter which passes only the diagonal chroma patterns at [4.433 MHz, 72 c/aph] and [4.433 MHz, 216 c/aph]. Having separated out the chrominance in this way, the luminance is derived by subtracting the chrominance from the composite input signal. The filter curves and overall 2D response of this separation process can be seen here.
Inevitably the separation is far from perfect, because any luminance content with diagonal components near the two spatial frequencies listed above will be treated as chrominance, resulting in cross-colour and (worse) undesirable effects on the Quadrant Assignment process. Although suggestions have been made for possible ways of improving the discrimination between luma and chroma (some based on principles similar to the Transform PAL Decoder) no practical method has as yet been devised.
27th January 2009
As mentioned above, cross-colour can disrupt the Quadrant Assignment process, and this can be particularly serious because of the latter's reliance on propagating information from one part of the frame to another (analogous to a flood-fill algorithm in computer graphics). In principle a single incorrectly-measured block can cause an erroneous quadrant assignment over a significant proportion of the frame: a highly undesirable characteristic!
One noticeable feature is that the measurement of vertical displacement is less vulnerable to this effect than the measurement of horizontal displacement. The reason is not hard to find: the vertical measurement relies on the presence of both a 72 c/aph and a 216 c/aph component in the chroma patterning, whereas the horizontal measurement uses only the 72 c/aph component. It is relatively unlikely that luminance will contain both 72 c/aph and 216 c/aph components in the same block.
It occurred to me that it should be possible to make the horizontal displacement measurement less sensitive to cross-colour by also using both the 72 c/aph and 216 c/aph components. This can be achieved because when you mix those two signals together you create not only the sum (which is the 288 c/aph signal used for the vertical measurement) but also the difference (which consists of a signal with a vertical frequency of 144 c/aph and a horizontal frequency of twice colour subcarrier, i.e. nominally 8.866 MHz).
Now, 8.866 MHz is exactly the right horizontal frequency to be used for the horizontal displacement measurement! It's the same frequency as results from taking the absolute value of the [4.433 MHz, 72 c/aph] signal, as described in the how it works section. The 144 c/aph vertical frequency is an inconvenience, but no more so than with the original method.
I have experimentally implemented this alternative process and it does indeed seem to make the horizontal measurement less sensitive to cross-colour. The disadvantage is that the 216 c/aph signal is often weaker than the 72 c/aph signal, so the resulting horizontal measurements are somewhat noisier. I am currently carrying out tests to see whether, on balance, results are better using this technique.
28th January 2009
The tests are still ongoing - it needs a large number of comparisons before one can be confident a modification to the process if definitely beneficial - but the indications are promising. Here are the outputs of the 'old' and 'new' versions of the software on one of the more troublesome frames from Dad's Army - Room at the Bottom:
6th February 2009
A minor diversion to discuss a mathematical curiosity. The horizontal displacement measurement must be insensitive to the signs of U and V (i.e. whether they are negative or positive) and in the original scheme this was achieved by taking the absolute values of their 72 c/aph components. This has the effect of doubling the spatial frequency, so for example the U component, with a frequency of [4.433 MHz, 72 c/aph], becomes [8.866 MHz, 144 c/aph] when the absolute value is taken.
Another way of looking at this is that taking the absolute value is equivalent to squaring and then taking the square-root (i.e. ABS(x) = SQR(x*x)) so the effect on the frequency is the same as multiplying the signal by itself. When you multiply two periodic signals together you get sum and difference frequencies, so multiplying a signal by itself simply doubles the frequency.
In the new scheme, using both 72 c/aph and 216 c/aph components for the horizontal displacement measurement, insensitivity to the signs of U and V comes about without any extra effort. This is because when you multiply the 72 c/aph signal by the 216 c/aph signal, to generate sum and difference frequencies, the signs cancel out.
Looking in detail at the U chrominance, it consists of a signal at [4.433 MHz, 72 c/aph] and a signal at [-4.433 MHz, 216 c/aph]; note particularly the signs of the horizontal frequencies. So when you multiply the two together you get a difference frequency of [8.866 MHz, -144 c/aph]. What's interesting about this is that it's the same as what is produced by the original scheme except that the sign of the vertical frequency is negated! Physically what this means is that the 2D pattern of stripes slopes in the opposite direction (downwards to the right with the original method and downwards to the left with the new method).
Well, I thought it was interesting anyway!
11th February 2009
I have carried out numerous comparisons between the results obtained using only the 72 c/aph component for horizontal measurements, and those obtained using both the 72 and 216 c/aph components. The answer appears to be... it depends!
With much of the test material it makes little or no difference which mode is used. With 'good quality' material containing plenty of high frequency vertical energy (for example the Dad's Army episode) using both 72 and 216 c/aph helps protect against the adverse effects of cross colour. With 'poor quality' material containing little high frequency vertical energy (including much of the Doctor Who material) using only 72 c/aph gives cleaner and less noisy horizontal measurements.
Although it is not an ideal solution in every respect, there appears to be no alternative but to make this a configuration option. The user can then choose whichever of the modes seems to give the best results with the particular material concerned. I have accordingly extended the 'Settings' dialogue as below:
19th February 2009
A very sad day for Colour Recovery. I have received this from James Insell at the BBC:
I regret that I have to terminate the licences you have been granted. Please provide a written statement confirming that no copies of the licensed information have been retained. Should it be more convenient for you to return the portable hard drives that the information was provided on, rather than delete the data, then I leave this to your discretion.
The source material to which I have been denied access includes all the test sequences (e.g. Top of the Pops) which I have used since the start of the project to evaluate my process and assess the impact of any changes to the software. Without this material being available to me I am unable to continue developing the Colour Recovery application, and the progress I have been making recently has gone to waste.
21st February 2009
Rather than waste the recent enhancements to my Colour Recovery process I decided to risk the wrath of the BBC by delaying deletion of the test material for a couple of days. This has allowed me to issue a 'rush' release of the software application (version 1.20a) which has the following improvements compared with the previous release:
- The vertical chrominance filters have been improved; in particular the stopband attenuation is now at least -48dB.
- The horizontal chrominance filter has been improved, and its centre frequency has been shifted slightly to be nearer the actual subcarrier frequency found in typical Film Recorded material.
- The user can now select either 72 c/aph only or 72+216 c/aph modes for the measurement of horizontal distortion. The former usually works better for material with poor vertical frequency response (e.g. a large amount of spot-wobble) and the latter with material having plenty of vertical high-frequency energy.
- The disambiguation algorithm has been completely rewritten, reducing the likelihood of propagating erroneous distortion measurements to other regions.
Whilst each of these modifications individually has a relatively minor impact, together they result in a worthwhile improvement in performance. However, for obvious reasons, I have not been able to carry out the degree of testing that I would normally prefer.
1st June 2010
I realised that I could make an improvement to the error concealment algorithm. To reduce the number of quadrant assignment errors, all versions of the process to date accept a new quadrant value for a particular block only if that same value is calculated on two consecutive frames. If the value changes only for one frame, it is deemed a transient error and ignored.
The weakness of this algorithm is that if the hue in the block is close to one of the axes (i.e the U or V value is very small) it is to be expected that the quadrant will change from frame to frame, e.g. because of noise. This does not represent an error, but the error concealment algorithm will behave as though it is.
To overcome this weakness I have modified the algorithm so that rather than comparing quadrants it compares octants. If the octant value is either the same for two consecutive frames, or differs by one (i.e. the two octants are adjacent) it will be accepted. Now, a small frame-to-frame variation in the hue will not cause the quadrant assignment to be rejected, even if the hue crosses the U or V axis.
This map, derived from a 'colour wheel', indicates the different octants by using slightly darker colours. The boundaries between the octants are rather irregular because they depend on the amplitudes of the demodulated U and V signals, not just the signs.
20th June 2010
Something I've been meaning to do for a long time is to recode some of the more speed-critical assembly language routines to use MMX instructions. I've now completely rewritten the cross correlation and horizontal filter routines (for various reasons the vertical filter is not amenable to be adapted to MMX). The overall speed improvement is modest (about 6%) but every little helps!
2nd September 2010
As mentioned elsewhere, the Colour Recovery utilities I have developed use QuickTime as a convenient file interchange format. One disadvantage of QuickTime is that it has a tendency to perform unwanted YUV to RGB conversions internally; for example when reading a YUV-encoded input file into a YUV memory buffer (which is something my utilities commonly do) the video data may be converted from YUV to RGB and then back to YUV on the way!
Not only is this quite unnecessary and inefficient, the process is lossy. For example it results in the luminance being clipped to the range 16 to 234, which is highly undesirable, and the chrominance being upconverted from 4:2:2 to 4:4:4 and then downconverted to 4:2:2 again, which may introduce artefacts. Unfortunately I don't know of any way to disable this behaviour.
I have therefore added a software workaround to the Quadrant Editor and Splice utilities. Now, if they detect that the input file is in an uncompressed YUV format ('2vuy' or '2Vuy') they bypass the QuickTime codec completely and read the video data directly from the file. This ensures that they are transparent to uncompressed YUV data.
15th December 2010
I have been verifying the operation of the Colour Recovery process using a source frame consisting of white noise, and looking at the Process noise spectra. I noticed that in the 216 c/aph X mode the 360 c/aph component (which theoretically shouldn't be present in the source anyway) wasn't contributing anything useful to the horizontal (X) measurement, because although ending up at the correct vertical frequency (144 c/aph) the horizontal frequency had the wrong sign!
In addition, using the U/V separation filter on the 360 c/aph component, which was a necessary step in contributing to the horizontal measurement, wasn't very effective because the filter's vertical 'separation frequency' is actually 67.5 c/aph rather than 72 c/aph. Although the error is acceptably small at 72 and 216 c/aph, at 360 c/aph it is significant.
So using the 360 c/aph component (if present) contributes nothing useful to the horizontal measurement process, and indeed may add unwanted noise; in version 1.26a this has been removed. The 360 c/aph component is still used in the vertical measurement process, because the above issues don't apply (no U/V separation is required) and it has been shown to be beneficial on some source material.