In this article I'll discuss the messier details of producing PNG intermediate images for dynamic color replacement. I'll cover garbage masks and interpolation tables...things necessary to make this technique work on real world images. Neither of these is a new concept in and of itself, of course, but I will simply detail how they are used within the color replace technique to improve the results. Note that this describes a "programmer interface" to making these images...but I will also hint at what might make a nice user interface which I might build soon.

Garbage masks

We'll start with the simpler of the two concepts, garbage masks, which are often necessary, but might be considered a last resort. Say we have a photo of a model wearing a red sweater, and we want to make the red dynamically changeable. She also is wearing red lipstick, which we want to always stay red. Also her hat has some red in it. There is probably no automatic way to distinguish between the sweater and lips/hat based on pixel color alone, so we have to use a more manual approach to specify parts of the image where the effect is allowed to be applied and where not.

Scribbling over the lips and hat in cyan makes sure they are not considered part of the sweater, even though their color is within range

So we use a garbage mask. The easiest way to create one is to copy the original image, and draw on it in a paint program. Typically we'll pick a color as far from the target hue as possible, say, cyan. (r 0%, g 100%, b 100%). Then we simply draw over the lips in pure cyan, preferably using a soft edge brush. Then our program has logic that says, if a pixel is cyan in the mask image, make sure the same pixel is left fully opaque (and therefore its original color) in the final image. We have some logic that calculates the distance from cyan, giving it a smooth falloff as the distance increases, effectively blending the opaque original pixel with the pixel as it would be without the mask. If more than 10% or so from cyan, it will be considered fully "unmasked."

If we are lucky, we can be quite sloppy when painting the mask and see no ill effects, since the area around the lips is not going to be affected anyway. This means it can be done without tedious precision work in the paint program, just a quick scribble to mask out the lips and hat. Occasionally we'll have an image where we need to mask very close to the part we don't want to mask, which can be time consuming. Whenever possible, though, we'd like to select the affected areas by their color alone, to eliminate this manual step.

Interpolation tables

To explain the concept of an interpolation table, let me start with a simple example. Imagine I want to write a program to increase the brightness of a gray scale ("black and white" image). I might say, for each pixel, multiply its value by 1.4. However, if that makes the pixel brighter than 100% (white), simply make it 100%.

Then I decide that that isn't so good, as everything brighter than 70% becomes pure white. So, let's make it so if it is within 10% of white, reduce the value we multiply it by. We might come up with a formula for doing this that makes it less abrupt than having an absolute cutoff of 10%.

We might also notice that black pixels are staying black, which isn't what we want. So we might change our formula so that it first adds 5% to every pixel, prior to multiplying. And so on.

While we could probably come up with an elegant mathematical formula to do this, honestly, we don't really want to have to think that hard. Instead, what if we just make a list of pairs of numbers, that says "if the pixel is originally 10%, make it 30%. If it is 25%, make it 70%. If it is 55%, make it 95%" (etc). If the value isn't found in the list, we can estimate ("interpolate") the in-between values.

A graphical representation of the interpolation table at left (blue lines representing actual pairs in the table)

var table = new InterpolationTable (
 [0, 5], [10, 30], [25, 70],
 [55, 95], [95, 97], [100, 100] );
 
var a = table.mapValue(40);
// "a" will now be 82.5 (40 is halfway between 
// 25 and 55, so the result will be halfway
// between 70 and 95

While the algorithm I used to do the interpolation works (you can see the function in the Chameleon code, where it is used for something else), it could probably be sped up quite a bit, by smart programmers who like to think about that kind of problem. (hint: bsearch. or precompute a larger table. or whatever)

So, on to how to use these tables to make the conversion process produce better results than the "raw" process described in the previous article.

Say we have an image, such as the one above, of a model wearing a red sweater, and whose skin is typical flesh tone: a pale orange. Since orange is near red, it will be affected as the color changes, but we don't want it to be. While we could use a garbage mask to cover all her skin, it would be nice if we could do something a bit more automated, especially since such a mask would likely have to be high precision where the sweater ends and the skin begins.

A simple solution is to apply a cutoff, so that if the hue varies by more than, say, 15 degrees in the positive direction (that is, toward yellow), the pixel will be left opaque and its original color. Of course we don't want a hard cutoff point, but a more gradual falloff, much as we did with the garbage mask (in that case, by using a soft edge brush to paint it the mask). We can shape this curve using an interpolation map so that it includes the pixels we want and excludes those we don't. If we see a lot of "halfway" pixels, whose transparency is partially affected by hue, we should make the table have a more abrupt falloff to eliminate this bothersome effect. If we see some abrupt color changes that look wrong, we can have it more smooth. And of course we can adjust where the falloff happens (the "width" of hue affected). With the right user interface (such as one that shows the image with color removed and transparency applied, and a maybe a moving checkerboard pattern behind the image to make it apparent where the transparency is), a user could do this in seconds with a few slider controls (width and falloff rate on each side of the target hue).

Another issue is that we might have colored reflections in the surface that appear incorrect when the color changes. Say a blue sky is reflecting on a shiny red car. If we make the car black, these might show up as purple rather than the blue they should be. One (rather half-assed) solution is to simply "pre-pinch" the hue to be pure red....making the car would always look as if the sky was neutral gray. Not perfect, but not terrible either: most people would not notice a big problem with the image, but it does make the image appear different than the original even when the background hue matches the target hue (for example, when the car is red to begin with, and we drag the color widget to red).

Again, the interpolation table comes in handy. Since we already rotate the hue so that the target is red before performing our calculation (and shift it back afterwards), this is fairly easy. For instance we might do this:

var table = new InterpolationTable (
  [0, 0], [5, 1.5], [10, 3], [15, 5],
  [35, 12], [40, 55], [55, 59], [60, 60] 
  );
 
if (hsv.hue < 60)  // red to yellow range
  hsv.hue = table.mapValue(hsv.hue);
else if (hsv.hue > 300) // red to magenta range
  hsv.hue = 360 - table.mapValue(360-hsv.hue);

This will narrow the range of hues that are in the "no man's land" that would tend to appear incorrect when the base color changes. Here, that range is between 35 and 40 degrees (a narrow band of orange, and another narrow range of fuschia). The example code above applies the effect symmetrically in both the yellow and magenta directions, but this does not have to be the case.

A more sophisticated approach would leave the hue of the original pixel the same, but adjust the effect the hue shift has on the transparency, which will in turn adjust the color of the final pixel. This way the image would appear identical when the background hue matches the target hue, a benchmark we strive for. It should be possible to make sure the reflection is indeed blue (as opposed to, say, purple). On most images there will only be a couple of colors noticeably reflected in the object (say, green trees and blue sky reflected in a car), so on such an image this should give very good results if done well. I have done some experimentation with this approach, but none of the images in the demo uses it, as it is extremely tedious to get right using my "programmer interface." A good user interface might allow you to select a pixel with the mouse (say, in what should be a blue reflection), then drag a slider to determine which of the range of possible hues the reflection itself should be. The appropriate interpolation map would then be automatically generated, and the image recomputed. It would probably be easiest to judge with the image shown against a black background, but of course you would be able to use the color picker to see it against any color.

If the original image is not saturated enough, such as if the original color is rather muted, we might not be able to get a good white, black, or any pure color. The image where we change the skin color of the woman is a good example of this type of situation (although it helped that she was not fair skinned). To address this we can push the saturation of the color with an interpolation table make it work better. We don't have to do it to all colors in the image, but only the ones within the target range. For her image, I used two interpolation tables, taking into account both value and saturation, in order to get the most impressive (and surprisingly realistic) effect.

At any stage in the calculation, we can use interpolation tables to tweak the results. To produce the images you see, I was able to choose from a set of tables, or type in new values, in a web page. Still, this is a bit too tedious and geeky an interface ... not only does it require an intense understanding of what is happening, but it just takes too long to tweak it into shape. All of the demo images could be greatly improved with the interface I envision, given that I wasn't willing to spend hours on each image experimenting.