How to Use Image Comparison
Contents
1. Introduction
2. Image Comparison Method 'default'
2.1 Algorithm Description
2.2 Usage
3. Image Comparison Method 'search'
3.1 Algorithm Description
3.2 Usage
4. VNCRobot Scripting Language and Image Comparison
1. Introduction
Unlike other automated testing software, tools working over the RFB
(VNC) protocol like VNCRobot work just with the image of the remote
desktop. That's why image comparison is one of the few means to verify
the contents of the server desktop and behavior of the tested object
(software, application, OS, ...).
Imager comparison algorithm (method) is a module which accepts two
images on input and produces a number (usually percentage) indicating
how much the images match. VNCRobot v1.3 provides two basic image
comparison methods:
- 'default' - Histogram based image
comparison method.
- 'search' - Method allowing to search
certain remote desktop image area whether it contains another image.
Users wishing to use custom image comparison algorithms may take
advantage of the VNCRobot 1.3 Open API
to write their own Java code and plug it into VNCRobot.
2. Image Comparison Method 'default'
2.1 Algorithm Description
The default method is based on histogram comparison. An image histogram
is in simple words a list of colors used in the image together with
number of pixels of each color. The method first calculates histograms
both of the remote desktop image and the template image and compares
them. Result of the image comparison is then calculated as a number of
matching pixels divided by the total number of image pixels.
The following example illustrates how the method works. Let us have two
simple images, image 1 and image 2:

Each image has 120x100=12000 pixels. The following table shows the
histograms and number of matching pixels:
Color
|
Image 1
|
Image 2
|
Matching pixels |
White |
9000 pixels
|
6000 pixels
|
6000 |
Red
|
3000 pixels
|
6000 pixels
|
3000 |
Sum of matching pixels:
|
9000 |
The comparison result is in this case 9000px/12000px, which is 0.75, or
better 75%.
2.2 Usage
Though this image comparison method is quite simple and has many
limitations, it is quite reliable and has a few advantages. It is for
example robust against changes in the location of application windows
on
the remote desktop. This is visible in the following example of three
different images which will always produce 100% match if compared:

In order to get reliable results, consider the following hints:
- Change the remote desktop background to a solid color to minimize
the number of mismatching pixels. Choose such a color which is not
contained in the application window which you are going to test.
- Remove all icons and items from the remote server desktop and
make it as simple as possible. Note that most desktops also display a
clock/date which introduce mismatching image areas.
- It is a good practise to use the 'Mouse move' command to move
your mouse pointer to a constant position before comparison. Mouse
pointer forms part of the remote desktop image and it is a source of
mismatching pixels.
- Do not perform image comparison against images with lossy
compression like e.g. JPEG. PNG format is preferred because it
preserves
100% of the image information and guarantees reliable and stable
comparison results.
- Check if the application window(s) you are testing have stable
sizes. Variable window size significantly affects the histogram based
comparison results.
Note that the default method is designed to compare two images of
the same size. If your template image has different size than the
remote desktop, both images are cropped to the size of their
intersection befor comparison. The following picture illustrates such a
situation:

When a custom comparison area is specified via the cmparea
parameter, the remote desktop image is cropped from the area location,
i.e. from its [x,y] coordinates. The values of width and height of the
area are ignored in this case a are rather derived from the
intersection
size.
It is not recommended to use the default method together with a
custom comparison area because it is easy to make a mistake and compare
rectangles which do not correspond. If you need to check a certain area
of your remote desktop image whether it contains a certain image, use
the 'search' image comparison method instead.
3. Image Comparison Method 'search'
3.1 Algorithm Description
The search method performs a simple pixel by pixel search of a template
image in the specified remote desktop image area. Though the algorithm
was slightly modified for better performance, it basically runs through
all pixels of the remote desktop image (or its part) and tries to find
match(es) of the template image. The template image must be
smaller than the remote desktop image.
The older implementation of this algorithm was not able to handle the passrate
option. It was always looking for an exact pixel-by-pixel match
regardless of the desired pass rate. This behavior was changed in
VNCRobot 1.3.6 and the new implementation is able to search even for
image occurencies which have a certain amount of pixels different from
the template image. The pass rate is newly converted to a number of
pixels (template_pixel_count * (1 - passrate)
)
and the
remote desktop image is searched for all instances having a number of
different pixels lower than or equal to this number. If your template
image size is e.g. 100x100 pixels and you specify 99% pass rate,
VNCRobot will find all areas of remote server desktop having up to 100
different pixels. Note that the lower pass rate you specify, the lower
the performance will be and the longer the search will take.
The method returns 0% match if the template is not found or 100% if at
least one match is located. Coordinates of the found matches are then
provided in form of variables which can be accessed during script
execution and further processed. See the Compareto command
specification for more info on the variable format. Also note that
the number of occurencies is limited by a parameter in the
Compareto command preferences which can be customized in the VNCRobot
Preferences window.
It is recommended to save the template images in PNG format through the
Compareto, Screenshot
of Waitfor match windows of VNCRobot. There
are tools allowing you to crop the desired area of the remote desktop
image and save it as a template. If you use images saved by other
applications, it may happen that VNCRobot will not be able to match the
image due to an incompatible pixel format.
3.2 Usage
The method can be widely used to verify that the remote desktop image
contains a desired component or window. It is also possible to search
for a component or an icon and then click onto its coordinates.
It is recommended to use the default method together with a custom
comparison area (parameter cmparea
of the scripting
language commands) to narrow the search area. Though you may search the
whole remote desktop, the algorithm is quite inefficient and it may
take
up to seconds to finish. The performance can be also severely affected
if there are too many matches. You can improve the performance by
proper
selection of the template image. The following example suggests how to
create templates for an efficient search.

Example of a remote desktop
Template Image
|
Comment
|

|
Incorrect. There will be
thousands of matches within the remote desktop's red rectangle.
|

|
Better. Though there will
be just one match, the large white stripe in the upper part will cause
delays in searching because the desktop background is white and the
method processes pixels by rows from the top left corner. |

|
Best. The first red pixel
is located close to the top left corner which will make the search
efficient. |
4. VNCRobot Scripting Language and Image Comparison
Image comparison modules are closely integrated with 3 commands of the
VNCRobot 1.3 scripting language:
- Compareto -
Compare a template image to the remote desktop.
- Screenshot -
Save a remote desktop screenshot to a file and eventually compare it to
a template image.
- Waitfor match/mismatch
- Pause the execution until ('match') or while ('mismatch') the remote
desktop matches to a template image.
All three commands support almost the same image comparison parameters.
There are Compareto, Screenshot of Waitfor
match windows allowing to construct and maintain the commands
easily. See the following picture showing a fragment of code with these
commands together with the corresponding image comparison GUI.

It is usually necessary to branch the script behavior based on the
image comparison result. There are two ways to ensure it:
- Use the
onpass
and onfail
command
parameters
(ontimeout
in case of the
Waitfor
command) to specify what should be done when image comparison succeeds
or fails. A disadvantage is that it is allowed to specify just one
command or procedure name.
- Use the
if
statement together with the command exit
code.
The following two examples are equivalent and terminate the script
execution with an exit code 1 when the image comparison fails:
# Example #1 - 'onfail' parameter
Compareto test.png method=default onfail="exit 1"
# Example #2 - 'if' statement
if ({_EXIT_CODE} > 0) {
exit 1
}
Another example demonstrates how to search for a button template image
and then click it:
Compareto button.png
method="search"
# If the button is found, the exit code will be 0
if ({_EXIT_CODE} == 0) {
Mouse click to=x:{_SEARCH_X}+5,y:{_SEARCH_Y}+5
}
See the VNCRobot 1.3 Language
Reference document for additional examples and more information on
the scripting language.