How to Use Image Comparison


Contents
1. Introduction
2. Image Comparison Method 'default'
    2.1 Algorithm Description
    2.2 Usage
3. Image Comparison Method 'search'
    3.1 Algorithm Description
    3.2 Usage
4. VNCRobot Scripting Language and Image Comparison

1. Introduction

Unlike other automated testing software, tools working over the RFB (VNC) protocol like VNCRobot work just with the image of the remote desktop. That's why image comparison is one of the few means to verify the contents of the server desktop and behavior of the tested object (software, application, OS, ...).

Imager comparison algorithm (method) is a module which accepts two images on input and produces a number (usually percentage) indicating how much the images match. VNCRobot v1.3 provides two basic image comparison methods:
Users wishing to use custom image comparison algorithms may take advantage of the VNCRobot 1.3 Open API to write their own Java code and plug it into VNCRobot.


2. Image Comparison Method 'default'

2.1 Algorithm Description

The default method is based on histogram comparison. An image histogram is in simple words a list of colors used in the image together with number of pixels of each color. The method first calculates histograms both of the remote desktop image and the template image and compares them. Result of the image comparison is then calculated as a number of matching pixels divided by the total number of image pixels.

The following example illustrates how the method works. Let us have two simple images, image 1 and image 2:

Default method example images

Each image has 120x100=12000 pixels. The following table shows the histograms and number of matching pixels:

Color
Image 1
Image 2
Matching pixels
White 9000 pixels
6000 pixels
6000
Red
3000 pixels
6000 pixels
3000
Sum of matching pixels:
9000

The comparison result is in this case 9000px/12000px, which is 0.75, or better 75%.

2.2 Usage

Though this image comparison method is quite simple and has many limitations, it is quite reliable and has a few advantages. It is for example robust against changes in the location of application windows on the remote desktop. This is visible in the following example of three different images which will always produce 100% match if compared:

Examples of matching images

In order to get reliable results, consider the following hints:
Note that the default method is designed to compare two images of the same size. If your template image has different size than the remote desktop, both images are cropped to the size of their intersection befor comparison. The following picture illustrates such a situation:

Cropping of comparison images

When a custom comparison area is specified via the cmparea parameter, the remote desktop image is cropped from the area location, i.e. from its [x,y] coordinates. The values of width and height of the area are ignored in this case a are rather derived from the intersection size.

It is not recommended to use the default method together with a custom comparison area because it is easy to make a mistake and compare rectangles which do not correspond. If you need to check a certain area of your remote desktop image whether it contains a certain image, use the 'search' image comparison method instead.


3. Image Comparison Method 'search'

3.1 Algorithm Description

The search method performs a simple pixel by pixel search of a template image in the specified remote desktop image area. Though the algorithm was slightly modified for better performance, it basically runs through all pixels of the remote desktop image (or its part) and tries to find match(es) of the template image. The template image must be smaller than the remote desktop image.

The older implementation of this algorithm was not able to handle the passrate option. It was always looking for an exact pixel-by-pixel match regardless of the desired pass rate. This behavior was changed in VNCRobot 1.3.6 and the new implementation is able to search even for image occurencies which have a certain amount of pixels different from the template image. The pass rate is newly converted to a number of pixels (template_pixel_count * (1 - passrate)) and the remote desktop image is searched for all instances having a number of different pixels lower than or equal to this number. If your template image size is e.g. 100x100 pixels and you specify 99% pass rate, VNCRobot will find all areas of remote server desktop having up to 100 different pixels. Note that the lower pass rate you specify, the lower the performance will be and the longer the search will take.

The method returns 0% match if the template is not found or 100% if at least one match is located. Coordinates of the found matches are then provided in form of variables which can be accessed during script execution and further processed. See the Compareto command specification for more info on the variable format. Also note that the number of occurencies is limited by a parameter in the Compareto command preferences which can be customized in the VNCRobot Preferences window.

It is recommended to save the template images in PNG format through the Compareto, Screenshot of Waitfor match windows of VNCRobot. There are tools allowing you to crop the desired area of the remote desktop image and save it as a template. If you use images saved by other applications, it may happen that VNCRobot will not be able to match the image due to an incompatible pixel format.

3.2 Usage

The method can be widely used to verify that the remote desktop image contains a desired component or window. It is also possible to search for a component or an icon and then click onto its coordinates.

It is recommended to use the default method together with a custom comparison area (parameter cmparea of the scripting language commands) to narrow the search area. Though you may search the whole remote desktop, the algorithm is quite inefficient and it may take up to seconds to finish. The performance can be also severely affected if there are too many matches. You can improve the performance by proper selection of the template image. The following example suggests how to create templates for an efficient search.

Example remote desktop

Example of a remote desktop

Template Image
Comment
Template #1
Incorrect. There will be thousands of matches within the remote desktop's red rectangle.
Template #2
Better. Though there will be just one match, the large white stripe in the upper part will cause delays in searching because the desktop background is white and the method processes pixels by rows from the top left corner.
Template #3
Best. The first red pixel is located close to the top left corner which will make the search efficient.


4. VNCRobot Scripting Language and Image Comparison

Image comparison modules are closely integrated with 3 commands of the VNCRobot 1.3 scripting language:
All three commands support almost the same image comparison parameters. There are Compareto, Screenshot of Waitfor match windows allowing to construct and maintain the commands easily. See the following picture showing a fragment of code with these commands together with the corresponding image comparison GUI.

Comparison parameters


It is usually necessary to branch the script behavior based on the image comparison result. There are two ways to ensure it:
The following two examples are equivalent and terminate the script execution with an exit code 1 when the image comparison fails:

# Example #1 - 'onfail' parameter
Compareto test.png method=default onfail="exit 1"

# Example #2 - 'if' statement
if ({_EXIT_CODE} > 0) {
  exit 1
}

Another example demonstrates how to search for a button template image and then click it:

Compareto button.png method="search"

# If the button is found, the exit code will be 0
if ({_EXIT_CODE} == 0) {
  Mouse click to=x:{_SEARCH_X}+5,y:{_SEARCH_Y}+5
}


See the VNCRobot 1.3 Language Reference document for additional examples and more information on the scripting language.