Image Operation APIs

Image Operation APIs provide functions for reading, writing, processing, and comparing images for automation workflows. These functions are based on the Image object.

`Image` Object

Importing

Add the following script at the top of your script to reference the Image class:

JavaScript

Python

const { Image } = require('leanpro.visual');

from leanproAuto import Image

The Image object provides the following methods and properties:

Method Name	Description
width	Image width.
height	Image height.
clip	Clips the image according to a specified rectangular area and returns a new `Image` instance.
drawImage	Draws another image onto the current image and returns a new `Image` instance.
filterColor	Applies a color filter to the image, supporting single or multiple colors, and returns the processed image.
getData	Gets the image data, supporting `Buffer` or Base64 encoded string formats.
resize	Scales the image to the specified width and height, returning the scaled image.
save	Saves the image to the specified file path.
fromData	Creates a new `Image` instance from binary data or a string (Static method).
fromFile	Loads an image from a file path and creates a new `Image` instance (Static method).
imageEqual	Compares two images for equality, supporting pixel tolerance, area ignore options, etc. (Static method).
imageCompare	Compares two images and generates a difference result, returning comparison analysis data (Static method).

Type Definitions

The Image object contains member properties, member methods, and static methods, enabling most image operations to be performed on the Image object.

The Image object is defined as follows:

JavaScript

Python

class Image {
    width: number;
    height: number;
    clip(rect: {x: number, y: number, width: number, height: number}): Image;
    drawImage(image: Image, x: number, y: number): Image;
    filterColor(color: Colors | Colors[]): Promise<Image>;
    getData(option?: {encoding: 'base64' | null}): Promise<Buffer | string>;
    resize(width: number, height: number): Promise<Image>;
    save(filePath: string): Promise<void>;

    static fromData(bufferOrString: Buffer | string): Promise<Image>;
    static fromFile(filePath: string): Promise<Image>;
    static imageEqual(image1: Buffer | string | Image,
        image2: Buffer | string | Image, 
        options?: CompareOptions,
        compareInfo?: ImageCompareInfo): Promise<boolean>;
    static imageCompare(image1: Buffer | string | Image,
        image2: Buffer | string | Image, 
        options?: CompareOptions): Promise<ImageCompareResult>
}

class Image:
  def width(self) -> int:
  def height(self) -> int:
  def clip(self, rect: Rect) -> "Image":
  def drawImage(self, overlay_img: "Image", position=(0,0)) -> "Image":
  def getData(self, option: Optional[Any]) -> bytes:
  def save(self, filePath: str) -> None:
    
  @staticmethod
  def fromData(bytes: bytearray) -> "Image":
  @staticmethod
  def fromFile(filePath: str) -> "Image":
  @staticmethod
  def imageEqual(self, image1: Union[str, bytearray], image2: Union[str, bytearray], options: Optional[any]=None, compareInfo: Optional[any]=None) -> "bool":
  @staticmethod
  def imageCompare(self, image1: Union[str, bytearray], image2: Union[str, bytearray], options: Optional[any]=None) -> ImageCompareResult:

Common Usage

A common usage of the Image object is to read two images and compare them. The comparison process is as follows:

Use Image.fromData() or Image.fromFile() to read two images and generate their respective Image instances;
Use Image.imageCompare() to compare the differences between the two Image objects. The script is as follows:

JavaScript

Python

const { Image } = require('leanpro.visual');
const fs = require('fs');
(async function() {
    let buf1 = fs.readFileSync('c:/temp/image1.png');
    let image1 = await Image.fromData(buf1);
    console.log(`The width of image1 is ${image1.width}, and the height is ${image1.height}`);
    
    let image2 = await Image.fromFile('c:/temp/image2.png');
    console.log(`The width of image2 is ${image2.width}, and the height is ${image2.height}`);
})()

from leanproAuto import Image

buf1 = None
with open('c:/temp/image1.png', "rb") as f:
    buf1 = f.read()
image1 = Image.fromData(buf1)
print(f"The width of image1 is {image1.width}, and the height is {image1.height}")

image2 = Image.fromFile('c:/temp/image2.png')
print(f"The width of image2 is {image2.width}, and the height is {image2.height}")

The Image.fromData() method can accept a Buffer or base64 string to generate an Image instance. For details, see Image.fromData();
The Image.fromFile() method can accept an image file path to generate an Image instance. For details, see Image.fromFile().

The above examples read image data from a Buffer and a file, respectively, and print the width and height.

Note: You cannot create an object directly using new Image(). You need to use fromData or fromFile to generate an Image instance. Additionally, the Image object currently only supports images in PNG format.

Instance Attributes

width

The width of the image, which is a number.

JavaScript

Python

const width = await image.width
assert.equal(width, 200)

assert image1.width() == 200

height

The height of the image, which is a number.

JavaScript

Python

const height = await image.height
assert.equal(height, 200)

assert image1.height() == 200

Instance Methods

clip(rect: Rect)

Used to crop an image, returning another Image object. Pass a Rect object (i.e., {x:number, y: number, width: number, height: number}) to crop the image according to the rectangle. If the passed data is invalid resulting in a cropped image with width or height of 0, it returns null.

JavaScript

Python

const clipImage = image.clip({x:0, y:0, width: 200, height: 200})
assert.equal(clipImage.width, 200)
assert.equal(clipImage.height, 200)

clipImage = image.clip({x:0, y:0, width: 200, height: 200})

assert clipImage.width() == 200
assert clipImage.height() == 200

resize(width: number, height: number)

Used to resize an image, returning another Image object. Passing a new width and height will scale the current image to the corresponding dimensions. If the aspect ratio of the passed width and height is inconsistent with the original image, the returned result will be distorted accordingly.

JavaScript

const resizeImage = await image.resize(200, 200)
assert.equal(await resizeImage.width, 200)
assert.equal(await resizeImage.height, 200)

filterColor(color: Colors | Colors[])

Extracts specified colors from the image, returning an image containing only the target colors. Passing one or more colors will filter out all colors except the specified ones and return a new image object. Supported colors and color codes are as follows:

JavaScript

export enum Colors {
  black = 'black',
  grey = 'grey',
  white = 'white',
  red = 'red',
  orange = 'orange',
  yellow = 'yellow',
  green = 'green',
  cyan = 'cyan', 
  blue = 'blue',
  purple = 'purple'
}

The script to extract colors can be written as:

JavaScript

let image = await Image.fromFile(filepat)
let imageOnlyRed = await image.filterColor("red"); // Keep only red
let imageOnlyYellowGreen = await image.filterColor(["yellow", "green"]); // Keep only yellow and green

drawImage(image: Image, x: number, y: number)

Used to draw another image onto the current image. Returns the newly drawn image (Image object).

For example:

JavaScript

Python

let screenshot1 = await model.getButton("Five").takeScreenshot();
let screenshot2 = await model.getButton("Six").takeScreenshot();
let image1 = await Image.fromData(screenshot1);
let image2 = await Image.fromData(screenshot2);

image1 = image1.clip({ x: 10, y: 10, width: image1.width - 20, height: image1.height - 20 })
image2 = image2.clip({ x: 10, y: 10, width: image2.width - 20, height: image2.height - 20  })

let combinedImage = image1.drawImage(image2, image1.width + 5, 0);

# Get screenshot
screenshot1 = model.getButton("Five").takeScreenshot()
screenshot2 = model.getButton("Six").takeScreenshot()

# Construct Image object
image1 = Image.fromData(screenshot1)
image2 = Image.fromData(screenshot2)

# Crop (remove 10px border)
image1 = image1.clip({
    "x": 10,
    "y": 10,
    "width": image1.width - 20,
    "height": image1.height - 20
})
image2 = image2.clip({
    "x": 10,
    "y": 10,
    "width": image2.width - 20,
    "height": image2.height - 20
})

# Combine: Draw image2 to the right of image1, with a 5px gap
combined_image = image1.drawImage(image2, position=(image1.width() + 5, 0))

The above code crops 10 pixels from the surrounding edges of two button screenshots and then draws them onto a single image with a 5-pixel gap between them.

If you want the two images to be arranged vertically, you can use code like this:

JavaScript

Python

let combinedImage = image1.drawImage(image2, 0, image1.height() + 5);

combined_image = image1.drawImage(image2, position=(0, image1.height() + 5))

You can also use negative x, y values so that the second image is drawn on the other side of the first image. For example, the following statement will draw image2 on the left and image1 on the right:

JavaScript

Python

let combinedImage = image1.drawImage(image2, -(image1.width + 5), 0);

combined_image = image1.drawImage(image2, position=(-(image1.width() + 5), 0))

getData(filePath: string)

Returns the content data of the image, which can be used to save to a file or add to a report. Note that its return value type is a Promise, so await needs to be added in async functions.

A common usage is reading a local image file and then uploading it to the run report as an attachment:

JavaScript

Python

let image1 = await Image.fromFile('c:/temp/image1.png');
let image1Buffer = await image1.getData();
this.attach(image1Buffer, 'image/png');

image1 = Image.fromFile('c:/temp/image1.png')
image1_buffer = image1.getData()
request.attach(image1_buffer, 'image/png')

save(filePath: string)

Saves the image. Passing a file path will save the Image object to the target path. It saves as a .png file by default. Note that its return value type is a Promise, so await needs to be added in async functions.

JavaScript

Python

const imagePath = path.join(process.cwd(), './testing-cache', './image.png');
await image.save(imagePath)

image.save(image_path)

Static Methods

Image object static methods do not need to be instantiated; they can be called directly via Image.[staticMethodName](). This includes construction methods for Image objects (such as Image.fromData() and Image.fromFile()), and methods for comparing differences between two images (such as Image.imageCompare() and Image.imageEqual()).

Image.fromData(bufferOrString: Buffer | string)

Creates an Image object from a Buffer or base64 string data.

JavaScript

Python

import { Image } from 'leanpro.visual';

const buffer = Buffer.from('image-data');
Image.fromData(buffer).then(image => {
  // do something with the image object
});

image_path = "image.png"
with open(image_path, "rb") as f:
  data = f.read()
image1 = Image.fromData(image_path)

Image.fromFile(filePath: string)

Creates an Image object from a file. This method only supports image files in PNG format. If a non-PNG file is passed, it will fail to parse and create the image object.

JavaScript

Python

import { Image } from 'leanpro.visual';

const filePath = '/path/to/image.png';   // Supports PNG only
Image.fromFile(filePath).then(image => {
  // do something with the image object
});

image_path = "image.png"   # Supports PNG only
image1 = Image.fromFile(image_path)

To process other formats (such as JPG, BMP, etc.), please convert the file to PNG format before calling fromFile.

Image Comparison Methods

In the automation process, to verify the correctness of results, execution results can be judged by verifying the difference between the result image and the expected image.

Currently, two methods of image judgment are provided:

Image.imageEqual(): Compares image differences and returns a boolean value.
Image.imageCompare(): Compares image differences and returns a comparison result object. This method can generate a difference result image by subtracting specific pixels of the two images, marking the differences with colors. It is very suitable for displaying difference images in reports for manual secondary verification.

Since operating systems, resolutions, and color settings may affect image display, the same control may display differently under different environment settings. Therefore, when comparing screenshots, a comparison tolerance is generally set. Within the tolerance threshold, two images are considered identical. Tolerance categories are as follows:

Color Tolerance: For two corresponding pixels of two images, if their RGB colors are within a certain range according to a specific distance algorithm, they are considered the same.
Pixel Tolerance: For the two images as a whole, if the number of different pixels is within a certain range, the two images are considered exactly the same. Pixel tolerance can also be set by percentage, i.e., the percentage of different points in all pixels of the image.

Image.imageEqual(image1, image2, options?, compareInfo?): Promise<boolean>

imageEqual is used to compare whether two images are the same, returning a Promise of a boolean value. If it returns true, the two images are the same; false means there are differences. This API is suitable for quickly verifying whether two images are consistent. If you need to obtain a result image showing the difference locations, please use the imageCompare API.

JavaScript

imageEqual(image1: Buffer | string | Image,
  image2: Buffer | string | Image, 
  options?: CompareOptions,
  compareInfo?: ImageCompareInfo): Promise<boolean>;

image1, image2 can be a Buffer, base64 string, or an Image instance.
options is used to specify image comparison parameters. It has the following parameter settings:

JavaScript

interface CompareOptions {
  colorTolerance?: number,        //default to 0.1
  pixelNumberTolerance?: number,  //no default value
  pixelPercentTolerance?: number, //default 1, means 1% 
  ignoreExtraPart?: boolean       //default to false
  auto?: boolean                  //default to false
}

Where:

colorTolerance: Color tolerance, default is 0.1. Generally does not need modification.
pixelNumberTolerance: Pixel count tolerance, no default value.
pixelPercentTolerance: Pixel percentage tolerance, default is 1, i.e., 1%. For example, if two images each have 10,000 pixels and at most 150 different pixels are allowed, set it to 1.5.
ignoreExtraPart: Whether to ignore non-overlapping parts, default is false. Two comparison images can be of different sizes. When comparing, the top-left corners are aligned, and parts exceeding the overlap are usually considered different. If set to true, the excess parts will be ignored.
auto: If set to true, it automatically identifies the two images based on image features and scales them to a consistent ratio for comparison. Defaults to adapting to the proportion of the smaller image.

Note: The auto option is not suitable for comparing images containing text content. Due to differences in text rendering by the system, text content rendering effects under different scaling ratios are not simply proportional scaling. Therefore, in this case, the auto option may not provide accurate comparison results.

When either pixelNumberTolerance or pixelPercentTolerance exceeds the set value (or default value), the comparison will return false. If you only need to use one setting, you can set the other to a larger value. For example, if you only need pixelNumberTolerance and want to ignore pixelPercentTolerance, you can set pixelPercentTolerance to 100.

Besides the return value indicating whether they are the same, sometimes detailed information is needed, such as image dimensions, number and percentage of different pixels, etc. In this case, relevant information can be obtained by passing a compareInfo object.

The type definition of compareInfo is as follows:

JavaScript

interface ImageCompareInfo {
  image1: { width: number, height: number },
  image2: { width: number, height: number },
  diffPixels: number,        // Number of different pixels
  diffPercentage: number     // Percentage of different points
}

When used, if compareInfo is passed as an empty object, calling imageEqual will fill compareInfo with data in the above structure.

Usage Example:

JavaScript

Python

(async function() {
    try {
      let pngSource = await Image.fromFile(__dirname + '/../source.png');
      let pngTarget = await Image.fromFile(__dirname + '/../target.png');

      let compareInfo = {};
      let isEqual = await Image.imageEqual(pngSource, pngTarget, 
        {pixelNumberTolerance: 300, ignoreExtraPart: true, auto: true}, compareInfo);
      console.log('Is Equal?', isEqual);
      console.log('Auto-filled compareInfo data:', JSON.stringify(compareInfo, null, 2));
  } catch(err) {
      console.log(err)
  }
})()

import os

try:
  # Construct file path
  base_dir = os.path.dirname(__file__)
  png_source = Image.fromFile(os.path.join(base_dir, "../source.png"))
  png_target = Image.fromFile(os.path.join(base_dir, "../target.png"))

  # Pass comparison parameters
  options = {
      "pixelNumberTolerance": 300,
      "ignoreExtraPart": True,
      "auto": True
  }

  compare_info = {}
  is_equal = Image.imageEqual(png_source, png_target, options, compare_info)

  print("Is Equal?", is_equal)
  print("Auto-filled compareInfo data:")
  print(json.dumps(compare_info, indent=2, ensure_ascii=False))

except Exception as err:
    print("Error:", err)

This code example demonstrates how to pass two images for comparison while ignoring non-overlapping parts. If there are more than 300 different pixels, the images are judged as different, and detailed data of compareInfo is output.

Is Equal? false
Auto-filled compareInfo data: {
  "image1": {
    "width": 375,
    "height": 266
  },
  "image2": {
    "width": 402,
    "height": 224
  },
  "diffPixels": 3502,
  "diffPercentage": 3.3100814760203408
}

Image.imageCompare(image1, image2, options?)

imageCompare is used to compare two images and return a data object containing detailed information, especially a result image marking the difference parts of the two images. If you need to obtain a result image showing the difference locations, please use this API.

It has the following signature:

JavaScript

imageCompare(image1: Buffer | string | Image,
  image2: Buffer | string | Image, 
  options?: CompareOptions): Promise<ImageCompareResult>

image1, image2: Can be Buffer, base64 string, or an Image instance.
options: Optional parameter, its setting method is the same as the options setting of imageEqual. The ignoreExtraPart parameter will affect the generation of the difference image: if set to false, non-overlapping parts of the image will be marked in red; if true, no special processing is done for non-overlapping parts.

The return value in Node.js is a structure named ImageCompareResult:

JavaScript

interface ImageCompareResult {
  equal: boolean,
  info: ImageCompareInfo,
  diffImage: Image
}

In Python, the return value is a dictionary (dict), with a structure equivalent to the interface above:

Python

{
  "equal": bool,
  "info": { ... },        # ImageCompareInfo, same as compareInfo in imageEqual
  "diffImage": "<base64 string>"
}

Where:

equal Indicates whether the images are considered the same according to tolerance settings.
info The structure is the same as compareInfo of imageEqual, including pixel dimensions, how many pixels are different, percentage of different pixels, etc.
diffImage is the returned difference image object. Identical pixels are displayed in default white, and different points are displayed in red. The original pattern in the image will be displayed in the target image in a light color, facilitating the location of the difference parts.

In these two image comparison APIs, imageEqual is a wrapper for imageCompare to more intuitively return the judgment result of whether two images are equal. If you only want to know if they are equal, use imageEqual. If you want to know more detailed information or need a difference image in addition to whether they are equal, use imageCompare.

Usage Example

The following example demonstrates how to compare two images and generate a difference image:

JavaScript

Python

const { Image } = require('leanpro.visual');
const fs = require('fs');

(async function() {
  try {
    let pngSource = await Image.fromFile(__dirname + '/image1.png');
    let pngTarget = await Image.fromFile(__dirname + '/image2.png');

    let result = await Image.imageCompare(
      pngSource,
      pngTarget,
      { pixelNumberTolerance: 300, auto: true }
    );

    let diffImage = result.diffImage;
    console.log('resultMeta', JSON.stringify(result.info, null, 2));

    let imageData = await diffImage.getData();
    fs.writeFileSync(__dirname + '/diff.png', imageData);
  } catch(err) {
    console.log(err);
  }
})();

import os
import json
import base64
from leanproAuto import Image

try:
    base_dir = os.path.dirname(__file__)

    # Read two images
    png_source = Image.fromFile(os.path.join(base_dir, "image1.png"))
    png_target = Image.fromFile(os.path.join(base_dir, "image2.png"))

    # Call comparison method
    result = Image.imageCompare(
        png_source,
        png_target,
        {"pixelNumberTolerance": 300, "auto": True}
    )

    # Print comparison info
    print("resultMeta", json.dumps(result["info"], indent=2, ensure_ascii=False))

    # Get diff image base64 data and save as PNG file
    diff_b64 = result["diffImage"]
    diff_bytes = base64.b64decode(diff_b64)
    with open(os.path.join(base_dir, "diff.png"), "wb") as f:
        f.write(diff_bytes)

except Exception as err:
    print("Error:", err)

This code compares two images, prints difference information, and saves the difference image as a "diff.png" file.

Suppose we have the following two images:

image1 (375 * 266)	image2 (402 * 224)

Depending on the parameters, the following difference images can be generated:

ignoreExtraPart = true	ignoreExtraPart = false

The left side is the case where ignoreExtraPart = true, where excess parts are also marked as red; the right side is ignoreExtraPart = false, ignoring the excess parts.

Control and Model Image Comparison

A common scenario in Windows automation is comparing the runtime control screenshot with the saved object screenshot in the model. This can be achieved by calling the modelImage method of {{book.test_object}} to get the screenshot PNG image of {{book.test_object}}, returned as a base64 string. If there is no corresponding screenshot for the object in the model, it returns null.

The following example gets a button object from the object model, takes a screenshot of the control and gets the screenshot in the model comparison, prints the comparison information, and saves the difference image to the directory.

JavaScript

Python

const { Image } = require('leanpro.visual');
const { WinAuto } = require('leanpro.win');
const fs = require('fs');

// Load object model (Fill in tmodel file path according to actual situation)
const model = WinAuto.loadModel(__dirname + "\\test.tmodel");

(async function () {
  // Get button object from model
  let five = model.getButton("Five");

  // Control current screenshot
  let controlImage = await five.takeScreenshot();

  // Saved screenshot in model
  let modelImage = await five.modelImage();

  // Compare model image with control screenshot
  let result = await Image.imageCompare(modelImage, controlImage);

  // Save diff image
  fs.writeFileSync(__dirname + '/diff.png', await result.diffImage.getData());

  // Print difference statistics
  console.log(result.info)
})();

import os
import json
import base64
from leanproAuto import Image, WinAuto

try:
    base_dir = os.path.dirname(__file__)

    # Load object model (Fill in tmodel file path according to actual situation)
    model = WinAuto.loadModel(os.path.join(base_dir, "test.tmodel"))

    # Get button object from model
    five = model.getButton("Five")

    # Control current screenshot
    control_image = five.takeScreenshot()

    # Saved screenshot in model
    model_image = five.modelImage()

    # Compare model image with control screenshot
    result = Image.imageCompare(model_image, control_image)

    # Print difference statistics
    print("Diff Info:", json.dumps(result["info"], indent=2, ensure_ascii=False))

    # Save diff image
    diff_b64 = result["diffImage"]
    diff_bytes = base64.b64decode(diff_b64)
    diff_path = os.path.join(base_dir, "diff.png")
    with open(diff_path, "wb") as f:
        f.write(diff_bytes)
    print("Diff image saved to:", diff_path)

except Exception as err:
    print("Error:", err)

Image Operation APIs

Image Object

Importing

Type Definitions

Common Usage

Instance Attributes

width

height

Instance Methods

clip(rect: Rect)

resize(width: number, height: number)

filterColor(color: Colors | Colors[])

drawImage(image: Image, x: number, y: number)

getData(filePath: string)

save(filePath: string)

Static Methods

Image.fromData(bufferOrString: Buffer | string)

Image.fromFile(filePath: string)

Image Comparison Methods

Image.imageEqual(image1, image2, options?, compareInfo?): Promise<boolean>

Usage Example:

Image.imageCompare(image1, image2, options?)

Usage Example

Control and Model Image Comparison

`Image` Object