Screen Automation

Screen object is used to obtain screen properties and perform screen operations. It helps retrieve screen bounds, capture screenshots, get the color of a specific point, and highlight regions on the screen.

You can import the Screen object as follows:

const { Screen } = require('leanpro.common');

from leanproAuto import Screen

Screen provides the following methods:

Type Definitions

export class Screen {
    static screenRect(moniter?: number): Rect
    static all(moniter?: number): Rect[]
    static colorAt(x: number, y: number): string;
    static capture(rect?: Rect): Buffer;
    static captureToFile(filePath: string, rect?: Rect): void;
    static takeScreenshot(filePath: string, monitor?: number): string | void;
    static highlight(rect: Rect, duration?: number);
    // static screenSize(): {width: number, height: number}; // deprecated
}

class Screen():
    def all() -> "List[Rect]"
    def screenRect(monitor: Optional[int]=None) -> "Rect"
    def colorAt(x: int, y: int) -> str
    def capture(rect: Optional[Rect]=None) -> "bytearray"
    def captureToFile(filePath: str, rect: Optional[Rect]=None) -> None
    def highlight(rect: Rect, duration: Optional[int]=None) -> "any"
    # def screenSize() -> TypedDict # deprecated 弃用

Rect Class

Rect class describes the shape and position of a control. In CukeTest, all controls can be represented by a rectangle (Bounding Rectangle). Rect object has the following attributes:

• x: number type, horizontal pixel coordinate relative to parent.
• y: number type, vertical pixel coordinate relative to parent.
• width: number type, width of the rectangle in pixels.
• height: number type, height of the rectangle in pixels.

interface Rect {
    x: number;
    y: number;
    width: number;
    height: number;
}

class Rect:
    x: int
    y: int
    width: int
    height: int

API Introduction

screenRect(monitor)

Get the bounds of a specified monitor or all screens.

Parameters:

• monitor: (Optional) Monitor index to retrieve bounds for. If not provided or -1, returns the combined size of all screens. 0 for the first monitor, 1 for the second, etc.

Return Value:

• Rect object, representing the bounds of the specified monitor.

Example Usage:

const { Screen } = require('leanpro.common');
const mainScreenRect = Screen.screenRect();
console.log('Main Screen Rect:', mainScreenRect);

const monitor1Rect = Screen.screenRect(1);
console.log('Monitor 1 Rect:', monitor1Rect);

from leanproAuto import Screen
main_screen_rect = Screen.screenRect()
print('Main Screen Rect:', main_screen_rect)

monitor1_rect = Screen.screenRect(1)
print('Monitor 1 Rect:', monitor1_rect)

all()

Get the bounds of all screens. Useful in multi-screen environments.

Return Value:

• Array of Rect objects, each representing a screen's bounds.

Example Usage:

const { Screen } = require('leanpro.common');
const screens = Screen.all();
screens.forEach((screen, i) => {
    console.log(`Screen ${i}:`, screen);
});

from leanproAuto import Screen
screens = Screen.all()
for i, screen in enumerate(screens):
    print(f"Screen {i}: {screen}")

screenSize()

Not recommended, use screenRect instead.

Gets the current screen size (width and height). No parameters required.

Return Value:

• {width: number, height: number} in pixels.

colorAt(x, y)

Get the color of a specific point on the screen in hexadecimal RGB format, e.g., FFFFFF.

Parameters:

• x: number, horizontal coordinate from the left edge of the screen.
• y: number, vertical coordinate from the top edge of the screen.

Return Value:

• string, hexadecimal RGB color code.

Example Usage:

const { Screen } = require('leanpro.common');
const x = 500;
const y = 300;
const color = Screen.colorAt(x, y);
console.log(`Color at (${x}, ${y}): ${color}`);

from leanproAuto import Screen
x = 500
y = 300
color = Screen.colorAt(x, y)
print(f"Color at ({x}, {y}): {color}")

capture(rect)

Capture a screenshot of a specific rectangular region. If no rectangle is specified, captures the entire screen.

Parameters:

• rect: (Optional) Rect object specifying the region to capture. Defaults to full screen if not provided.

Return Value:

• Buffer containing the captured image data. Can be saved to a file or used to create an Image object via Image.from() for reports or other purposes.

Usage Examples

1. Capture the entire screen and attach to report

const { Screen } = require('leanpro.common');
const image_buffer = Screen.capture();
this.attach(image_buffer, 'image/png');

from leanproAuto import Screen
import base64
image_buffer = Screen.capture()
# Convert buffer to base64 string
image_string = base64.b64encode(image_buffer).decode('utf-8')
request.attach(image_string, "image/png")

In this example, Screen.capture() without the rect parameter captures the entire screen. The image data is returned as a Buffer object, which is then attached to the report via attach().

2. Capture a specific region and save as PNG

const { Screen } = require('leanpro.common');
const fs = require('fs');
const rect = { "x": 555, "y": 358, "width": 126, "height": 34 };
const buffer = Screen.capture(rect);

fs.writeFileSync('screenshot.png', buffer);

from leanproAuto import Screen
rect = { "x": 555, "y": 358, "width": 126, "height": 34 }
buffer = Screen.capture(rect)

with open('screenshot.png', 'wb') as f:
    f.write(buffer)

Here we define the rectangular region by its coordinates (x, y) and size (width, height), then capture the region using Screen.capture(rect) and save the result as a PNG file.

captureToFile(filePath, rect)

Capture a screenshot and save it to a specified file path. You can capture the entire screen or a specific region.

Parameters:

• filePath: string, file path including name and extension. Supported formats: .jpg, .png, .bmp, etc.
• rect: (Optional) Rect object. If not provided, the entire screen is captured.

Return Value:

• None.

Example: capture entire screen

const { Screen } = require('leanpro.common');
Screen.captureToFile("C:/Users/Username/Desktop/screenshot.png");

from leanproAuto import Screen
Screen.captureToFile("C:/Users/Username/Desktop/screenshot.png")

Example: capture specific region

const { Screen } = require('leanpro.common');
Screen.captureToFile("C:/Users/Username/Desktop/screenshot.png", { "x": 555, "y": 358, "width": 126, "height": 34 });

from leanproAuto import Screen
Screen.captureToFile("C:/Users/Username/Desktop/screenshot.png", { "x": 555, "y": 358, "width": 126, "height": 34 })

The rectangle's top-left corner is at (555, 358), width is 126 pixels, height is 34 pixels.

highlight(rect, duration)

Highlight a specific rectangular area on the screen. Useful for marking and tracking regions during testing, visually showing the area being tested or analyzed.

Parameters:

• rect: Rect object, defines the area to highlight.
• duration: (Optional) number, duration in milliseconds. If not provided, the highlight lasts until the next highlight call or until the test ends.

Return Value:

• None.

Usage Examples

Highlight a specific screen region

const { Screen } = require('leanpro.common');
Screen.highlight({ "x": 555, "y": 358, "width": 126, "height": 34 });

from leanproAuto import Screen
Screen.highlight({ "x": 555, "y": 358, "width": 126, "height": 34 })

In this example, the rectangle's top-left corner is (555, 358), width is 126 pixels, height is 34 pixels.

takeScreenshot(filePath, monitor)

Capture a screenshot and save it to a specified file path. This method consolidates the previous Util.takeScreenshot functionality.

Parameters:

• filePath: string, file path including name and extension. Supported formats: .jpg, .png, .bmp, etc.
• monitor: (Optional) number, monitor index to capture. -1 or omitted captures all screens combined. 0 is the first display, 1 is the second, etc.

Return Value:

• string or void, the path to the saved screenshot file or nothing.

Usage Example

The following example shows how to use Screen.takeScreenshot to capture a screenshot and save it to a specified file path:

const { Screen } = require('leanpro.common');
Screen.takeScreenshot("C:/Users/Username/Desktop/screenshot.png");