Controls' Virtualize

In automated testing of desktop applications, we usually rely on object recognition technology to locate and operate controls (such as buttons, menus, text boxes, etc.). However, some controls cannot be obtained through traditional object recognition methods because they are custom-drawn (such as non-standard controls, special icon buttons, dynamically generated elements). In this case, Controls' Virtualize technology can be used to solve the problem.

The core idea of control virtualization is:
Dynamically generate a virtual control for a recognized control (such as a toolbar as a whole) at runtime, which is completely identical to the actual control in size and position. In this way, even if the sub-elements within it cannot be directly recognized, we can still automate operations on its internal elements through methods provided by the virtual control (such as OCR text recognition, visual-based clicking).

Applicable Scenarios

Imagine the following scenario:
The toolbar of an application is custom-drawn and does not use standard controls to render buttons. The toolbar as a whole can be recognized by automation tools, but each button within it (small icon with text) cannot be recognized individually. At this time, if we want to click the "Open" button, we cannot directly get it through methods like model.getButton("Open").

In this case, we can achieve clicking the "Open" button through the following steps:

1. Identify the entire toolbar control: Assume the toolbar has been loaded into the script as a Pane or other recognizable object through the model.
2. Get virtual control: Call the getVirtual() method (without passing any parameters) on the toolbar to obtain a virtual control object covering the entire toolbar area.
3. Click based on text: Virtual controls usually provide clickVirtualText() or similar OCR-related methods. We can use this method to locate and click the area where the button is located using the text content on the button (such as "Open").

Usage Example

Here are example codes using JavaScript and Python. When the toolbar object name is "ToolBar", we can write:

await model.getPane("ToolBar").getVirtual().clickVirtualText('Open');

model.getPane("ToolBar").getVirtual().clickVirtualText('Open')

Through the above code, the test script will recognize the position of the text "Open" in the virtual control, and then simulate a click on that area, thereby successfully clicking the "Open" button in the actual toolbar.