Graphics Basic

Image Automation is used as a general-purpose interface automation solution that operates by matching patterns or specifying regions. It can be used on various platforms, such as Windows or Linux. CukeTest provides pattern objects Pattern and region objects (virtual controls) to accomplish image automation. It can be used not only with models in interface automation related modules (leanpro.win, leanpro.qt, leanpro.java, leanpro.common, etc.), but also directly in scripts without introducing models.

Concept Introduction

Pattern

Pattern is a picture - when recognizing the pattern object in the model manager, the pattern object is created by framing it in the current screen and taking a screenshot; while in the script you can create the pattern object directly by specifying the picture string (base64 encoding), the path to the picture file. When the pattern object is used, it looks for the area in the current screen (at runtime) that matches the pattern object, and then performs the specified operation (such as clicking, dragging, hovering, etc.). Since the search process uses the image matching technology, it takes some time for calculation before the operation, which is usually 0.5 ~ 1 second on mainstream computers (1920*1080 resolution case).

Identification properties of pattern objects

Since the pattern object needs to match the operation area in real time in the screen when it is actually used, in order to make the matching operation more stable, flexible and controllable, CukeTest provides several properties for the pattern object to allow users to better operate the matching, these properties can be modified directly in the Model Manager through the Modify Parameters panel, or can be used directly in the script.

1. similarity: Similarity threshold. For each match, the similarity between the match result and the target pattern is calculated, and this similarity can be obtained by calling the score() method on the Pattern object, while the similarity threshold is used to ignore matches with similarity lower than the threshold to get the most accurate match result, the default threshold is 0.98.

   In the actual automation process is usually to reduce the threshold to ensure more flexible pattern matching, because when the pattern is matched to more than one result, the operation on the pattern (such as click, hover, etc.) will only take effect on the one with the highest similarity score, so the principle of adjusting the threshold should be greedy for more, not less.
   If a higher similarity threshold than 0.98 is needed to match the target result, it may be that the percentage of valid content in the pattern is too small, for example, if the percentage of text in a button is relatively small, then the recognition process may be confused with other buttons. This time should be to reduce the background when intercepting the pattern, so as to increase the proportion of valid content.

2. resizeFactor: The scaling factor. If the size of the pattern object does not match the size in the actual scene, for example, the pattern shrinks or enlarges because of the responsive layout, then the scaling factor can be adjusted to match. If you add a 64px wide square pattern object, but the size of the matching target becomes a 128px wide square, you can set a scaling factor of 2 to do the matching. The default factor is 1, which means no scaling.