Image Recognition Module

The Image Recognition module enables your robotic process to locate images on the screen. With this functionality, a robotic process can extract text from an image, search for elements on the screen, or take full or partial screenshots.

This page describes how to integrate image recognition methods into your robotic process using Appian RPA's Image Recognition module, which provides an easier and more robust development experience.

For more experienced developers, more options are available using the Java module.

Using the Image Recognition module

The Image Recognition module contains one method to enable the robotic process to locate an image on the screen. After the robotic process has located the image, it can click the image based on your configurations.

Locate Image

This method locates an image on the screen with multiple search options, such as location and match tolerance, and may click it once found.

You can configure CONFIGURATION, OUTPUT, and AFTER COMPLETION parameters for this method.

Configuration

In the CONFIGURATION area of the method pane, you can configure the following parameters:

  • Image(s) to search: Browse for the file containing the image you want to locate on the screen. Images need to be .png file format. You can add more than one image, if you'd like. If you include more than one image, the robotic process searches for any of them. If one is located, the action is considered successful. You'll need to add the image as a support file before you can browse for it in this field. You can also use the Create support image tool via the agent on the resource to send the support file directly to the robotic process.
  • Search:
    • Full screen: When selected, the module looks for the image on the entire screen.
    • Screen region: When selected, enter the points on the screen to search. Together, the four points form an area on the screen to look for the image. Enter values for Top left X position, Top left Y position, Bottom right X position, and Bottom right Y position to define the search area.
    • From Y position: When selected, enter a Y-coordinate to use as a search reference point. The robotic process looks for the image below that point on the screen. Y-coordinate values start at 0 at the top of the screen and get larger as they move down. For example, set this value to 500 if you want to search for images on the bottom half of the screen.
  • Wait time: Select whether you want the robotic process to wait when searching for an image:
    • Don't Wait: If the image isn't found immediately, the action completes without waiting.
    • Enter max wait time: The robotic process repeatedly searches for the image until it is found or the wait time is met. If the wait time is reached while the robotic process is looking for the image, it won't time out until it completes the current search. When you select this option, the Max wait time(seconds) option appears. Enter the maximum time (in seconds) the robotic process should wait. This number must be greater than zero.
  • Pixel tolerance: Choose a value for an acceptable amount of pixelation in the image the robotic process finds vs. the image you are looking for. Pixelation degrades image quality by making an image appear grainy or less clear. Available options include:
    • Exact match
    • Very low (1%)
    • Low (2%)
    • Medium (5%)
    • High (10%)
    • Very high (15%)
    • Custom: If you select Custom, specify the tolerance value using an integer between 1 and 100 or by writing an expression. If you are searching for a high quality image, but would allow the robotic process to find a lower quality version of it, enter a lower number. Be aware that higher tolerance values may impact performance.
  • Color tolerance: Choose a value for an acceptable amount of color change in the image the robotic process finds vs. the image you are looking for. Available options include:
    • Exact match
    • Very low (1%)
    • Low (2%)
    • Medium (5%)
    • High (10%)
    • Very high (15%)
    • Custom: If you select Custom, specify the tolerance value using an integer between 1 and 100 or by writing an expression. For example, if you search for a color image but would allow the robotic process to find a black and white version of it, enter 100% here. Be aware that higher tolerance values may impact performance.
  • Mouse actions: Select Enable click actions to configure how the robotic process should interact with the image once found.
    • Button: Choose whether the mouse should use the Left or Right button to click.
    • Click type: Choose Single or Double.
    • Click position: Choose where to click on the image: Top left, Top center, Top right, Center left, Center, Center right, Bottom left, Bottom center, or Bottom right.
    • Offset: Select this checkbox to offset the click from the position selected. Use these configurations to choose the direction and distance:
      • Number of pixels: Choose the distance (in pixels) to offset the click, relative to the selected position.
      • Direction: Choose the direction to offset the click.
  • While locating imageā€¦: Select Throw exception if not found to throw an exception if the robotic process doesn't locate any of the images in the time allowed.

Output

The Locate Image method saves the returned information in a custom data type called LocateImageResult. This data type contains the following properties:

Property Type Description
imageName String The name of the images the method searched for and found. If no images were found, this property is null.
found Boolean Returns true if the method located at least one image that was provided in the configuration.
pointsWhereFound Point X and Y coordinates where the images were found on the screen.
colorTolerance Decimal Indicates the tolerated color variation from the original image.
pixelTolerance Decimal Indicates the tolerated pixel variation from the original image.

In the OUTPUT area of the method pane, configure the following parameters to store this result:

  • Operator: Select an option to store the result as a single-value variable (is stored as) or add the result as a multiple-value variable (is appended to).
  • Target: Enter the name of the robotic process variable where you want to store or append the result. If your process variable is type CDT, you can specify which field of a CDT to store the result into using dot notation.

To access this information at later points in the robotic process, create a robotic process variable that uses LocateImageResult as the type. You can then use dot notation with the variable to access the values. For example, if you create a variable called result using the LocateImageResult type, you can use pv!result.found in the expression editor of a subsequent action to return if the image was found or not.

After completion

In the AFTER COMPLETION area of the method pane, configure the following parameters:

  • Wait before executing next action: Enter the number of seconds the robotic process waits before it continues to the next action in the workflow.
  • Take a screenshot: If you select this option, the robotic process takes a screenshot of the resource immediately after the action completes successfully.

Using Locate Image in an action

This section describes the steps to add the Locate Image method to an action in your robotic process workflow. When the robotic process reaches this point in the workflow, it searches for the image you specify within the defined area. If it finds the image, the robotic process clicks the image. You can add more than one image for the robotic process to find. If the robotic process can't find the first image, it searches for the second using the same configuration settings.

To use the Locate Image method:

  1. In the Appian RPA console, got to the Robotic Processes tab.
  2. Open a robotic process configuration.
  3. Upload support files for the images you want to search for:
    • Navigate to the Support files section.
    • Hover over the top-most folder in the structure (usually Root) and click the Upload icon rpa-upload-icon.png. The Choose file window displays.
    • Click Choose file.
    • Browse to and select an image you want the robotic process to search for. Images need to be in .png format.
    • Click OK. The selected file appears in the Support files tree.
  4. Navigate to an action in the workflow.
  5. Click the action and click the list icon 951644.png.
  6. In the Module tree, open the Image Recognition module and click Locate Image. The Locate Image pane displays.
  7. In the CONFIGURATION area of the pane, configure the following parameters:
    • In the Image(s) to search field, browse to and select the images you added as support files.
    • In the Search field, choose to search the Full screen, a Screen region, or From Y position.
    • In the Wait time field, specify whether the robotic process waits when looking for the image and, if so, how long in seconds the robotic process waits.
    • If searching an area smaller than the full screen, configure additional options so the robotic process knows where to look. These options include Pixel tolerance and Color tolerance.
    • Click Enable click actions if you want the robotic process to click on the image after it's found. Configure additional parameters as desired.
    • Click Throw exception if not found to configure the robotic process to throw an exception if the image isn't found.
  8. In the OUTPUT area of the method pane, configure the parameters described in Output parameters. This method returns LocateImageResult data.
  9. In the AFTER COMPLETION area of the method pane, configure the parameters described in After completion parameters.
  10. Click OK to save your changes to the action.
  11. Click SAVE CHANGES in the page toolbar.

rpa-image-low-code-module.png

Output parameters

For each method in this module, you can configure the following parameters in the OUTPUT area of the method pane:

  • Operator: Select an option to store the result as a single-value variable (is stored as) or add the result as a multiple-value variable (is appended to).
  • Target: Enter the name of the robotic process variable where you want to store or append the result. If your process variable is type CDT, you can specify which field of a CDT to store the result into using dot notation.

After completion parameters

For each method in this module, you can also configure the following parameters in the AFTER COMPLETION area of the method pane:

  • Wait before executing next action: Enter the number of seconds the robotic process waits before it continues to the next action in the workflow.
  • Take a screenshot: If you select this option, the robotic process takes a screenshot of the resource immediately after the action completes successfully.

Improve performance

The Image Recognition module searches for images based on configurations you set. Some of these configurations can impact the robotic process's overall performance and execution time. For example, the larger the search area, the longer it takes for the action to complete because the robotic process has to look for the image in multiple places. If you find that the Locate Image method is taking longer than you expect, consider the following tips:

  • Search for one image at a time. By limiting the amount of potential matches, the robotic process can limit the amount of time it spends searching.
  • Search for the image in a smaller area. If you're looking for an icon in a toolbar, use the Screen Region configuration to specify the exact area where that toolbar appears.
  • If searching the whole screen, use a low tolerance for an image match. This will ensure that the robotic process doesn't return too many potential matches.
  • Test different tolerance values. The higher the color or pixel tolerance for an image match, the longer the image takes to find, which ultimately means performance will be slower. Higher tolerance values can also lead to false positives, so it's important to test different options to determine the tolerance value you're willing to accept in terms of the impact on matches and execution times.

Create support image tool

You can use the Appian RPA agent to capture support images directly on your resource. Once you take the screenshot, you can send it to a robotic process in the console or to the global support files.

To create a support image on your resource:

  1. Right-click the agent icon and click Create support image.

    falcon-1.png

  2. Click Take new screenshot or click and drag the mouse on an area of the screen to take a partial screenshot.
  3. In the side panel, under Save image, select the dropdown and choose to save your image in the global support files or to a robotic process.

    Falcon saveImage

  4. Enter a name for the image.
  5. Click Save image. The image will be sent to the console and available as a robot support file or as a global support file.

    Falcon supportfiles


This version of the Appian RPA documentation was written for Appian 21.4, and does not represent the interfaces or functionality of other Appian versions.
Open in Github Built: Tue, Nov 23, 2021 (07:31:45 PM)

On This Page

FEEDBACK