Image Recognition Module

The Image Recognition module enables your robotic process to locate images on the screen. With this functionality, a robotic process can extract text from an image, search for elements on the screen, or take full or partial screenshots.

This page describes how to integrate image recognition methods into your robotic process using Appian RPA's low-code Image Recognition module, which provides an easier and more robust development experience.

For more experienced developers, more options are available using the Java module.

Using the Image Recognition low-code module

The Image Recognition module contains one method to enable the robotic process to locate an image on the screen. After the robotic process has located the image, it can click the image based on your configurations.

Locate Image

This method locates an image on the screen with multiple search options, such as location and match tolerance, and may click it once found. Configuration options include:

  • Image(s) to search: Browse for the file containing the image you want to locate on the screen. Images need to be .png file format. You can add more than one image, if you'd like. If you include more than one image, the robotic process searches for any of them. If one is located, the action is considered successful. You'll need to add the image as a support file before you can browse for it in this field. You can also use the Create support image tool via the agent on the resource to send the support file directly to the robotic process.
  • Search:
    • Full screen: When selected, the module looks for the image on the entire screen.
    • Screen region: When selected, enter the points on the screen to search. Together, the four points form an area on the screen to look for the image. Enter values for Top left X position, Top left Y position, Bottom right X position, and Bottom right Y position to define the search area.
    • From Y position: When selected, enter a Y-coordinate to use as a search reference point. The robotic process looks for the image below that point on the screen. Y-coordinate values start at 0 at the top of the screen and get larger as they move down. For example, set this value to 500 if you want to search for images on the bottom half of the screen.
  • Wait time: Enter the amount of time (in seconds) the robotic process should try searching for the image before considering it as not found. The robotic process will repeatedly search for the image until it is found or the wait time is met. If the wait time is reached while the robotic process is looking for the image, it won't time out until it completes the current search.
  • Pixel tolerance: Choose a value for an acceptable amount of pixelation in the image the robotic process finds vs. the image you are looking for. Pixelation degrades image quality by making an image appear grainy or less clear. Available options include:
    • Exact match
    • Very low (1%)
    • Low (2%)
    • Medium (5%)
    • High (10%)
    • Very high (15%)
    • Custom: If you select Custom, specify the tolerance value using an integer between 1 and 100 or by writing an expression. If you are searching for a high quality image, but would allow the robotic process to find a lower quality version of it, enter a lower number. Be aware that higher tolerance values may impact performance.
  • Color tolerance: Choose a value for an acceptable amount of color change in the image the robotic process finds vs. the image you are looking for. Available options include:
    • Exact match
    • Very low (1%)
    • Low (2%)
    • Medium (5%)
    • High (10%)
    • Very high (15%)
    • Custom: If you select Custom, specify the tolerance value using an integer between 1 and 100 or by writing an expression. For example, if you search for a color image but would allow the robotic process to find a black and white version of it, enter 100% here. Be aware that higher tolerance values may impact performance.
  • Mouse actions: Select Enable click actions to configure how the robotic process should interact with the image once found.
    • Button: Choose whether the mouse should use the Left or Right button to click.
    • Click type: Choose Single or Double.
    • Click position: Choose where to click on the image: Top left, Top center, Top right, Center left, Center, Center right, Bottom left, Bottom center, or Bottom right.
    • Offset: Select this checkbox to offset the click from the position selected. Use these configurations to choose the direction and distance:
      • Number of pixels: Choose the distance (in pixels) to offset the click, relative to the selected position.
      • Direction: Choose the direction to offset the click.
  • While locating imageā€¦: Select Throw exception if not found to throw an exception if the robotic process doesn't locate any of the images in the time allowed.
  • Operator: Choose whether to store the result or append the result to a robotic process variable.
  • Save Into: Choose the robotic process variable to store this information. The variable should be of type LocateImageResult, described below.

The Locate Image method saves the returned information in a custom data type called LocateImageResult. This data type contains the following properties:

Property Type Description
imageName String The name of the images the method searched for and found. If no images were found, this property is null.
found Boolean Returns true if the method located at least one image that was provided in the configuration.
pointsWhereFound Point X and Y coordinates where the images were found on the screen.
colorTolerance Decimal Indicates the tolerated color variation from the original image.
pixelTolerance Decimal Indicates the tolerated pixel variation from the original image.

To access this information at later points in the robotic process, create a robotic process variable that uses LocateImageResult as the type. You can then use dot notation with the variable to access the values. For example, if you create a variable called result using the LocateImageResult type, you can use pv!result.found in the expression editor of a subsequent action to return if the image was found or not.

The following section describes how to add the method to an action in the workflow.

Using Locate Image in an action

This section describes the steps to add the Locate Image method to an action in your robotic process workflow. When the robotic process reaches this point in the workflow, it searches for the image you specify within the defined area. If it finds the image, the robotic process clicks the image. You can add more than one image for the robotic process to find. If the robotic process can't find the first image, it searches for the second using the same configuration settings.

To use the Locate Image method:

  1. Go to the Appian RPA Console and open the Robotic Processes tab.
  2. Open a robotic process configuration and navigate to the Support files section.
  3. Hover over the top-most folder in the structure and click the Upload icon.
  4. Browse for the images you want the robotic process to search for and click OK. Images need to be in .png format.
  5. Back on the Robotic process configuration page, navigate to an action in the workflow.
  6. Click the action and click the list icon 951644.png.
  7. From the Module tree, open the Image Recognition module and click Locate Image.
  8. In the Image(s) to search field, browse for the images you added as support files.
  9. Choose to search the Full screen, a Screen region, or From Y position.
  10. If searching an area smaller than the full screen, configure additional options so the robotic process knows where to look.
  11. Choose how long the robotic process should look for the image in the Wait Time field.
  12. Click Enable click actions if you want the robotic process to click on the image after it's found. Configure the options how you prefer.
  13. You can also choose to throw an exception if the image isn't found.
  14. Use the Operator dropdown to store the result or append the result of the expression to a robotic process variable.
  15. In the Save Into field, enter the name of a robotic process variable.
  16. Click OK to save.

rpa-image-low-code-module.png

Improve performance

The Image Recognition module searches for images based on configurations you set. Some of these configurations can impact the robotic process's overall performance and execution time. For example, the larger the search area, the longer it takes for the action to complete because the robotic process has to look for the image in multiple places. If you find that the Locate Image method is taking longer than you expect, consider the following tips:

  • Search for one image at a time. By limiting the amount of potential matches, the robotic process can limit the amount of time it spends searching.
  • Search for the image in a smaller area. If you're looking for an icon in a toolbar, use the Screen Region configuration to specify the exact area where that toolbar appears.
  • If searching the whole screen, use a low tolerance for an image match. This will ensure that the robotic process doesn't return too many potential matches.
  • Test different tolerance values. The higher the color or pixel tolerance for an image match, the longer the image takes to find, which ultimately means performance will be slower. Higher tolerance values can also lead to false positives, so it's important to test different options to determine the tolerance value you're willing to accept in terms of the impact on matches and execution times.

Create support image tool

You can use the Appian RPA agent to capture support images directly on your resource. Once you take the screenshot, you can send it to a robotic process in the console or to the global support files.

To create a support image on your resource:

  1. Right-click the agent icon and click Create support image.

falcon-1.png

  1. Click Take new screenshot or click and drag the mouse on an area of the screen to take a partial screenshot.
  2. In the side panel, under Save image, select the dropdown and choose to save your image in the global support files or to a robotic process.

Falcon saveImage

  1. Enter a name for the image.
  2. Click Save image. The image will be sent to the console and available as a robot support file or as a global support file.

Falcon supportfiles


This version of the Appian RPA documentation was written for Appian 20.4, and does not represent the interfaces or functionality of other Appian versions.
Open in Github

On This Page

FEEDBACK