Falcon Java Module
This content applies solely to Appian RPA, which must be purchased separately from the Appian base platform.

The Falcon module enables your robotic process to locate images on the screen. With this functionality, a robotic process can extract text from an image, search for elements on the screen, or take full or partial screenshots.

This page describes how to integrate Falcon methods into your robotic process using the Java module. See the Image Recognition Low-code Module page for guidance in an easier and more robust development experience.

Using the Falcon Java module

To use the Falcon Java module, you must first include the module dependency in the pom.xml.

The core interface of the Falcon module is IFalcon. This interface provides methods to take full or partial screenshots and search for images.

This code snippet calls the interface:

1
2
IFalcon falcon =
    IFalcon.getInstance(this, windows);

Where windows is an instance of IWindows and this is a reference to the robot that uses the module.

Once you have called the interface, you can utilize the following methods to take advantage of computer vision capabilities.

Taking screenshots

The Falcon module allows the robotic process to take full or partial screenshots using the following methods:

Method Description
File copyScreenToTemporalFile() Take a screenshot of the entire screen and save it in a temporary file.
File copyPartialScreenToTemporalFile(Rectangle crop) Take a screenshot from a cropped area and save it in a temporary file.
void copyScreenToFile(File file) Take a screenshot of the entire screen and save it in the file received as a parameter.
void copyPartialScreenToFile(File file, Rectangle crop) Take a screenshot from a cropped area and save it in the file received as a parameter.

Searching images

The Falcon module allows you to search for images on the screen or images located within other images.

Before you can search for images, you must first add the image you want the robot to search for to your support files. The image must be taken from the resource where the robot will run to avoid differences in screen settings.

To search for images, you'll need to do three things:

  1. Capture support images
  2. Initialize the images as IFalconImage instances
  3. Utilize the searchInScreen methods

Capturing support images

The first step to searching for an image with the Falcon module is to capture the image. You can use the Capture support image tool to automate sending this image to the robotic process as a support file.

Initializing images

Images must be of type IFalconImage to work directly with the Falcon module. IFalconImage is the image abstraction used by the Falcon module. We recommend using the Appian RPA maven plugin, jidoka-maven-plugin, to convert support files to IFalconImage.

The plugin is integrated in all robotic processes and generates a support class to automatically obtain supporting code for images. The generated class is created with the package and the name defined in the properties.

This class is mainly used to initialize all images as IFalconImage. The generated class will assign default values to options related to the image (FalconImageOptions):

  • description: The file name.
  • file: The object file with the file.
  • colorTolerance: .05f.

Although you can change the implementation of this class, it is not recommended since you may need to generate the class again after adding more images to the support files. Instead, if you want to modify the class, create a new class that inherits from the generated class.

To use the plugin, specify the following properties in the robot's pom.xml:

1
2
3
4
5
6
7
<properties>
    <jidoka.serverId>server-id</jidoka.serverId>
    <jidoka.consoleURL>consoleURL</jidoka.consoleURL>
    <jidoka.robotName>robot-id</jidoka.robotName>
    <jidoka.imageSupportClassFullName>com.novayre.jidoka.robot.JidokaImageSupport</jidoka.imageSupportClassFullName>
    <jidoka.createImageSupportClass>true</jidoka.createImageSupportClass>
</properties>
  • jidoka.serverId: The server's identifier located in settings.xml with the credentials of the console that contains the images. Be sure the settings.xml configuration file includes the API key in the password field.

    The credentials are the Appian service account username and API key created in the Appian Admin Console. Note that this isn't the same API key used to authenticate with Maven.

    1
    2
    3
    4
    
    <server>
    <id>server-id</id>
    <password>APY_KEY_VALUE</password>
    </server>
    
  • jidoka.consoleURL: The URL of the Appian RPA Console that contains the images. For example https://mycompany.appiancloud.com/rpa/
  • jidoka.robotName: The technical identifier of the robotic process for which the support class is going to be generated. The identifier is located on the Robotic process configuration page, under the General Information tab.
  • jidoka.createImageSupportClass: This must be set to true to generate the support class.

Alternatively, if you do not want to use the plugin, you can use the following methods to initialize your images as IFalconImage:

Method Description
IFalconImage getImage(File file) Returns a falcon image from a file.
IFalconImage getImage(File file, float tolerance) Returns a falcon image from a file depending on the specified tolerance.
IFalconImage getImage(InputStream inputStream) Returns a falcon image from an input stream.
IFalconImage getImage(InputStream inputStream, float tolerance) Returns a falcon image from an input stream depending on the specified tolerance.
IFalconImage getImage(URL url) Returns a falcon image from a URL.
IFalconImage getImage(URL url, float tolerance) Returns a falcon image from a URL depending on the specified tolerance.
IFalconImage getImage(ImageInputStream imageInputStream) Returns a falcon image from the interface ImageInputStream, located in the package javax.imageio.stream.
IFalconImage getImage(ImageInputStream imageInputStream, float tolerance) Returns a falcon image from the interface ImageInputStream, located in the package javax.imageio.stream, depending on the specified tolerance.

Tolerance is the precision with which the original image is identified inside another image. Tolerance values are type float and range from 0 to 1. A low tolerance may cause Falcon to be unable to identify the image because it is looking for an image with minimal differences from the searched image. Alternatively, high tolerance may cause Falcon to identify the wrong image since it will accept images with more differences than the searched image. The recommended starting value is 0.05 and can be adjusted as necessary.

To access the support file that contains your falcon image, use the getCurrentDir() method. For example:

1
Paths.get(server.getCurrentDir(), "test.png")

Locating images

Once you have saved the image to search for in your support files and initialized it as IFalconImage, your robotic process can then locate the image.

When looking for an image, it's recommended to use the interface IWaitFor to determine how long the robotic process should look for an image. You can indicate if an exception should be thrown if the image is not found during the allocated time.

For example, the code below tells the robotic process to search for the image defined in the image parameter for 20 seconds.

1
windows.waitFor(this).image(20, image));

For more information about the interface IWaitFor, see the Javadocs.

You can use any of the following methods to search for the image:

Methods Description
boolean searchInScreen(IFalconImage falconImage) Locates an image on the screen. The method returns true if the image is found and updates the parameter falconImage with information about its position.
boolean searchInScreen(IFalconImage falconImage, boolean searchAll) When searchAll is true, this method locates all possible instances of an image on the screen.
boolean searchInScreen(IFalconImage falconImage, Rectangle crop) Searches an image using a specified rectangle on the screen. The method returns true if the image is found and updates the parameter falconImage with information about its position.
boolean searchInScreen(IFalconImage falconImage, Rectangle crop, boolean searchAll) When searchAll is true, this method locates all possible instances of an image within the specified rectangle.

FAQ

Why can't Falcon find my image in the robotic process working folder?

For Falcon to find the images, you must include them in the Support Files area of the robotic process configuration.

To access images added as support files and assign them to a File variable, use the code:

1
File supportFile = Paths.get(server.getCurrentDir(), "folder", "fileName").toFile();'

Where:

  • server.getCurrentDir(): The working directory of the robotic process, which is where the contents of the root folder are downloaded.
  • "folder": Must be an existing folder within the support files.
  • "fileName": The name of the file.

Why isn't the robotic process clicking on the image?

The problem may be that your scale setting is above 100%. On a Windows machine, the Scale and layout setting must be set to 100%.

To check your scale setting percentage, navigate to the Windows Display menu.

rpa-faq-3.png

Tutorial: Build a Low-Code Bot with Image Recognition

Learn how to use the Image Recognition module in a robotic process

Open in Github Built: Wed, Aug 17, 2022 (01:05:05 PM)

On This Page

FEEDBACK