Falcon Module

Falcon is a platform component that provides computer vision features. With this functionality, the robots can extract text from the screen, search for elements on screen, whose position is unknown, or take full or partial screenshots.

To use the Falcon module in our projects, we must include the following dependency in our file pom.xml:

1
2
3
4
5
<dependency>
    <groupId>com.novayre.jidoka.module</groupId>
    <artifactId>jidoka-falcon-api</artifactId>
    <version>${jidoka.version}</version>
</dependency>

where <version>${jidoka.version}</version> specifies the module current version.

1
2
3
<properties>
    <jidoka.version>X.Y.Z</jidoka.version>
</properties>

With this change, the dependency jidoka-falcon-api-X.Y.Z.jar will be loaded, and so we will have the Falcon module available.

IFalcon

The core interface of the Falcon module is com.novayre.jidoka.falcon.api.IFalcon.

This interface provides the necessary methods to search images within other images, determine the position of certain elements on screen through image recognition, or extract text from an image.

To obtain an instance of the interface you should use the method:

1
2
IFalcon falcon =
    IFalcon.getInstance(this, windows);

Where windows is an instance of IWindows and this is a reference to the robot that uses the module.

Using the Falcon module

The Falcon module allows you to take both full or partial screenshots. It provides the following methods:

  • File copyScreenToTemporalFile(): this method takes a screenshot and saves it in a temporary file.
  • File copyPartialScreenToTemporalFile(Rectangle crop): this method takes a screenshot from a rectangular cropped area.
  • void copyScreenToFile(File file): this method takes a screenshot and saves it in the file received as a parameter.
  • void copyPartialScreenToFile(File file, Rectangle crop): this method takes a screenshot from a rectangular cropped area and saves it in the file received as a parameter.

Searching images

We can search for images within other images, and even on screen, by using the following methods:

  • Point search(BufferedImage largerImage, BufferedImage smallerImage): this method locates the position of an image that is inside another image, if this is the case.
  • boolean searchInScreen(IFalconImage falconImage): this method locates an image on the screen. It will return true if the image is found, and it will update the parameter falconImage with useful information about its position.
  • boolean searchInScreen(IFalconImage falconImage, boolean searchAll): like the previous one, but this method will locate all the possible matchings with an image.
  • boolean searchInScreen(IFalconImage falconImage, Rectangle crop): it searches an image using the specified rectangle on the screen.
  • boolean searchInScreen(IFalconImage falconImage, Rectangle crop, boolean searchAll): like the previous one, but locating all the possible matchings.

The interface IWaitFor is especially appropriate when we are waiting for an image to appear on screen. This way, if we use its method image giving it an image IFalconImage as a parameter (actually, an object of type IImageResource), we will wait for a certain time until it is detected:

1
windows.waitFor(this).image(20, image));

Here, we are waiting 20 seconds to detect the image defined by the parameter image.

The attributes of type IFalconImage are the image abstraction used by the Falcon module. We can instantiate it in different ways, depending on the original image:

  • IFalconImage getImage(File file): this method uses a reference to a file.
  • IFalconImage getImage(File file, float tolerance): similar to the previous method, but including another parameter for the tolerance.
  • IFalconImage getImage(InputStream inputStream): in this method the instance is generated from an input stream.
  • IFalconImage getImage(InputStream inputStream, float tolerance): similar to the previous method, but including another parameter for the tolerance.
  • IFalconImage getImage(URL url): this one generates the instance from an URL.
  • IFalconImage getImage(URL url, float tolerance): similar to the previous method, but including another parameter for the tolerance.
  • IFalconImage getImage(ImageInputStream imageInputStream): this method generates the instance from the type ImageInputStream (package javax.imageio.stream).
  • IFalconImage getImage(ImageInputStream imageInputStream, float tolerance): similar to the previous method, but including another parameter for the tolerance.

The tolerance is the precision with which the original image is identified inside another image. A low value of tolerance may cause Falcon to be unable to identify the image, even though it appears clear and obvious to the human eye. On the other hand, high values of tolerance may cause Falcon to identify wrong images, giving false positives. The tolerance may be seen as the error margin that we give to Falcon.

The tolerance values will be of type float ranging from 0 to 1. A good starting value is usually 0.05, to be later adjusted if necessary. A low tolerance value means that less differences are accepted between the searched image and the found image, and a high value means the opposite.

It is important to note that, when working with the Falcon module, the images to be searched should be as much different from the rest of the screen as possible, so that ambiguity to find it is minimized. We can achieve this by making the reference pixels (usually the first ones at the upper left-hand corner) be different enough from the rest of them.

As an example, suppose we want to search an image within this guide, assuming that a page is the screen on which we are searching. The image to be found would be the following picture:

appian.png

We can see that in this image, the initial pixels (upper left-hand corner) are different from the rest. This fact reduces the number of candidate images to evaluate, thus saving processing time.

In addition, it is necessary that the screenshots that will be used to obtain the images to be found, are taken on the same resource on which the robot will run, since there may be different screen settings on different machines, especially the screen resolution. An easy way of taking the screenshots, which we recommend, is by using the resource remote control from the Appian RPA Console.

Support image creation

Appian RPA Agent includes a feature to create support images on the resource. From here, it is possible to take screenshots and send them directly to a robot in the console or to the global support files.

To use this feature, we have to do a mouse right click on the agent icon in the system tray and choose the option Create support image.

falcon-1.png

Once the option is activated, we can see that an application is started on the resource, making it look like the image below.

falcon-2.png

Then, we can take a screenshot by pressing button Take new screenshot or select with the mouse an area on the screen to transform it to an image, using the zoom if necessary. Whatever the option we choose, we must select from the drop-down combo box a robot or the option for global support files. By doing so, a text field will be enabled to fill in the name of the file to upload. Optionally, we can choose the target folder where the image taken will be saved or overwrite an existing image. In addition, we can use the keyboard to zoom or move the region to crop.

2236871.png

After pressing button Save image, the image will be sent to the console and it will be available as a robot support file or as a global support file.

Support class for images

With the Appian RPA Maven plugin, you can generate a support class to automatically obtain supporting code for images. We will obtain a method for each uploaded image as support files, which will return an object of type IFalconImage with which we can work directly.

The generated class will be called JidokaImageSupport and will be located in the package com.novayre.jidoka.robot.

The generated class will assign default values to options related to the image (FalconImageOptions). Such values will be as follows:

  • description: file name.
  • file: object File with the file.
  • colorTolerance: .05f.

Since the class is created as a project resource, we are completely free to change its implementation. However, it is possible that we need to generate the class again, after adding more images to the support files. For this reason, we recommend not to change the class directly, but create a new one which inherits from the generated class. In this new class, we can do the necessary changes for our particular case.

The main use is to initialize all the images as IFalconImage, without writing anything.

Generating JidokaImageSupport.java

The generation of the support class for uploaded images as support files uses the Appian RPA plugin for maven jidoka-maven-plugin. This plugin is integrated in all the robots, so its use only depends on the specified settings.

To make this plugin work properly, you should specify certain properties related to it in the robot's pom.xml. You can achieve this by running a package, for example.

The properties to be specified are as follows:

1
2
3
4
5
6
<properties>
    <jidoka.serverId>server-id</jidoka.serverId>
    <jidoka.consoleURL>consoleURL</jidoka.consoleURL>
    <jidoka.robotName>robot-id</jidoka.robotName>
    <jidoka.createImageSupportClass>true</jidoka.createImageSupportClass>
</properties>
  • jidoka.serverId: identifier of the server present in the Maven installation file settings.xml with the credentials of the console that contains the images. The credentials are the Appian service account username and API key created in the Appian Admin Console. Note that this isn't the same API key used to authenticate with Maven.
    1
    2
    3
    4
    5
    
    <server>
    <id>jidoka</id>
    <username>appian_service_account_username</username>
    <password>service_account_api_key</password>
    </server>
    
  • jidoka.consoleURL: URL of the console that contains the images.
  • jidoka.robotName: identifier name of the robot for which the support class is going to be generated.
  • jidoka.createImageSupportClass: the support class will be generated only if its value is true.

Falcon example robot

We will develop a robot to automate the data entry into a web application, for example. This time, instead of using the Browser module, we will introduce the Falcon module. We will see how to make our robot identify elements on the page and interact with them based on certain screenshots of those elements.

Since the images rendered by each browser can change, it is important to get the images using the same browser we are going to use. In this case, we are using Chrome browser to open the webpage.

Source Code

You can get the source code of the robot from the following link:

Description Link
Source Code robot-developmentguide-falcon.zip
Support Files Root.zip
Readme file to deploy Support Files readme.txt

Workflow

falcon-workflow.png

In this workflow, like in previous examples, we assume that there is at least one item to be processed whenever the robot runs, though in real robots we would start by checking if there are any items.

Implementation

We will analyze the methods init, openPage, login, goToAmountPage and processItem, highlighting what is related to the Falcon module.

This time, apart from performing the usual tasks in every example robot we have seen so far, such as obtaining the instances of the modules used by the robot or setting the default pause time, the method init will initialize the example screenshots, so that they can be used with the Falcon API. These screenshots will be sent to the robot's local working directory on the resource before running but, to do so, all the example files must have been properly set up on the settings section Support Files.

The elements should be extracted from screenshots taken on the same resource on which the robot will run, and using the default browser, because this will be the browser that our example robot will use to open the pages. We must bear in mind that the Falcon module works by identifying smaller images within other images to locate them. To ensure that the robot is able to identify the image, we must extract it just as it's seen. This rule will significantly reduce the number of false positives and will prevent Falcon from being unable to find it though the image appears clear to the human eye. A good way of taking these screenshots is through the remote-control button available for this purpose in the Appian RPA console.

Below you can find the screens of the first version of the web to be browsed by the robot.

falcon-login.png

AEB_home.png

The previous screenshot corresponds to a home page that allows you to navigate to the amounts screen, where the robot will process the items.

AEB_start.png

The screenshot above corresponds to an amounts entry screen. The robot will enter the data and send the form.

From these images (which have been taken on the same machine on which the robot will run), we have cropped certain elements that our robot will use through the Falcon module to perform tasks, position itself or make a delay.

Below you can find the extracted images of the elements and, below each one, the file name it will have in the images folder of the robot's local directory.

Image File name
2232293.png login-mark.png
2232295.png amount-button.png
2232296.png amount-page-mark.png
2232297.png name-field-placeholder.png

Try to capture the most representative pixels of the image and, if possible, they should be small images. With these two practices, the falcon's behavior is reinforced.

The method init will instantiate these images as objects of type IFalconImage, an abstraction of an image to enhance it with the power of the Falcon API.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
   /**
     * Action "Init".
     * 
     * @throws IOException
     *             if an I/O error occurs
     */
    public void init() throws IOException {
        
        // Get an instance of IJidokaServer to communicate with the server.
        server = (IJidokaServer< ? >) JidokaFactory.getServer();
        
        /*
         * Get an instance of IJidokaRobot to interact specifically with Windows
         * OS.
         */
        windows = IJidokaRobot.getInstance(this);

        // Get an instance of the Hawk-Eye module
        falcon = IFalcon.getInstance(this, windows);
        
        /*
         * Set the standard pause after typing or using the mouse. This is not
         * mandatory, but it is very usual.
         */
        windows.typingPause(PAUSE);
        windows.mousePause(PAUSE);
        
        // Initialize the IFalconImages objects to work with the images
        loginMarkImage = initImage(0.1f, 0.1f,"login-mark.png", "Login page");
        amountButtonImage = initImage(0.1f, 0.1f,"amount-button.png", "Amount Button Image");
        amountPageMarkImage = initImage(0.1f, 0.1f,"amount-page-mark.png", " Amount page");
        nameFieldPlaceholderImage = initImage(0.1f, 0.1f,"name-field-placeholder.png", "Name Field is available");
        
        /*
         * Set the number of items.
         * 
         * In a real robot, this number can be obtained by managing an
         * application, a sheet in an input Excel, etc.
         * 
         * It is very important to use this method as soon as we figure out this
         * information, because it helps the platform to calculate accurate
         * statistics and time measures.
         */
        server.setNumberOfItems(ITEMS.length);
        
        // Initialize the string with the name of the browser to use.
        browser = "chrome.exe";
    }

In the method above, we obtain the instances of IFalconImage through the Falcon method getImage, passing an object of type File as a parameter, which will match the image file for the element. An IFalconImage is created for every element with which the robot will interact. These files are locally accessible from the robot's local working directory.

The next action that our robot will do is opening the default browser.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
   /**
     * Action "Open Page".
     * 
     * @throws JidokaUnsatisfiedConditionException
     *             if some conditions are not met
     * @throws IOException 
     */
    public void openPage() throws JidokaUnsatisfiedConditionException, IOException {

        /*
         * Build the path based on the working directory.
         * 
         * For this robot, the application to launch is located inside the
         * working directory.
         */
        Path path = Paths.get(server.getCurrentDir(), appFolder, "index.html");
        
        // Use browser Chrome.
        // In this robot we are not using the Browser module
        windows.keyboard().windows("r").pause().type(browser).enter().pause();
        
        // Wait until the browser page is opened.
        windows.waitCondition(10, 1000,
                "Wait until the browser is opened",
                null, true,
                (i, c) -> ( windows.getWindow(".*(Nueva pesta├▒a - Google Chrome).*")!= null 
                            || windows.getWindow(".*(New Tab).*")!=null));
        
        // Go to the navigation bar.
        String loginPage = path.toString();
        windows.keyboard().alt("d").pause().type(loginPage).enter();
        
        // Wait until the login page is loaded.
        windows.waitFor(this).image(10, true, loginMarkImage);
    }

In the code above, we will open the login screen which will be the starting point for the robot. It will wait for the browser to open in a window whose title matches ".New Tab.". Additionally, to make sure that the page has loaded, it will wait until the login button appears on screen. The method wait is overloaded to allow you to wait until an image is found on the screen.

Remember that waitFor allows you to use waiting conditions pre-defined by the Appian RPA development team, without the need for writing the code of the condition.

Once the login page has been loaded, the robot will continue to the next action, which corresponds to the method login. Here, the keyboard will be used for typing the credentials and pressing Enter.

1
2
3
4
5
6
7
8
9
10
11
/**
 * Action "Login".
 * 
 * @throws Exception
 */
public void login() throws Exception {
    
    // Type the user and password and type 'Enter'.
    windows.keyboard().type("testuser").tab().pause().type("testpassword")
            .pause().enter();
}

We can see in the code above, that it is using the instance keyboard to type a dummy username, change the input focus by pressing the tab key, enter a dummy password, and press Return to access the application's home page.

In the next action, the robot will wait until the home page is loaded. After that, it will navigate to the amounts screen ("importes").

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
   /**
     * Action "Go to Amount Page".
     * 
     * @throws JidokaUnsatisfiedConditionException
     *             if some conditions are not met
     */
    public void goToAmountPage() throws Exception {
        
        // Wait until the initial page is loaded.
        windows.waitFor(this).image(10,  true, amountButtonImage);
        
        // Press amount button.
        amountButtonImage.clickOnCenter();
        
        // Wait for the amount page.
        windows.waitFor(this).image(10, true, amountPageMarkImage);
        
        // Tab x 6 to put the focus in the first field.
        windows.keyboard().tab(6);
    }

In the code above, we are using again a predefined wait sentence, this time to wait until the robot finds the amounts button ("importes") on the screen by searching the reference image.

In the next line, we can find one of the methods provided by the IFalconImage interface: clickOnCenter to click on the center point of the coordinates on which the robot has found the image, which is where the button is located. To use this method, we must have searched and found the button image on the screen explicitly or have used it as an argument in a method wait. The reason why is that once the image is found, its coordinates are assigned to the IFalconImage instance.

The robot will wait until an image with the text "Introducci├│n de importes" is detected on the screen, that represents the moment when the amount page is loaded. Then, it will use the tab key several times to put the focus on the first input field.

An IFalconImage type image can search for itself on the screen. Two of the available methods are search and found. The first one searches the image on the screen, while the second one tells us whether it was found or not. The method found should not be called without previously calling search. This way, we determine whether the company field exists or not. In the first version of the example website this field won't exist, whereas it will in the second version.

The items to be processed save the company name, the employee name and the amount.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
   /**
     * Action 'Process Item'.
     * 
     * @throws JidokaUnsatisfiedConditionException
     *             if some conditions are not met
     * @throws IOException 
     */
    public void processItem() throws JidokaUnsatisfiedConditionException, IOException {

        // Get the current item.
        Item item = ITEMS[currentItemIndex - 1];

        /*
         * Notify the start of the item processing.
         * 
         * This step is very important because it helps the platform to
         * calculate accurate statistics and time measures.
         * 
         * The second parameter is used to make it easier to locate logs.
         */
        server.setCurrentItem(currentItemIndex, item.getFullName());

        // Insert the item data.
        windows.keyboard().type(item.getCompany()).tab().pause();
        
        // Insert the name and the amount.
        windows.keyboard().type(item.getFullName()).tab().pause().
            type(item.getAmount()).tab().pause();

        server.sendScreen("Data inserted");
        
        // Save.
        windows.keyboard().space();

        // Wait until the information is saved.
        windows.waitFor(this).image(30,true, nameFieldPlaceholderImage);
        
        // Notify the result of the processed item.
        server.setCurrentItemResultToOK();
        
        // Go back to the correct field.
        windows.keyboardSequence().pressShift().typeTab(3).
            releaseShift().apply();
        
    }

We can see that an item processing is performed. The item has been defined as a list of POJOs which store the name, the company and the amount.

To perform the delay until the form is sent, we wait until the image taken for the input "Nombre" is found. As long as it is not detected, or the placeholder doesn't show up, the robot will not determine that the image exists on screen. In other words, the form has not reloaded after sending the items information to the server.

It will continue processing items until there are no items left, and once the process has ended, the robot will stop running. In the method cleanUp, it will try to close the browser received as a parameter in the robot setting. Remember that we should leave the resource in the same conditions as it was before the robot execution.

Tutorial: Build a bot using the Falcon module

Learn how to use the Falcon module in a robotic process

Open in Github Built: Fri, Nov 12, 2021 (02:39:09 PM)

On This Page

FEEDBACK