Using UI Automation

One of the automation techniques available on the Appian RPA platform is the use of the UI-Automation framework.

Microsoft UI Automation is an accessibility framework that allows Windows applications to provide and consume programmatic information about user interfaces (UI). It provides programmatic access to most user interface elements on the desktop.

In order to be able to use this framework from Java, we have included in the API a new component that provides us with a set of objects (java classes) that allows us to invoke this functionality natively. This component is ui-automation.

The documentation in this section is not intended to serve as documentation of Microsoft UI Automation or ui-automation objects. All the documentation about this framework can be found in their respective links.

This section of the documentation will show us how to use your objects from the Appian RPA API.

Pre-requisites

According to Microsoft's official documentation, this framework should run on almost all version of Windows, form XP onwards. We have verified its use, without installation of additional components, in systems based on Windows 7 and Windows 10. In the case of needing to use it in previous versions, it is possible that some components not present in the system are needed.

Initializing the UI-Automation module

In order to use the framework ui-automation it is not necessary to declare any additional dependence to the existing one in a Appian RPA robot. The necessary dependencies are already included in the core module jidoka-client-api.

1
2
3
4
5
<dependency>
    <groupId>com.novayre.jidoka.module</groupId>
    <artifactId>jidoka-client-api</artifactId>
    <version>${jidoka.version}</version>
</dependency>

The inital object from where we access the rest of the elements of the framework is UIAutomation. This object is instantiated in the following way:

IUIAutomation automation = UIAutomation.getInstance();

Once the object is instantiated we can begin to launch a new program or hook the instance to a working application.

1
2
3
automation.launchOrAttach(CALC_EXECUTABLE);
 
waitFor.window(CALC_TITLE_REGEX);

Inspector of elements

When automating an application one of the most important activities is to know how to reference the components of the user interface of the application (text fields, buttons, checkbox, list, etc.) to use them in the code of Appian RPA robots.

The philosophy we follow in Appian RPA is to use the element inspector closest to the technology of the applications we are automating. In this case we use "Microsoft's Inspect" element inspector.

Download and Installation

To get the item inspector you need to download the latest version of the Windows SDK. The Inpsec.exe utility is included in the SDK.

It is not necessary to install all components. The only necessary ones are:

  • Windows SDK Signing Tools for Desktop Apps
  • Windows SDK for UWP Managed Apps

Once installed, the executable inspect.exe is located in the path *C:\Program Files (x86)\Windows Kits\10\bin<version><platform>*.

Inspect Elements

Once the tool has been executed, we will be able to navigate between the elements of the open windows and obtain the necessary properties to reference the components in the robot code.

8028212.png

The UI-automation framework has mechanisms to locate the elements of the windows using different available properties. Depending on the application to be automated, we may have different attributes available. The most commonly used are:

  • Automation ID
  • Name
  • ClassName
  • ControlType

Principal components

UIAutomation

This is the main component that we obtain when initializing the module. The usual use of this comment will be to start applications. We can do it using the following method:

1
2
Application calcApp = automation.launchOrAttach(CALC_EXECUTABLE);
calcApp.waitForInputIdle(30);

Window

The window object represents a visible window of an open application. This window will represent the root component from which we will obtain the rest of the controls.

The following code is an example of obtaining a window from the Windows Calculator.

Window calcWindow = automation.getDesktopWindow(Pattern.compile(CALC_TITLE_REGEX)); calcWindow.focus();

Controls

From the window object we can obtain the controls on which we will have to act or obtain information. There are several useful methods to obtain these controls. The framework supports multiple components. All these controls have their own class and all extend from the AutomationBase class. We will use it to show some of the usual methods to get the controls:

  • calcWindow.getControlByAutomationId(automationID);
  • calcWindow.getControlByClassName(className);
  • calcWindow.getControlByControlType(index, ControlType.Button)
  • calcWindow.getControlByName(name)

We could also use the Search class, for example:

Button button1 = window.getButton(Search.getBuilder().automationId("btnOK").build())

Robot example: Calculator

In the console you have a template to generate a Window calculator automation example robot.

In this page we are going to comment on the methods included in the robot generated from this template.

Workflow

The robot has a workflow with four actions. The "calc" action contains the parameters that the robot will use to make a calculation with the Windows Calculator.

modules-ui-automation-workflow.png

These are the parameters to be specified in the robot configuration. In the "applyFormulas" method you can see how these parameters are defined and used.

modules-ui-automation-actions.png

"Start" Method

In this method we initialize the Appian RPA components used in the robot, including the UIAutomation object:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
/**
 * Initialize the modules
 *
 * @throws JidokaUnsatisfiedConditionException
 * @throws AutomationException
 * @throws PatternNotFoundException
 */
public void start() throws JidokaUnsatisfiedConditionException, PatternNotFoundException, AutomationException {
 
    server = JidokaFactory.getServer();
    windows = IWindows.getFreshInstance(this);
    waitFor = windows.getWaitFor(this);
 
    automation = UIAutomation.getInstance();
 
    server.setNumberOfItems(1);
     
    server.debug("Robot initialized");
}

"startCalculator" Method

Once the modules are uncivilized we will use the "startCalculator" method to open the windows calculator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
/**
     * Start calculator.
     */
    public void startCalculator() {
 
        try {
             
            // Open the Calculator APP and get the Window
            Application calcApp = automation.launchOrAttach(CALC_EXECUTABLE);
 
            calcApp.waitForInputIdle(30);
         
            Window calcWindow = automation.getDesktopWindow(Pattern.compile(CALC_TITLE_REGEX));
 
            calcWindow.focus();
             
            server.sendScreen("calc");
 
        } catch (Throwable ex) {
            throw new JidokaFatalException("Error initializing the APP: " + ex.getMessage(), ex);
        }
    }

"applyFormulas" Method

This method obtains the parameters used in this test. The identifiers of both systems are different, the way to obtain the information of some of its components can be different between versions of Windows as well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
/**
     * Apply formulas.
     *
     * @param first  the first operand
     * @param sign   the sign
     * @param second the second operand
     * @throws PatternNotFoundException the pattern not found exception
     * @throws AutomationException      the automation exception
     */
    @JidokaMethod(name = "Calculate", description = "Formula to calculate")
    public void applyFormulas(@JidokaParameter(defaultValue = "5", name = "first") String first,
            @JidokaParameter(defaultValue = "+", name = "sign") String sign,
            @JidokaParameter(defaultValue = "8", name = "second") String second)
            throws PatternNotFoundException, AutomationException {
         
        server.setCurrentItem(1, first + sign + second);
 
        for (String number : first.split("")) {
            clickOnControl(number);
        }
 
        clickOnControl(sign);
 
        for (String number : second.split("")) {
            clickOnControl(number);
        }
         
        clickOnControl("=");
         
        String result = "";
         
        // The property from which the value is obtained is different between win7 and win10
        if (SystemUtils.IS_OS_WINDOWS_7) {
            result = calcWindow.getTextBoxByAutomationId(ECalculatorControls.RESULT.getAutomationID()).getValue();
        } else {
            result = calcWindow.getTextBoxByAutomationId(ECalculatorControls.RESULT.getAutomationID()).getName();
        }
         
        server.setCurrentItemResultToOK(result);
    }

"clickOnControl" Method

This is the method in charge of clicking on the elements of the calculator. In the method we can see an example of how to make sure the control is available before clicking on it. It is not obligatory but it will be a good practice in the cases in which the controls can delay in appearing in the window:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
/**
     * Click on control.
     *
     * @param controlCode the control code
     * @throws AutomationException the automation exception
     */
    private void clickOnControl(String controlCode) throws AutomationException {
 
        String automationID = ECalculatorControls.getByCode(controlCode).getAutomationID();
 
        server.debug(String.format("Click on control [%s] with automationID [%s]", controlCode, automationID));
         
        // Wait element availability
        waitFor.wait("Waiting for : " + automationID, () -> {
             
            try {
 
                AutomationBase control = calcWindow.getControlByAutomationId(automationID);
                 
                return control != null;
                 
            } catch (AutomationException e) {
                return false;
            }
             
        });
         
        Button control = calcWindow.getButtonByAutomationId(automationID);
 
        control.click();
    }

"End" method

Finally, before finishing the robot, we will close the application:

1
2
3
4
5
6
7
8
9
10
11
12
13
/**
     * End.
     */
    public void end() {
 
        try {
 
            calcWindow.close();
             
        } catch (Exception e) {
            server.warn(e, e);
        }
    }
Open in Github Built: Fri, Nov 12, 2021 (02:39:09 PM)

On This Page

FEEDBACK