UI Automation Template

The UI Automation template details how to automate the Calculator application on a Windows machine. This template is based on the UI Automation module, which allows you to use techniques from the Microsoft UI Automation framework.

Microsoft UI Automation is an accessibility framework for Windows applications. It is typically applied to assistive technologies, such as screen readers, but it can be used in Appian RPA to detect the attributes of elements on a screen and then act on them.

When you use a template to create a robotic process, you are provided with a zip file containing your source code and a pre-built workflow in the Robotic process configuration page. Before you use this template, ensure you complete the prerequisites needed to use the UI Automation module.

This page explains the contents of the workflow and the source code generated by the UI Automation template. For an overview of the components involved in UI Automation, see the UI Automation Module.

Workflow

The UI Automation template's workflow contains four actions: Start, Open calculator, Calc, and End.

uiautomation-1.png

Similar to an Appian process model, each workflow has a start and end action. The other two actions on the workflow are responsible for opening the application and then performing calculations on the Calculator application.

If you click the list icon 951644.png on any action, you'll see it is associated with a method. Some of these methods are in the source code generated from the template. However, the Calc action references a low-code method called Calculate.

uiautomation-2.png

In the template, you will reference the parameters from this low-code method within your source code.

Source code

The source code contains a class named RobotCalculator that implements the IRobot interface and is annotated with @Robot. Every Appian RPA robotic process must have the IRobot interface and @Robot annotation to allow the Appian RPA API to identify the class as a robot.

RobotCalculator includes several methods and attributes. Some of the methods are directly associated with workflow actions; however, the basic template also generates additional methods that are called by the platform to prepare a robotic process or clean up a resource after the process ends.

The class contains the following methods:

Method Description
startUp() Called by the platform prior to the execution of the actions in the workflow. This is not directly associated with an action in the workflow.
start() Provides initial information to the process. This is associated with the Start action in the workflow.
startCalculator() Launches the Calculator application. This is associated with the Open calculator action in the workflow.
applyFormulas() Perform calculations on the Calculator application.
clickOnControl() Presses each button on the Calculator application. This method uses the ECalculatorControls enum to define the controls for the operating system.
end() Provides any final information to the process. This is associated with the End action in the workflow.
cleanUp() Called by the platform after all workflow actions have been executed, regardless of the execution's results. This is not directly associated with an action in the workflow.

The source code also contains ECalculatorControls enum. This is used to define the Ids for each button on the Calculator application for Windows 7 and Windows 10. It's unlikely you will need this enum in a traditional robotic process; however, if you want to build a library to manage an application, the elements of the ECalculatorControls enum may be useful.

Attributes

The first few lines in the template's source code reference the attributes and constants related to the robotic process:

  • CALC_EXECUTABLE (line 36) is a constant that represents the Calculator application's executable file.
  • CALC_TITLE_REGEX (line 41) is a constant that defines the title of the Calculator application.
  • IJidokaServer (line 44) is responsible for communicating with the Appian RPA Server. The server is what communicates information to the console.
  • UIAutomation (line 49) is the main component of the UI Automation module. This object allows you to access the other elements of the UI Automation framework.
  • Window (line 54) represents a visible window of an open application. This object will allow you to determine what kind of UI elements, or controls, your process can interact with.
  • IWindows (line 59) represents a Windows operating system. It enables functionality to interact with each window.
  • IWaitFor (line 64) determines a robotic process's wait time. This interface implements smart waits, which make the wait time dynamic by depending on certain actions to occur rather than a hard-coded time limit.
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
/**
	 * The Constant CALC_EXECUTABLE.
	 */
	private static final String CALC_EXECUTABLE = "calc.exe";

	/**
	 * The Constant CALC_TITLE_REGEX.
	 */
	private static final String CALC_TITLE_REGEX = "Calculadora|Calculator";

	/** The server. */
	private IJidokaServer< ? > server;

	/**
	 * The automation.
	 */
	private UIAutomation automation;

	/**
	 * The calc window.
	 */
	private Window calcWindow;
	
	/**
	 * The windows.
	 */
	private IWindows windows;
	
	/**
	 * The wait for.
	 */
	private IWaitFor waitFor;

startUp() method

The startUp() method is called prior to the execution of the actions in a workflow, so it is not associated with a workflow action. This method can be used to verify conditions on your resource, like determining if an application is available or verifying that the robotic process can be executed. The startUp() method is also where you can initialize the different instances needed by the robotic process.

Within the source code, you'll see the following declarations:

  • @Override (line 66) tells the platform to ignore the default implementation listed in the IRobot interface and instead use the following implementation.
  • server = JidokaFactory.getServer(); (line 69) initializes IJidokaServer.
  • windows = IWindows.getFreshInstance(this); (line 70) initializes IWindows, where this refers to the class that implements the IRobot interface. From this template, this would be the class RobotCalculator.
  • waitFor = windows.getWaitFor(this); (line 71) initializes IWaitFor, where this refers to the class that implements the IRobot interface. From this template, this would be the class RobotCalculator.
  • automation = windows.uiAutomation(); (line 73) initializes UIAutomation.
66
67
68
69
70
71
72
73
74
75
76
@Override
public boolean startUp() throws Exception {
	//Init the modules
	server = JidokaFactory.getServer();
	windows = IWindows.getFreshInstance(this);
	waitFor = windows.getWaitFor(this);

	automation = windows.uiAutomation();
	
	return IRobot.super.startUp();
}

start() method

The start() method is used to define the initial actions of the workflow and is associated with the Start action. In this template, the start() method is used to set the number of items that will be processed.

  • server.setNumberOfItems(1); (line 87) tells the server that there is only 1 item to process.
  • server.debug("Robot initialized"); (line 88) sends a log message of type debug to the server to indicate that the robotic process was correctly initialized.
85
86
87
88
public void start() {
	server.setNumberOfItems(1);		
	server.debug("Robot initialized");
}

startCalculator() method

The startCalculator() method launches the Calculator application using the UI Automation module. Once the application is open, the process will look for the title of the application specified in the CALC_TITLE_REGEX constant and set that as the root window where other process actions can occur.

  • Application calcApp = automation.launchOrAttach(CALC_EXECUTABLE); (line 99) opens the Calculator application from the executable file.
    • The launchOrAttach class method tells the process to attach to the application if it is open, otherwise launch the application and then attach to it.
  • calcApp.waitForInputIdle(30); (line 101) tells the process to wait for 30 milliseconds.
  • calcWindow = automation.getDesktopWindow(Pattern.compile(CALC_TITLE_REGEX)); (line 103) obtains the window of the Calculator application.
  • calcWindow.focus(); (line 105) sets the Calculator application as the active element.
  • server.sendScreen("calc"); (line 107) sends a screenshot of the current resource to the server. The screenshot will have description of "calc".
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
public void startCalculator() {

	try {
			
		// Open the Calculator APP and get the Window
		Application calcApp = automation.launchOrAttach(CALC_EXECUTABLE);

		calcApp.waitForInputIdle(30);
		
		calcWindow = automation.getDesktopWindow(Pattern.compile(CALC_TITLE_REGEX));

		calcWindow.focus();
			
		server.sendScreen("calc");

	} catch (Exception ex) {
		throw new JidokaFatalException("Error initializing the calculator: " + ex.getMessage(), ex);
	}
}

applyFormulas() method

The applyFormulas() method manages the app using the automationIds defined and gets the result.

  • @JidokaMethod (line 123) is an annotation used to mark the methods that should pass parameters from the workflow to the source code. The definition of the workflow item is listed in parentheses:
    • name = "Calculate" is the name of the low-code method. This is the method associated with the Calc action in the workflow.
    • description = "Formula to calculate" is the description listed on the low-code method.

Now that the applyFormulas() method is marked as being able to receive parameters from the workflow, you can define the parameters to pass using the @JidokaParameter annotation. The parameter names must match the names listed on the low-code method.

  • @JidokaParameter(defaultValue = "5", name = "first") String first, (line 124) defines the value of the first parameter.
  • @JidokaParameter(defaultValue = "+", name = "sign") String sign, (line 125) defines the operator of the sign parameter.
  • @JidokaParameter(defaultValue = "8", name = "second") String second) (line 126) defines the value of the second parameter.

uiautomation-2.png

Once the parameters are set, the process tells the server which item is currently being processed and starts entering the formula on the Calculator application.

  • server.setCurrentItem(1, first + sign + second); (line 129) tells the server which item is the current item being processed. This requires two parameters:
    • currentItemIndex is the 1-based item index of the total number of items to process.
    • currentItem is the identification of the item. In the template, the item is the calculation of three parameters defined above.
  • clickOnControl() (lines 132, 135, 138, and 141) is a method called to click each button on the Calculator application. The clickOnControl() method is configured starting on line 160.

Once the formula has been entered, you can retrieve the results and set the item's result to OK. Depending on which Windows version you are automating on, the property that you obtain results from will differ.

  • result = calcWindow.getTextBoxByAutomationId(ECalculatorControls.RESULT.getAutomationID()) (line 147 and 149) locates the textbox control on the Calculator window. This is where the sum of 5 + 8 is listed. For demonstration purposes, this template uses the ECalculatorControls enum to define the Id of the elements on the Calculator.
    • .getValue is the name of the property to retrieve results from if you are on Windows 7.
    • .getName is the name of the property to retrieve results from if you are on Windows 10.
  • server.setCurrentItemResultToOK(result); (line 152) sets the current item result to OK. This accepts a string parameter where you can add a description of the processing result. In the template, this will return the sum of 5 + 8.

If the formula is not entered successfully or an exception occurs, server.error() (line 154) will send a log message of type error to the server and JidokaItemException (line 155) is thrown.

114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
/**
 * Apply formulas.
 *
 * @param first  the first operand
 * @param sign   the sign
 * @param second the second operand
 * @throws PatternNotFoundException the pattern not found exception
 * @throws AutomationException      the automation exception
 */
@JidokaMethod(name = "Calculate", description = "Formula to calculate")
public void applyFormulas(@JidokaParameter(defaultValue = "5", name = "first") String first,
		@JidokaParameter(defaultValue = "+", name = "sign") String sign,
		@JidokaParameter(defaultValue = "8", name = "second") String second)
		throws AutomationException {
	try {
		server.setCurrentItem(1, first + sign + second);

		for (String number : first.split("")) {
			clickOnControl(number);
		}
	
		clickOnControl(sign);

		for (String number : second.split("")) {
			clickOnControl(number);
		}
		
		clickOnControl("=");
		
		String result = "";
		
		// The property from which the value is obtained is different between win7 and win10
		if (SystemUtils.IS_OS_WINDOWS_7) {
			result = calcWindow.getTextBoxByAutomationId(ECalculatorControls.RESULT.getAutomationID()).getValue();
		} else {
			result = calcWindow.getTextBoxByAutomationId(ECalculatorControls.RESULT.getAutomationID()).getName();
		}
			
		server.setCurrentItemResultToOK(result);
	} catch (Exception e) {
		server.error("Exception calculating result", e);
		throw new JidokaItemException("Error calculating the result");
	}

}

clickOnControl() method

The clickOnControl() method allows you to click on UI elements like buttons, menus, hyperlinks, and checkboxes within a window. These elements are called controls. In the template, this method is called by the applyFormula() method to interact with the button controls on the Calculator window.

First the method must obtain the Ids of the controls available on the window.

  • String automationID = ECalculatorControls.getByCode(controlCode).getAutomationID(); (line 168) obtains the automation Id of an element from the ECalculatorControls enum.
  • server.debug (line 170) sends a message of type debug to the server including a description about which control was clicked.

Once the automation Ids are located, they can be used to identify the controls available on the window so the process can interact with the buttons.

  • Button control = calcWindow.getButtonByAutomationId(automationID); (line 187) finds the button control located on the Calculator window.
  • control.click(); (line 189) clicks on the button control.

The template also provides an example of how to have the process wait to see if a control is available before interacting with it. You are not required to use the wait, however, it is a good practice in cases where the controls may be delayed in appearing.

  • waitFor.wait (line 173) waits until the control is available.
  • AutomationBase control = calcWindow.getControlByAutomationId(automationID); (line 177) finds the control located on the Calculator window.
  • return control != null; (line 179) returns the control when the Automation Id is not null.
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
private void clickOnControl(String controlCode) throws AutomationException {

	String automationID = ECalculatorControls.getByCode(controlCode).getAutomationID();
	server.debug(String.format("Click on control [%s] with automationID [%s]", controlCode, automationID));
		
	// Wait element availability
	waitFor.wait("Waiting for : " + automationID, () -> {
			
		try {

			AutomationBase control = calcWindow.getControlByAutomationId(automationID);
				
			return control != null;
				
		} catch (AutomationException e) {
			return false;
		}
		
	});
		
	Button control = calcWindow.getButtonByAutomationId(automationID);
	control.click();
}

end() method

The end() method is used to define the final actions of the workflow and is associated with the End action. The template calls server.info (line 197) to send a log message of type info to indicate the robotic process has finished.

1
2
3
4
5
6
/**
  Action to end the robot.
 */
public void end() {
	Server.info("End process");
}

cleanUp() method

The cleanUp() method is invoked by the platform when all actions in the workflow have finished their execution, or if the process ended with errors or failed.

The purpose of this method is to leave the resource in the same state it was before the robotic process started its execution. This is where you can add tasks to clean up the resource, like closing an application or logging out. If this method is not marked by the annotation @Override, the platform will call the default implementation present in the interface IRobot.

This template's cleanUp() method contains an if statement to address the actions that should occur at the end of a successful process and actions to occur in the event of exceptions.

If the process ends successfully:

  • calcWindow.close(); (line 205) will close the Calculator application window.

If an exception occurs:

  • server.warn(e, e); (line 209) will send a log message of type Warn to the server.
  • windows.killProcess("Calculator.exe"); (line 210) will terminate the Calculator application.
200
201
202
203
204
205
206
207
208
209
210
211
@Override
public String[] cleanUp() throws Exception {
	try {
		if(calcWindow != null) {
			calcWindow.close();
		}
	} catch (Exception e) {
		server.warn(e, e);
		windows.killProcess("Calculator.exe");
	}
	return new String[] {};
}

This version of the Appian RPA documentation was written for Appian 20.4, and does not represent the interfaces or functionality of other Appian versions.
Open in Github Built: Mon, Nov 15, 2021 (03:03:53 PM)

On This Page

FEEDBACK