This page explains how to read the results of test runs returned by Automated Testing for Expression Rules.
The results of a test run will be returned in a TestRunResult data type, which organizes the results by application, followed by rules and finally by test cases.
The following diagram displays the different data types containing test result information:
TestRunResult
|_ ApplicationTestResult
|_ ObjectTestResult
|_ TestCaseResult
Each of these data types captures different metrics for a test run, along with additional metadata for test cases, expression rules, and applications. Each data type can be saved as a variable of the corresponding type when used in an expression rule, interface, or process model. Starting from the innermost data type:
| Data Type | Description |
|---|---|
| TestCaseResult | Contains test statistics for each test case. Test cases are grouped by expression rule. |
| ObjectTestResult |
Contains test statistics for each expression rule. Expression rules are grouped by application.
|
| ApplicationTestResult | Contains test statistics for each application. |
| TestRunResult | Contains test statistics for the test run. A test run can contain zero or more application. |
The following table contains a list of the different fields on each of the data types in this hierarchy. For additional details, see Data Type Definition section.
| TestRunResult | ApplicationTestResult | ObjectTestResult | TestCaseResult |
|---|---|---|---|
| id | id | id | id |
| type | uuid | uuid | name |
| status | name | name | status |
| totalExecutionTime | status | version | errorMessage |
| totalElapsedTime | totalExecutionTime | url | failureMessage |
| queueTime | testCount | totalExecutionTime | executionTime |
| startTime | errorCount | totalElapsedTime | elapsedTime |
| endTime | failureCount | queueTime | queueTime |
| testCount | passCount | startTime | startTime |
| errorCount | problemObjectsTestResult | endTime | endTime |
| failureCount | passObjectsTestResult | lastModifiedBy | |
| passCount | lastModifiedOn | ||
| executedBy | testCasesResult | ||
| applicationTestResults |
The following sections describes the TestRunResult data type and its children.
Note: ApplicationTestResult and ObjectTestResult will be sorted by name. TestCaseResult, will have the same order in which the test cases appear in the expression rule object.
TestRunResult contains test statistics for the test run across all applications being tested. A test run can contain zero or more applications.
| Field - Type | Type | Description |
|---|---|---|
| id | Number(Integer) | Unique ID for each test run in the system. This id is passed to the a!testRunResultForId function to retrieve results for the test run. |
| type | Text |
|
| status | Text | Specifies the status of the test:
|
| totalExecutionTime | Number(Decimal) | Sum of executionTime for the individual test cases that are part of the test run in seconds. Difference between endTime and startTime fields in seconds.
Note: Rounded to three decimals as they represent milliseconds. |
| totalElapsedTime | Number(Decimal) | Time it took to execute all test cases in the test run, from the moment the test was queued to the time the last test case finished.
Difference between endTime and queueTime fields in seconds.
Note: Rounded to three decimals as they represent milliseconds. |
| queueTime | Data and Time | Time and date when the first test case in the system or application(s) was added to the execution queue.
Note: Rounded to three decimals as they represent milliseconds. |
| startTime | Date and Time | Time and date when the first test case in the test run started executing.
Note: The timezone matches that of the user account calling this function. |
| endTime | Date and Time | Time and date when the last test case in the test run finished executing. If test cases, are still executing, this will be the end time for the last finished test case. If no test cases have been completed, this field will be empty.
Note: The timezone matches that of the user account calling this function. |
| testCount | Number(Integer) | Total number of test cases across all expression rules in the system, or selected application(s) being tested.
Note: If partial results are retrieved, this matches test cases executed thus far. |
| errorCount | Number(Integer) | Total number of errors in the system or selected applications. Either in the definition of expression rules, or in the definition of the test cases.
Note: If partial results are retrieved, this matches the number of errors in test cases executed thus far. |
| failureCount | Number(Integer) | Total number of assertions failed in the system or selected application(s).
Note: If partial results are retrieved, this matches the number of assertions failed in test cases executed thus far. |
| passCount | Number(Integer) | Total number of test cases passed in the system or selected application(s).
Note: If partial results are retrieved, this matches the number of errors in test cases executed thus far. |
| executedBy | Text | The username of the person initiating the test. |
| applicationTestResults | Array of ApplicationTestResult | Test result for the system or selected application(s).
Note: Objects that are not part of any application, will be grouped in an ApplicationTestResult structure with the application name: Other Objects; in this case, uuid will be set to an empty string. |
The following table shows a sample test run for an application test, which contains a total of 10 test cases, out of which there is 1 failure and 9 successfully executed test cases.
TestRunResult id: 2 type: Application Test status: FAIL totalExecutionTime: 0.023 totalElapsedTime: 0.816 queueTime: 2016-04-19T21:15:30.430 startTime: 2016-04-19T21:15:30.890 endTime: 2016-04-19T21:15:31.240 testCount: 10 errorCount: 0 failureCount: 1 passCount: 9 executedBy: michael.smith applicationTestResults: …
ApplicationTestResult contains test result for all expression rules in an application, and provides statistics for an individual application.
| Field | Type | Description |
|---|---|---|
| id | Number(Integer) | Unique ID for the application test result. |
| uuid | Text | Uuid of the application being tested. |
| name | Text | Name of the application. Or, Other Objects for objects that don’t belong to any application. |
| url | Text | URL of the application. Or, empty for objects that don't belong to any application. |
| status | Text | Specifies the status of the application being tested:
|
| totalExecutionTime | Number(Decimal) | Sum of executionTime for the individual test cases that are part of the application in seconds.
Note: Rounded to three decimals as they represent milliseconds. |
| testCount | Number(Integer) | Total number of test cases across all expression rules in the application being tested. |
| errorCount | Number(Integer) | Total number of errors in the application. Either in the definition of expression rules, or in the definition of the test cases
Note: If partial results are retrieved, this matches the number of errors in test cases executed thus far. |
| failureCount | Number(Integer) | Total number of assertions failed in the application.
Note: If partial results are retrieved, this matches the number of errors in test cases executed thus far. |
| passCount | Number(Integer) | Total number of passed test cases in the application.
Note: If partial results are retrieved, this matches the number of errors in test cases executed thus far. |
| problemObjectsTestResult | Array of ObjectTestResult | Contains object information and a reference to each of the object’s test cases with failed assertions or errors in the definition of the expression rule or in the test case definition. |
| passObjectsTestResult | Array of ObjectTestResult | Contains object information and a reference to each of the object’s test cases with passed test cases. |
The following table shows the details on the application that was tested in the previous section. Although this example is for one application test, a system test can include as many ApplicationTestResult data types as applications are in the system.
applicationTestResults id: 5379 uuid: _a-0000dbf6-6dd4-8000-643f-01ef9001ef90_4076 name: Common Objects url: https://myapp.appian.com/suite/design/1EkOlUEUY4g_iybY1hw3HrlEJ0gPV_iNyOG7M2JYrQ status: FAIL totalExecutionTime: 0.023 testCount: 10 errorCount: 0 failureCount: 1 passCount: 9 problemObjectsTestResult: ... passObjectsTestResult: ...
This section provides details for the problemObjectsTestResult and passObjectsTestResult elements, which are arrays of type ObjectTestResult.
The ObjectTestResult contains data for each of the expression rules with test cases.
| Field | Type | Description |
|---|---|---|
| id | Number(Integer) | Unique ID for the object test result. |
| uuid | Text | Object uuid. |
| name | Text | Name of the object. |
| version | Number(Integer) | Object version. |
| url | Text | URL to view the rule in expression rule designer. |
| totalExecutionTime | Number(Decimal) | Sum of executionTime for the individual test cases that are part of the expression rule. Difference between endTime and startTime fields in seconds.
Note: Rounded to three decimals as they represent milliseconds. |
| totalElapsedTime | Number(Decimal) | Time it took to execute all test cases in the object, from the moment the first test case was added to the queue to the time the last test case finished.
Difference between endTime and queueTime fields in seconds.
Note: Rounded to three decimals as they represent milliseconds. |
| queueTime | Date and Time | Time and date when the first test case in the object was added to the execution queue.
Note: The timezone matches that of the user account calling this function. |
| startTime | Date and Time | Time and date when the first test case in the object began executing.
Note: The timezone matches that of the user account calling this function. |
| endTime | Date and Time | Time and date when the last test case in the object finished executing.
Note: The timezone matches that of the user account calling this function. |
| lastModifiedBy | Text | Username who last modified the object. This can be used to create a task to inform that user when a test fails. |
| lastModifiedOn | Date and Time | Date and time when the object was last modified. |
| testCasesResult | Array of TestCaseResult | Test results for each of the test cases setup for an object (for example, a rule).
Note: Test cases are filtered (that is, pass, failure) on the ApplicationTestResult data type. |
Both, the problemObjectsTestResult and passObjectsTestResult elements, are arrays of type: ObjectTestResult. The Common Objects application contains 4 expression rules, one of which has a failing expresion rule. The example below reflects this by showing 1 problemObjectsTestResult and 3 passObjectsTestResult elements. For simplicity, the sample snippet only shows values for two expression rules, the details for the remaining expression rules are represented with "…"
problemObjectsTestResult: id: 5387 uuid: _a-0000dbf6-6dd4-8000-643f-01ef9001ef90_4135 name: EPC_getShipmentDate version: 2 url: https://mytestsite.com/suite/design/lIBwMofBiV totalExecutionTime: 0.004 totalElapsedTime: 0.816 queueTime: 2016-04-19T21:15:30.430 startTime: 2016-04-19T21:15:31.060 endTime: 2016-04-19T21:15:31.240 lastModifiedBy: michael.smith lastModifiedOn: 2016-04-19T21:15:27.050 testCasesResult: ... testCasesResult: ... testCasesResult: ... passObjectsTestResult: id: 5375 uuid: _a-0000dbf6-6dd4-8000-643f-01ef9001ef90_4112 name: EPC_isUserTypeExternal version: 1 url: https://mytestsite.com/suite/design/lIBwMofBiV totalExecutionTime: 0.004 totalElapsedTime: 0.784 queueTime: 2016-04-19T21:15:30.430Z startTime: 2016-04-19T21:15:30.990Z endTime: 2016-04-19T21:15:31.210Z lastModifiedBy: michael.smith lastModifiedOn: 2016-04-19T21:12:49.420Z testCasesResult: ... testCasesResult: ... testCasesResult: ... passObjectsTestResult: ... passObjectsTestResult: ...
The TestCaseResult stores data relevant to each of the test cases in an object.
| Field | Type | Description |
|---|---|---|
| id | Number(Integer) | Unique ID for the test case result. |
| name | Text | Name of the test case. |
| status | Text |
|
| errorMessage | Text | Error message due to configuration error or timeout. |
| failureMessage | Text | Assertion error message. |
| executionTime | Number(Decimal) | Time it took to execute the individual test case in seconds.
Note: Rounded to three decimals as they represent milliseconds. |
| elapsedTime | Number (Decimal) | Time it took for the test case to finish executing, including time in the queue. Difference between queueTime and endTime fields in seconds. |
| queueTime | Data and Time | Time and date when the test case was added to the execution queue. |
| startTime | Date and Time | Time and date when the test case started executing. |
| endTime | Date and Time | Time and date when the test case finished executing. |
The following table shows the details on the 3 test cases that were created for the PO_getShipmentDate expression rule. Notice that these test cases are grouped under a problemObjectsTestResult data type, as the overall test on this expression rule is considered a failure because one of the three test cases in it failed.
problemObjectsTestResult:
id: 5387
uuid: _a-0000dbf6-6dd4-8000-643f-01ef9001ef90_4135
name: PO_getShipmentDate
...
testCasesResult:
id: 1255
name: December
status: FAIL
failureMessage: Assertion expression returned false
executionTime: 0.002
elapsedTime: 0.635
queueTime: 2016-04-19T21:15:30.430
startTime: 2016-04-19T21:15:31.060
endTime: 2016-04-19T21:15:31.060
testCasesResult:
id: 1253
name: October
status: PASS
executionTime: 0.001
elapsedTime: 0.705
queueTime: 2016-04-19T21:15:30.430
startTime: 2016-04-19T21:15:31.130
endTime: 2016-04-19T21:15:31.130
testCasesResult:
id: 1254
name: November
status: PASS
executionTime: 0.001
elapsedTime: 0.816
queueTime: 2016-04-19T21:15:30.430
startTime: 2016-04-19T21:15:31.240
endTime: 2016-04-19T21:15:31.240
Parsing Batch Test Results for Expression Rules
