Rule Testing with Appian

Introduction

This document provides general guidance and best practices for testing expression rules with the test cases feature.

For information on automated testing strategies and testing rules in bulk, see: Automated Testing for Expression Rules

What Can Test Cases Be Used For?

Test cases aren't just useful for teams that have large testing infrastructures or use automated testing, they can help improve the maintenance of individual rules as well.

Save Ad-Hoc Tests

  • The ad-hoc, informal tests that you write while working on a rule can be saved as test cases that you can delete later.

Inline Documentation

  • Test cases can describe the expected behavior and results of an expression rule which can inform other developers of what the rule should do.

Test-Driven Design

  • You can write test cases before you write your rule, then use them to determine when you’ve succeeded. This is known in the software-development world as "Test-Driven Development."

Use Cases

  • Write test cases that cover the various ways in which your rule can be used.

Edge Cases

  • Write test cases for rare but potential inputs that could break the rule.

Regression Testing

  • The test cases you write can also be run in bulk so you can test the rest of your application/system for quick feedback on potential unintended changes to objects that rely on the rules you edited.

General Guidance

The following list of guidelines can help direct your testing efforts and avoid common mistakes. These guidelines have been generalized; for examples on how they are specifically applied, see the Common Testing Scenarios section below.

  1. Test only your logic. Make sure your test cases are testing the logic of your rule and not testing Appian built-in logic. You can rely on Appian to function as expected. For example, don’t test that the sum() function can add numbers. If your rule is taking numerical values from multiple sources and summing them together, you may want to create tests that ensure the inputs are of the correct type or that it can handle nulls, but you should not test that 2 + 2 = 4.

  2. Make tests as specific as possible. Isolate one part of the expression rule when creating a test case, so the result of the test execution is more specific. For example, if your rule can have three different results, instead of having one test that checks three parts of a rule, have three separate tests per expected result.

  3. Make sure the test isn’t testing itself. If the assertion is the same as the expression rule definition, it’s what we call a "1=1 test," and we don’t learn anything from it. This is a surprisingly easy mistake to make.

  4. Write reliable tests. Tests that incorporate values from an external system (either via a query, or using a date/time function, etc.) in their definitions are considered "fragile," as their results can be affected by changes to those external sources. For example, a test that queries a table for the values in select rows could fail when the values in those rows are changed by another process; in this case the test failure would not reflect a failing of the rule's logic.

Test Case Management

Writing and maintaining unit tests are an integral part of ensuring the long term success of an application. These unit tests can be used to preserve the core functionality of expression rules. The Manage Test Cases dialog can assist in this maintenance. For more information on writing test cases, see Expression Rule Test Cases.

Manage test cases

All of your unit tests for the expression rules in your application can be managed from one place. From this view you are able to identify rules without any test coverage (rules without test cases), run the test cases for any number of expression rules in an application, and review the results of a run in order to address any failures.

Appian recommends running the test cases for rules at least once prior to deployment. You will be warned during direct deployments if any packaged rules are missing test coverage.

Throughout development you should ensure that all of your expression rules have test cases, and that all of their test cases are passing. From the Application Settings menu, select Manage Test Cases to access this view.

Application settings menu

Manage test cases summary view

From the Manage Test Cases dialog you can address any issues regarding your test health using the buttons and filters provided:

  1. Last modified by filter: Filters by the user or users who last modified any of the rules. This is particularly useful during code reviews or more generally throughout the development process to address any failures caused specifically by your own changes.
  2. Last modified from / to filter: A date range filter which finds rules modified within a particular date range. For example, this could be used to select all rules changed since the last deployment date.
  3. Only view rules in patch filter: Filters for rules in your patch. After you build your patch you can filter for patched rules to make sure all of your rules have test coverage and all of your tests are passing before you deploy to another environment.
  4. Only view rules without test cases filter: Filters by rules without test coverage. Rules without coverage are easy to identify in the grid by the ban icon next to the rule name (see #6 in the image above). The name of the rule in the grid links to the expression rule definition which will allow you to add test cases. For more details on writing test cases, see Expression Rule Test Cases.
  5. Run All / Run Filtered / Run Selected button: Runs the test cases for your selected rules. This button will display different actions depending on what filters you apply and what rules you select in the grid. Run All allows you to run the test cases for all rules in the application. Run Filtered allows you to run only the rules applicable to the filters you have selected. Finally, Run Selected allows you to run the test cases for only the rules you have selected from the grid.

Results summary view

After you have selected your expression rules and run their test cases you will see the results view. Note that running tests can take a few minutes depending on the complexity of your expressions. The expression rule statuses will refresh automatically as their test cases finish.

Results summary view

  1. Go back link: Returns to the summary view. If you need to run the tests for a different set of rules you can always go back to select any additional rules from the first view. The results for your current run however, will not be preserved and you will have to rerun all rules to return to the results view.
  2. An at-a-glance summary at the top of the results view gives a general overview of the rule results:
  3. Expression Rules: The number of rules for which test cases were executed
  4. Status As Of: The timestamp for the last time rules were run from this view
  5. Expression Rules Status: An overview of the result status for all selected expression rules. There are three possible statuses which are reported on in this section: passed, failed, and without test cases.
    • Passed: The number of rules for which all test cases passed successfully
    • Failed: The number of rules where one or more test cases for the rule failed, timed out, or hit an error
    • Without test cases: The number of rules in the current selection that are missing test coverage and do not have test cases defined
  6. Execution Status: The progress for the current execution. While testing is in progress, a progress bar will be shown displaying the progress of rule execution. An execution complete icon, the thumbs up icon, is shown whenever execution has completed. The progress bar will be shown again whenever test cases are being rerun from this view (see #4 rule results section).
  7. Filter by the status of the rules. You can filter the rule results grid by the three possible statuses for the rule which are: passed, failed, or no test cases. Hovering over the test status icon will give details into the number of test cases that ran for the rule and the success rate of the execution. For example, “5 of 5 passed” would indicate a passed status while “4 of 10 passed” would indicate a failed status.
  8. Run All Rules / Run Filtered Rules button: Runs the test cases for all, or a filtered selection, of the rules. As you address failing test cases, you will want to run them again until all test cases pass. You do not need to return to the main view in order to rerun rules. Either run all rules at once using the Run All Rules button or, by filtering, you can select a subset of the rules to run such as all failed rules or all rules in the patch. The Last Test Run column in the grid reflects the last time the test cases for that rule were executed from the dialog. At first, this timestamp may be the same for most rules, however, as you address failures and rerun, the column will update. Also note, the Status As Of section in the results summary updates to the timestamp for the last test execution from this view. If you close the dialog before your tests have finished executing, you will lose the progress of your current run and will have to rerun the tests in order to review your results again.

You can also manage you test cases when comparing and deploying your packages. If any of the rules in your package are missing test cases, Appian will remind you to add test cases before continuing with your deployment. Regardless of your application’s test coverage, you can always execute test cases during direct deployments.

Rule results

After getting a high level overview of the test health for your selected rules, you can drill down further to address test case failures for each rule.

Rule results view

  1. Selecting a row from the summary of the results grid will display a more detailed results view for that specific rule. This view shows more information about each test case and its results for the selected rule.
  2. The name of the selected expression rule links to the object definition where you can address test failures or modify the rule.
  3. Details around the last modifier and last modified timestamp for the rule.
  4. Filter by the test case status. These statuses are a simplified version of the statuses you are already familiar with from the expression rule. Passed, Failed, Error, and Timed Out. See Test Status and Results for further details.
    • Note: A test will trigger a time out if it takes over a minute to run. There are a number of ways a test case can time out, such as by having complex expressions or external integration dependencies. Review your code and check for dependencies to resolve any time outs.
  5. Run All Cases button: Reruns all of the test cases for the selected rule. This will update the timestamp in the Last Run column. Test cases will update automatically to the correct, corresponding test status.
  6. The name of the test case is a link to further details about the test case result.

Individual test case results

Selecting the test case name from the rule results grid will provide you with more information about the test case execution result such as the status, execution time, rule inputs, and results of the selected test case.

Individual results view

  1. Go back: Links to the previous view, allowing you to return to the grid containing the results of the rule’s entire execution. Your results are preserved if you go back to select another test case to review the results for.
  2. Summary of the results for the test case including its name, test case status, execution time, and assertion type. See more about Test Case Assertions.
  3. Inputs grid: Rule inputs for the test case. This grid contains the name and value of the rule input. If the value of the rule input is an expression, this column will contain the expression and not the executed form of the value.
  4. Results section: This section could contain different views depending on the assertion type:
    • Test output matches the asserted output: For this assertion type, a comparison on the failed assertion will be provided in the results section. If the test fails, the comparison will highlight the exact failure between the expected and the actual result. If the case passes, there will not be any highlighting.
    • Assertion expression evaluates to true: For this assertion type, the defined expression is expected to evaluate to true, therefore this test will fail if an error or time out is hit or the expression evaluates to anything other than true. The results view will include the assertion expression definition for the test case and the actual value and type of the evaluation if the test was able to run.
    • Test completes without errors: If the assertion passes, the evaluated value and type of the result from the execution will be printed in the results section. If there is an error, an error message will display in the results section. The actual type and value of the result will not be included because the expression could not be evaluated.
    • Time Out: If the test times out, a time out message will be provided in the results section.

Any time an error interrupts test execution, the error message will be displayed in the results section.

Common Testing Scenarios

The following scenarios describe common expression rule uses and how they can be tested. Each of these scenarios reflects different guidance and practices related to testing expression rules.

Dates or Times

Testing a rule that returns an array of years based on the current year (uses the today() function). This scenario reflects guidance on:

  • Writing reliable tests when dealing with dynamic values

  • Avoiding a "1=1 test"

  • Testing your application logic, not Appian's function library

CDT Input and Output

Testing a rule that takes a CDT array and returns a filtered subset of that CDT array. This scenario reflects guidance on:

  • Writing reliable tests when working CDTs and external systems

  • Only test expression rule logic

  • Isolate parts of the expression rule

Email Address Format Validation

Testing a rule that validates whether the provided text input is a valid email address. This scenario reflects guidance on:

  • Test-Driven Development

  • Isolate parts of the expression rule

Dates or Times

In this scenario, we have a rule that takes a single year as an input and returns an array of the years between the input year and the current year.

We use this rule to populate a dropdown for an application that allows employees to retrieve a copy of their W2 from a select year. In this case we pass the employee’s start year to the rule.

If that year was 2008, we would get a list of values as seen in the dropdown in the image below.

The Rule

Rule Inputs

startYear (type: Integer)

Rule Definition

1
2
3
4
5
a!localVariables(
  local!numYearsSinceStartYear: year(today()) - ri!startYear,
  local!arrayOfIndices: enumerate(local!numYearsSinceStartYear + 1),
  reverse(local!arrayOfIndices + ri!startYear)
)

How should we test this?

Because this rule uses the today() function, the same input will result in a different output depending on the year that it is run. If we create a test case with 2008 as the input, and we run it in 2016, the output will be an array of 9 values:

{2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016}

If we set that array as the asserted output, the test case will pass in 2016, but fail when run in 2017 because the same test case will return 10 values:

{2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017}

This is a clear sign that we should avoid asserting the value of this rule directly — any tests that assert a value from something that is dynamic or context dependent like this is likely fragile.

Since we can’t assert the literal results in this case, we might try to write an expression to calculate the array to compare. So, maybe we would write an expression that would calculate the array of values, like this:

1
2
3
4
5
a!localVariables(
  local!numYearsSinceStartYear: year(today()) - 2008,
  local!arrayOfIndices: enumerate(local!numYearsSinceStartYear + 1),
  reverse(local!arrayOfIndices + 2008)
)

But, in doing so, we’ve basically rewritten the original expression rule, which doesn’t actually test anything; since we’re rewriting the logic of the expression rule, it’ll just output the rule we’re seeking to test. We call this a "1=1 test."

So, how do we actually test our rule in this situation? Well, instead of testing that the result of one test case matches the result of running the entirety of the expression rule, we’ll break down the result of the rule into testable components.

Test Cases

Can return a single-item array

Here we assert that when the start year is the current year, we should always expect the rule to output only a single year. We don't need to assert that the year is 2016, 2017, or 2034 because we are confident that year(today()) is always going to return the right current year in the expression rule. Our tests don’t need to test Appian functionality.

Returns an array of the correct length

Here we assert that we'll always have a certain number of values in the output array given the number of years ago the start date is. A common error in rule logic is that values are sometimes off by one index. Testing for 1 and for 6 years is enough coverage that if they both pass, we can safely assume all intervening and exceeding years will work as well because the logic is straight-forward, and we are still relying on Appian functions to work as intended.

Output type is correct

When writing test cases, we should also keep in mind how other application objects use this rule, and what assumptions that are making about what the rule returns. For example, another rule might take the output of this rule and use it as the list parameter for the apply function. If this sometimes returned a scalar, then the apply function would break. We should protect & document these types of assumptions in test cases.

Above we are asserting that even if we only return a single year that it’s an array type. This test case can serve as documentation for other developers to understand that this is the expected behavior; if they make changes that affect the output data type, this test will fail.

Adapting the Rule and Test Cases

Weeks later it turns out there is a bug where the rule breaks when a future year is passed into it, so we add some additional logic to handle this case. To fix this bug, we'll add logic to check whether the number of years since the start date is negative; if so, we'll return an empty integer array to handle this case.

1
2
3
4
5
6
7
8
9
a!localVariables(
  local!numYearsSinceStartYear: year(today()) - ri!startYear,
  local!numYearsIsNegative: local!numYearsSinceStartYear < -1,
  if(
    local!numYearsIsNegative, 
    tointeger({}),
    reverse(enumerate(local!numYearsSinceStartYear + 1) + ri!startYear)
  )
)

We'll add another test case that asserts the output value is an empty integer array for any input values that are future start years.

CDT Input and Output / External Data

In this scenario, we have a rule that takes a set of purchase requests (a CDT array) and a type (like stationary, or electronic equipment, or office furniture), and returns only the requests that match the type. This is used by multiple parts of an application for ordering office supplies. One process model queries for all new purchase requests and sends them to this rule to be sorted by type and sent to the appropriate department.

The Rule

Rule Inputs

purchaseRequests (type: Purchase Request [Array])

type (type: Text)

Rule Definition

1
2
3
4
5
6
a!localVariables(
  /* Find the indices of the purchase requests of a given type, then */
  /* use those indices to only return those items from the array     */
  local!filteredIndices: wherecontains(ri!type,index(ri!purchaseRequests,"type",{})),
  index(ri!purchaseRequests, local!filteredIndices,{})
)

How should we test this?

Because the input to this rule will be different depending on the day (since we are querying a database that is constantly being updated), we can’t create a test output that will consistently be the same. We don’t want to run the same query on the purchase requests table and use its result as the input value here, because it’ll be different from moment to moment. So, we need to define a static set of purchase requests to test against to safeguard against making the test fragile.

Since our rule only filters a set of data, that’s the only part we really need to test. We don't need to test what the entire CDT definition of a purchase request data type looks like, and we don't need to test the values of all the fields in the purchase request because they aren't relevant here. So, we need to create our data set with purchase requests of different types and check to make sure that the result of running the rule only contains requests of a single type.

Test Cases

We came up with the following test cases to ensure our rule is working as intended.

Rule's logic filters properly

Since we know exactly what fields of the purchaseRequest input the rule needs, we can hardcode some example data by just adding a dictionary with the fields of the CDT.

Since the rule input is already of type array of PurchaseRequest, any values we pass into that rule input will be cast to the appropriate type. We don't need to query on real data, and we can hardcode our types to avoid any brittleness without sacrificing test coverage. We can specify the test input as the expression below:

1
2
3
4
5
6
{
  {type:"OFFICE"},
  {type: "ELECTRONICS"},
  {type:"PAPER"},
  {type: "ELECTRONICS"}
}

Notice how we didn't add any of the other fields to the CDT, we know we don't care about these for this specific rule, so don't bother!

This tests that a set of Purchase Requests with varying purchase types will filter out all requests that don’t match the specified type. We check that the remaining purchase requests don’t have any with a different type than the one specified. We've effectively tested the logic of this rule without having to rely on any actual data from an external source.

Returns an array of the correct length

Another aspect we might want to verify is that the expected number of items was returned. If we expect 7 results, we can check to make sure we get 7 unique results.

We can use the same test inputs for this case, so let's duplicate the test case (select it in the test cases grid and click the duplicate button) and assert the length of the test output.

Though the test cases in this scenario have relatively simple assertions you should always make separate test cases for them. This will assist you when troubleshooting the rule - if the assertions were all in one test case then you wouldn't immediately know if the output was wrong because it is returning the wrong status values or it's not returning all the items with a given status value.

Email Address Format Validation

In this scenario we have a rule that takes a text input, performs a lot of different logic to ensure it is in the format of a valid email address, and returns a boolean corresponding to whether or not it is a valid email address format.

We use this rule to validate a user's input on an email address field on a customer onboarding form and block the user from submitting the form until they provide a valid email address.

The Rule

Rule Inputs

address (type: Text)

Rule Definition

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
if(
  or(
    len(trim(ri!address)) > 255,
    length(split(trim(ri!address), " ")) > 1,
    count(split(ri!address, "@")) <> 2
  ),
  false(),
  a!localVariables(
    local!localPart: split(trim(ri!address),"@")[1],
    local!domainPart: split(trim(ri!address),"@")[2],
    if(
      or(
        length(split(local!domainPart, ".")) < 2,
        contains(split(local!localPart, "."), ""),
        contains(split(local!domainPart, "."), ""),
        not(isnull(stripwith(lower(local!domainPart), "abcdefghijklmnopqrstuvwxyz1234567890-."))),
        not(isnull(stripwith(lower(local!localPart), "abcdefghijklmnopqrstuvwxyz1234567890-._+'&%")))
      ),
      false(),
      true()
    )
  )
)

How should we test this?

When first identifying that you need to create this rule, it is likely that you are thinking about examples of valid and invalid email addresses to understand what rule's behavior will need to be. This rule actually has a lot of separate pieces of logic working together here - this is a good opportunity to use Test-Driven Development when writing this rule.

In Test-Driven Development you'd start creating a test case for each example email address strings you need to validate and whether or not they'd return true or false. Then once you were finished, you would start writing the expression rule itself and continuously test it via the test cases grid to start watching tests pass as you implement more and more of the necessary logic. This would help ensure that none of the relevant examples we are aiming to support are missing.

Test Cases

Happy Path Case

Error Cases

This rule has a lot of separate pieces of logic working together and it will be up to you to determine how much of it you'll want to document and test. In the example below we are testing that all the logic pertaining to the @ symbol works properly:

Edge Cases and Documenting the Output Type

Since this rule will be used by many other rules, we'll also want to make sure that we document what the output type will be in most occasions. The next two test cases below ensure that even in edge cases, the rule still returns a scalar boolean value.

Rules that have a lot of unrelated logic in them (like this one) should have test cases that describe each of these parts of logic separately. Like we mentioned in the CDT rule example, even though these are simple assertions and you could verify many of them together by just using and() in your assertion expression, separating your assertions to multiple test cases will make debugging your rule easier. Having multiple test cases is a good way to define the expected behavior of the rule, which isn't possible to decipher from a single test case with a complicated assertion.

Open in Github

On This Page

FEEDBACK