Configure Sync Options

After creating a synced record type, configure sync options to ensure your data remains up to date and readily available in Appian:

• Schedule syncs to capture data changed by external systems.
• Use smart services to sync data changed in Appian.
• Use web APIs to sync data from the source.

You can also configure availability options to keep your data available in Appian—even if a sync fails—and configure sync filters to specify which data is synced in Appian.

Scroll or drag to pan · ⌘+/⌘− to zoom · Esc to close

Choosing the right sync options

The combination of sync options you choose to configure depends on your business needs.

The following table outlines a few common data scenarios and which sync options you can use to keep that data fresh. Multiple scenarios may apply to your record type, so you may want to enable multiple sync options.

^* Only available on database-backed record types. To manage which data is synced in a service-backed record type, modify the record data source expression to filter out certain data.

Scheduled syncs

If your data is changed by external systems or even directly on your source, you can schedule syncs to automatically capture those changes in Appian.

You can schedule one or both of the following:

• Incremental syncs, which will sync new or changed data every few minutes or hours.
• Full syncs, which will sync all source data once each day or week.

Schedule incremental syncs

Applies to: Database-backed record types (tables only, not views) and service-backed record types.

If data is frequently created or modified by external systems, you can schedule incremental syncs to occur regularly throughout the day. An incremental sync captures any new or changed data since the last successful sync. Incremental syncs do not capture hard deletes. If data is hard deleted in your source, you must use a full sync to remove deleted data from the record type.

You control which data is synced and how frequently it’s sync using:

• Last modified field: A field of type Date and Time that indicates when data was last updated. Appian will use the value of this field to determine the latest data to sync.

• Sync schedule: The frequency in which Appian syncs the new or changed data from the source. The ideal frequency for your incremental syncs depends on how often data changes in the source system and when you need those changes in your apps.

  For example, if support cases are constantly created in the source system throughout the day, you might sync those cases every 15 minutes. Alternatively, if orders are periodically created in your source system and you'd rather capture all new orders in bulk, you could sync new orders every 4 hours.

To schedule incremental syncs:

1. (Service-backed record types only) Configure a changed data expression:
   • Create an integration to return changed data.
   • Create an expression rule to call the integration.
2. Select a Last Modified field and configure the incremental sync schedule.

Create an integration to return changed data

To schedule incremental syncs on a service-backed record type, you must create an integration object that returns only new or changed data from your source.

The integration must have the following rule inputs:

You can use the lastSuccessfulSync rule input to easily identify new or changed data to sync in Appian.

For example, an API with a field called modifiedOn is used to track when a row of data is updated. In your API, you can compare the date and time of the modifiedOn field with the date and time of the last successful sync.

Then, you can indicate that any rows with a date and time greater than or equal to the last successful sync should be synced in Appian during the next incremental sync. In the following table, this would mean that rows 3 and 5 will be captured in the next incremental sync.

Create a changed data expression

Once you create an integration to return changed data from your external system, create a changed data expression to call that integration and return the new or changed data to sync.

The changed data expression must have the following configurations:

For example, a changed data expression could look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
 a!localVariables(

  /* Call the integration that returns changed data */
  /* and store the response in a local variable */
  local!integrationResponse: rule!RE_getNewOrChangedOrders(

    /* If there are more than 1,000 rows to sync, execute the expression multiple times */
    /* and pass sequential values {1, 2, 3...} for batchNumber until an empty set is returned */
    batchNumber: ri!batchNumber,

 /* Passes the required rule input to the integration to provide */
 /* the date and time of the last successful sync */
    lastSuccessfulSync: ri!lastSuccessfulSync
  ),

  if(
    /* This will return true if the integration succeeds */
    local!integrationResponse.success,
    local!integrationResponse.result.body,

    /* If the integration is unsuccessful, handle accordingly */

    if(
      tointeger(index(local!integrationResponse.result, "statusCode", 0)) = 416,
      {},
      local!integrationResponse.error
    )
  )
 )

Set the incremental sync schedule

An incremental sync can sync up to 50,000 rows at a time. As a best practice, scheduled incremental syncs should capture small portions of data throughout the day.

To schedule incremental syncs:

1. In your record type, go to Sync Options.
2. Remove any sync filters configured to filter by a relationship. You cannot have both a scheduled incremental sync and relationship-based sync filter configured.
3. Select the Schedule incremental syncs checkbox.

4. (Service-backed record type only) Select a changed data expression to identify the new or changed data to sync.

   Tip:  If you don't have a changed data expression, you can generate the integration and changed data expression to get started.

5. For Last Modified Field, select a field of type Date and Time that indicates when a row of data was last changed. This field is also used to determine what data to sync when multiple syncs occur at the same time.
6. For Sync changes every, use the dropdown to determine how often the sync should occur.
7. For Start syncs at, select the time when the first incremental sync should occur each day.
8. Click SAVE.

The first incremental sync of the day will start at the specified time. If the start time has already passed when you're configuring the incremental sync, then the syncs will start the next day at the specified time.

Schedule full syncs

Applies to: All synced record types.

If data is changed less frequently by external systems, you can schedule a full sync to occur once a day or once a week.

A full sync will purge your existing synced data and replace it with the latest data available in your source. This ensures that any data created, updated, or deleted in the source is reflected in the record type.

For faster syncs, Appian recommends scheduling full syncs outside of peak traffic or working hours. When working with multiple synced record types, consider scheduling each full sync to occur at a different time to have optimal sync performance.

To schedule a full sync:

1. In your record type, go to Sync Options.
2. Select the Schedule full sync checkbox.
3. For Repeat, choose how frequently the sync should occur:
   • Select Daily to sync all data once a day. Then, select the Time when the sync should occur.
   • Select Weekly to sync all data once a week. Then, select the Day and Time when the sync should occur.

4. Select the time and the timezone when you want the sync to occur each day.

5. (Optional) For service-backed record types, add a rate limit to prevent too many requests to the source at once.
6. Click SAVE.

Smart service syncs

In addition to scheduling syncs, you can leverage different smart services to automatically sync new or changed data from your process models, interfaces, or expressions. Syncs started by these smart services are called smart service syncs.

You can use the following smart services to sync data:

• Smart services that automatically sync data, which sync data created, changed, or deleted in Appian.
• The Sync Records smart service, which sync data created or changed by other smart services or systems.

Use smart services that automatically sync data

Applies to: Database-backed and Salesforce-backed record types.

It’s easy to keep your data fresh when you use Appian smart services to create, update, or delete enterprise data.

When you use the following smart services in your process models, interfaces, or expression rules, Appian will automatically sync those changes so they’re immediately available in your record type.

Considerations for updating data

• Smart services that do not automatically sync data: Appian will not automatically sync updates made using other smart services, plug-ins, or external systems.

  For example, the following services and tools will not automatically trigger a sync:

  • Query Database smart service.
  • Execute Stored Procedure smart service.
  • Call an integration smart service to update data in any other web service besides Salesforce.
  • Manual edits using tools like phpMyAdmin or MySQL workbench.

  To sync these types of data changes, schedule full or incremental syncs or use the Sync Records smart service.

• High rate of writes: When working with large datasets, you may create multiple record types that point to the same database table, and use source filters to sync different portions of data in each record type. For apps requiring a high rate of writes, use this pattern to sync data independently from the writes to the source instead of using smart services that automatically sync data.

• Smart service functions in portal interfaces: A sync will not occur when you use these smart service functions directly in a portal interface. To sync data changed by a portal, use a!startProcess() to trigger a process containing a smart service that automatically syncs data. Learn more about working with data in a portal.

• Writing to Salesforce: If the Salesforce integration fails to write to the source, then a sync will not occur. Likewise, if Appian does not receive a notification that the write to the source was successful, the sync will not occur.

• Writing to a database using a data store entity: To use any of the smart services using a data store entity, you will need to create a custom data type (CDT) to write to the source. Depending on how your table data is structured in the CDT, the maximum number of rows that can be synced will vary:

  • If you're using a flat CDT, you can sync up to 50,000 rows at a time.
  • If you're using a nested CDT, you can sync up to 1,000 rows at a time.

  If you update more than the maximum number of rows, the smart service will successfully write to the data source, but the sync will fail.

  Additionally, your CDT should not contain any fields that accept an array of values. If you write data to a field that accepts an array of values, those changes cannot be synced.

Use the Sync Records smart service

Applies to: All synced record types.

If data is changed by smart services that don't automatically sync data, or data is changed in the source and you can easily access the identifiers of that new data, you can use the Sync Records smart service to sync your data changes.

The Sync Records smart service allows you to specify which data to sync from the source using their unique identifiers. There are a few ways to sync your data using this smart service:

• Generate a web API that includes the a!syncRecords() function and add the API's URL to your source system's webhook so it can notify Appian of changes and sync them.
  
  
• Use the a!syncRecords() function in an interface so end users can choose to sync data whenever they submit a form or click a button.
  
  
• Use the Sync Records smart service in a process model to trigger a sync after a call to an integration or stored procedure executes, or to start a sync using a record action.

Depending on your business needs, the way you choose to use this smart service or function may vary. See the Sync Records smart service page for examples.

Configure a sync by identifier expression

To use the Sync Records smart service to sync data in a service-backed record type, you must configure a sync by identifier expression on your record type.

To configure a sync by identifier expression:

1. Create an expression rule that returns a list of identifiers for the specific rows of data you want to sync.

   The expression rule must have the following configuration:

   Elements of the Expression	Required Configurations
   Integration response	The expression calls an integration that returns the identifiers of new or changed data from the source. The response body must cast to a List of Map, List of Dictionary, or a List of CDT. There must be at least one Number (Integer) field that's non-null for every entry to act as the primary key for the record type.
   Rule inputs	The expression must have a rule input of type Number (Integer). This allows the integration to return a list of identifiers. </li>

2. In the record type, go to Sync Options.
3. Under Configure sync by identifier expression, click + CONFIGURE EXPRESSION RULE.

4. In the dialog, enter the name of your expression rule and select it from the dropdown list.

   Tip:  If you do not have a sync by identifier expression yet, you can create one by clicking .

5. Click OK.
6. Click SAVE.

Modify the sync by identifier expression rule

While the sync by identifier expression only needs to return the identifiers for the data you want to sync, you may consider modifying the expression if you want to:

• Change the default local!deleteMissingRecords behavior so that the sync fails when any requested records are not returned by the integration.
• Add more error handling and customized messages.
• Format the data returned by your integration.

Manual full syncs

Applies to: All synced record types.

If you're testing your record type configurations in a development environment, or deploying the record type to a new environment, you may need to start a manually sync all source data in the record type.

You can manually start a full sync by:

• Saving the record type.
• Clicking START FULL SYNC on the Sync Options page of the record type.
• Selecting one or more record types on the Records Sync Status tab of the Monitor view and clicking START FULL SYNC.
• Using an import customization file.

Multiple syncs at the same time

Since you can configure a combination of sync options on your record type, there may be instances when different sync types occur at the same time.

When multiple syncs try to change the same row of data at the same time, the following will occur:

Availability options

To ensure consistent and reliable access to your data—even during connectivity issues or when managing large data volumes—you can configure several availability options for your record types.

Configure these options to determine how Appian handles sync failures and data volume:

• Keep data available at high volumes.
• Skip failed full and incremental syncs.
• Skip failed smart service syncs.
• Recover failed syncs.
• Rate limit.

Keep data available at high volumes

Applies to: Database-backed record types in capability tiers with a synced row limit.

If your data source has more rows than your tier's synced row limit, you can enable Keep data available at high volumes to automatically manage which data is synced.

When enabled, Appian dynamically syncs the latest rows of data—up to your tier's synced row limit—during any full, incremental, or smart service sync. You choose a field to sort by, such as lastModified or the primary key field, and Appian sorts that field in descending order to determine which rows are most recent.

For example, say you have an Order record type with 20,000,002 rows in the source, but your tier's limit is 20 million. If you sort by a lastModified field, Appian syncs the 20 million most recently modified orders and excludes the 2 rows with the oldest lastModified values.

To enable the Keep data available at high volumes option:

1. In the record type, go to Sync Options.
2. Select the Keep data available at high volumes checkbox.
3. Select a field to sort by:

   • For record types with a single primary key field: Select the primary key field or a field of type Date or Date and Time.

   • For record types with a composite primary key: Select a field of type Date or Date and Time. You cannot use a composite primary key as the sort field.

   Tip:  To ensure the data continues to sync quickly, the selected field should have an index configured on your database table. If the field does not have an index, you can generate a database index for the sort field directly from the record type.

4. Click SAVE.

Generate a database index for the sort field

When configuring the Keep data available at high volumes sync option, the selected field to sort by must have an index configured on the database table. This will ensure your data continues to sync quickly.

If you select a field that does not have a database index, you can automatically generate a database index from the record type.

To generate a database index from the record type:

1. Select a field to sort by.

2. From the banner, click Generate an index on this field in your database.

   

   Scroll or drag to pan · ⌘+/⌘− to zoom · Esc to close

3. Click GENERATE. Appian will automatically create the index on the field in your database and generate an index script.
4. Save the generated index script for future deployment.

If you generate a database index from the record type, but the field is still identified as lacking an index, there may have been an issue publishing the index. This may be a result of generating an index with the same name as an existing index in your database table. If this is the case, rename the existing index on your table before attempting to generate a new one.

Sync failure behavior with keep data available at high volumes

In capability tiers with a synced row limit, enabling Keep data available at high volumes provides extra flexibility for syncs that add data beyond the row limit. If a smart service sync or incremental sync exceeds the row limit, the sync still succeeds as long as the total row count stays within 10% of your tier's limit. Appian automatically removes the oldest data based on the selected sort field to keep the record type under the limit.

However, a sync fails if the record type exceeds the 10% threshold when a smart service sync or incremental sync occurs.

Skip failed full and incremental syncs

Applies to: All synced record types.

By default, the Skip failed full and incremental syncs option is enabled on all new record types to help ensure your data remains available, even if a scheduled sync fails. With this option enabled, the record type will skip any scheduled full sync or scheduled incremental sync that fails and continue using the previously available data.

When these syncs are skipped, the record type will have all data except for the changes made outside of Appian, which will be captured during the next successful sync. Any full or incremental syncs that failed and were skipped are captured in the Sync History so you can investigate the underlying reason for the failure.

If your data is time sensitive and you do not want to skip failed syncs, consider disabling this option. If you disable this option, any failed full or incremental syncs will cause the record data to be unavailable, which will also cause any interfaces or queries that reference it to be unavailable.

To skip failed full syncs and incremental syncs:

1. In the record type, go to Sync Options.
2. Select the Skip failed full and incremental syncs checkbox.
3. Click SAVE.

Skip failed smart service syncs

Applies to: Database-backed record types.

By default, if a smart service sync fails, the record data will be unavailable. This will cause any interfaces or queries that reference the record type to also be unavailable. Failed smart service syncs don't occur often, but in the rare case they do, you can enable Skip failed smart service syncs to keep your record data available.

When smart service syncs are skipped, the record type will have all data except for the new changes made by Appian. To capture these missed changes, you must configure a scheduled sync or enable recovery syncs on the record type.

The first failed and skipped smart service sync will be captured in the Sync History. However, subsequent smart service syncs that fail and skip won't appear in the Sync History, so Appian recommends configuring a scheduled sync and enabling recovery syncs to avoid data inconsistencies.

You should only enable this option if potential data inconsistency won't interfere with your business requirements. When a smart service sync fails and is skipped, there are many inconsistencies that occur, including missing, outdated, or incorrect data.

For example, let's say you have a grant management application that tracks requests for funding from the government. Due to compliance requirements, it's essential that the state of each grant is accurate. If you enable this option on the Grant record type and a sync fails to update its status, the source system may view a grant as approved even if it still shows as pending in Appian.

To skip failed smart service syncs:

1. In the record type, go to Sync Options.
2. Enable one or more of the following options:
   • Schedule full syncs.
   • Schedule incremental syncs.
   • Recover failed syncs.
3. Select the Skip failed smart service syncs checkbox.
4. Click SAVE.

Recover failed syncs

Applies to: All synced record types.

It's easy to keep your data fresh when you schedule incremental syncs, or use smart services that automatically sync data in your workflows. However, there are scenarios when these syncs may fail due to issues connecting to the data source. When the sync fails, the record data becomes unavailable.

To quickly recover data in these scenarios, you can enable recovery syncs. When enabled, a recovery sync will start immediately after a smart service sync or a scheduled incremental sync fails due to connection issues. The recovery sync will fetch and re-sync all of your source data to ensure you have the latest changes as quickly as possible.

If the recovery sync fails, the record data will remain unavailable. Review troubleshooting data sync to resolve any potential issues.

To enable recovery syncs:

1. In the record type, go to Sync Options.
2. Select the Recover failed syncs checkbox.
3. Click SAVE.

Rate limit

Applies to: Service-backed record types.

When syncing data from a web service, Appian will make a series of requests to the web service until it returns all of the data. This can result in many requests in a short amount of time to the source system, and some systems may start to reject requests if too many arrive within a certain interval.

To prevent overloading the web service, you can configure a rate limit to determine the maximum frequency of requests during a full sync.

As a best practice, you should include a buffer in your rate limit. For example, if the web service allows up to 10 requests per second, it's safer to choose 5 requests per second for the sync rate limit to account for other simultaneous requests.

Additionally, the rate limit determines the maximum number of requests in the given time period. As a result, the actual rate could be slower if additional formatting or manipulation of the data is necessary after the request is made.

To add a rate limit:

1. In the record type, go to Sync Options.
2. Select the Rate limit checkbox.
3. Use the dropdown to choose the number of batch requests made to the source when a full sync occurs.
4. Click SAVE.

Feedback