Data Sync in Appian Records

This page explains what it means to enable data sync in your record type. This page also provides guidance on when to enable sync, and how to monitor and keep your synced data up-to-date.

If you encounter an error while enabling sync, see Troubleshooting Syncs.

What is data sync?

Data sync allows you to operate with the benefits of onsite data, while leaving your source data where it is. When you enable data sync, you are caching your source data in Appian. With a cache of your data, this means Appian will only have to execute queries from the cached data instead of the external source whenever you view or interact with the record data.

Think of data sync like a cache on your web browser. When you cache local data on your browser, your site content loads faster and improves the speed of your browsing. The same applies when you sync your source data—your record data will load faster for more performant queries and reports.

sync-flow

While your synced data allows for faster queries, your source data is still what backs the record type. This means that any updates or changes to your source data will need to be re-queried (or re-synced) in Appian.

To keep your synced data up-to-date, any changes to your source data written by Appian will be immediately synced and reflected in the record type. You can also schedule a daily sync to sync any updates written to your source by an external system once a day. Using immediate and daily syncs, you get the performance benefits of data sync while continuously working with fresh data.

In addition to faster queries and reports, data sync also unlocks additional functionality that can speed up your development time. Learn more about these sync-enabled features below so you can build your applications faster:

Benefits of data sync

Appian leverages the proximity to your cached data to provide additional features that are more powerful and flexible than we could automatically provide for external data. These features make it easier to change your data model, access the data you need, and even configure parts of your application automatically.

For example, additional data modeling capabilities like record type relationships and custom record fields provide a low-code alternative to manually building database views. Instead of creating database views to relate and transform data, you can use relationships to relate data from a variety of sources, and create custom record fields to perform data transformations—all without complex SQL.

The following features are only available when you sync your data in Appian.

Establish relationships between record types to easily reference related record data throughout your applications. Relationships can be established between record types with data sync enabled, which allows you to relate data from different source types, like Salesforce and a database table—going beyond traditional relationships you may establish in a database view.

Using relationships, you can easily build advanced records and reports that display data from a variety of sources. Simply reference the relationship in your interface or query to return related record data. Learn more about referencing related record data.

Custom record fields allow you to extend your data by transforming and manipulating existing record data into new custom record fields. Using predefined templates or expression mode, you can perform operations that calculate new values, group your data, manipulate text, and more, then display those values as new fields in the record type.

CF Range GIF

Not only does data sync make it faster and easier to configure your data model, it makes configuring the entire record type faster. When you enable data sync, Appian can automatically configure parts of your record type for you:

  • Auto-generated user filters: A user filter is automatically generated on the record type whenever you define a many-to-one relationship. The user filter will leverage related data to filter your list of records, and you can modify the user filter at any time.

  • Generate a Summary view interface: Generate a Summary view interface with all your record data. Once you generate the interface, the interface expression will automatically populate in the Summary view.

  • Generate record actions: Allow Appian to configure common record actions for you. Choose whether your record action should create, update, or delete records, then Appian will configure the record action using existing objects or creating new design objects for you.

  • Suggested relationships: Any time you add a relationship to a record type, the inverse relationship will be suggested on the related record type. For example, if the Customer record type has a one-to-many relationship with the Order record type, a Suggested Relationship will appear on the Order record type to add a many-to-one relationship with the Customer record type. With a click of a button, you can configure that relationship.

When to use data sync

There are some data structures and business scenarios that are better fit for data sync than others. Consider the following requirements before enabling data sync in your record type:

  • Row requirements:
    • You do not need to use more than 1,000,000 rows. Learn more about the row limit.
  • Field requirements:
    • Your source has one primary key field.
    • You do not need more than 100 fields, including custom record fields.
    • You do not need to sync text columns containing strings longer than 4,000 characters.
  • Writes to your source:
    • Your source data is only changed in ways that can be immediately synced, or it changes infrequently outside of Appian so it doesn't need to be synced more than once a day.
  • Database encryption:
    • You do not need to enable the database encryption capability offered by Appian Cloud relational database. This is an additional layer of protection that can be enabled on top of the out-of-the-box disk encryption provided by Appian Cloud.

      This requirement only applies to customers that are on Advanced or Enterprise Support.

Row limit for record types with sync enabled

When you enable data sync, a record type can sync up to 1,000,000 rows of data from your selected source. Data sources that exceed or are expected to exceed the row limit may not be a good fit for data sync if you need access to all rows. However, if you only need a portion of a data source that meets or exceeds the row limit, you can configure source filters to limit the number of rows synced in Appian.

Source filters allow you to determine which rows of data from the source are synced in Appian based on the conditions you specify. When a source filter is applied, only the data that meets the specified criteria will be available in the Appian system and visible to end users. You can add source filters when defining the source of the record type, or anytime after by navigating to the Data Model page.

Reaching the row limit

If your source is within 10% of the limit (900,000 to 1,000,000 rows), the sync will successfully complete but a warning will appear in the Status column on the Record Sync Status page to let you know the source is approaching or has reached the row limit. After the data source exceeds 1,000,000 rows, the record type will become unavailable. If the record type reaches the row limit after an immediate sync, the write operation will be successful, but the sync will fail and the record type will become unavailable.

If you expect the source data to exceed the row limit, consider applying multiple source filters. If your record type continues to exceed the row limit, data sync may not be the best option for your business scenario. See Troubleshooting Syncs for steps to disable data sync.

Exceeding the field character limit

Any field values with more than 4,000 characters will be truncated in Appian. If your source contains fields with more than 4,000 characters, the sync will still complete successfully and you can use the truncated data throughout your applications. However, it's not recommend to use fields with more than 4,000 characters since it will result in a mismatch between the synced data and the source data. To ignore fields that exceed the character limit, deselect them from the field list when configuring the source of the synced record type.

API limit to sync rows from Salesforce

When your record type uses a Salesforce object as the source, syncing 1,000 rows of data requires a single API call. If your record type has the maximum number of rows (1,000,000), the API limit on your Salesforce instance must be at least 1,000 per day. If you have multiple record types that use a Salesforce object with sync enabled, ensure your API limit is large enough to support all your record types.

For example, say you have five record types that total to 400,000 rows, and each record type has a schedule daily sync. To successfully sync each record type, you would need at least 120,000 API calls per month. If the API limit is not high enough, the sync will fail and the record type will be unavailable.

Enabling data sync

You can enable data sync when you configure the source of a record type to be a database, Salesforce, or other web service.

Appian recommends creating new record types with data sync enabled. If you enable data sync on an existing record type, new record fields with new UUIDs will be created. As a result, any existing references to your record fields will break, and you'll need to update them with the new record field references.

Instead, if you create a new record type with sync enabled, you can ensure that any existing record field references continue to work. Then, you can incrementally update references in your applications to point to your new record type with sync enabled. This way, you can continuously make changes to your record type without impacting your applications.

Enable data sync on new record types

To enable data sync on a new record type, see:

Enable data sync on existing record types

In some cases, you may decide that converting your existing record type to use sync is more efficient than creating a new record type. For example, if your existing record type isn't widely used, you won't need to update many references in your application if you convert your record type to use data sync.

To enable data sync on an existing record type:

  1. On your existing record type, go to the Source & Default Filters tab.
  2. From SOURCE, click the gear icon and select Change Source.
  3. Configure the source of your record type to be a database, Salesforce, or a web service with data sync enabled.

Once you change the source of your existing record type to use data sync, you will need to update all references to the original record fields throughout your applications. This includes references within the record type as well, such as record field references in the record list.

Update existing record field references

To find and update record field references, it's best to start by reviewing the record type's dependent objects. To review these objects, select the record type and click DEPENDENTS. Replace any existing record field references in these objects with the new record field references.

If you want to find specific field references in your environment:

  1. Find an instance of a reference that you need to replace.
  2. Comment it out by wrapping the reference in /* and */.
  3. Copy the string and go back to the APPLICATIONS View.
  4. In the header bar of the APPLICATIONS View, click OBJECTS.
  5. In the OBJECTS View, paste the string into the search box under DESIGN OBJECTS.

    You can search for multiple strings at once by separating each string with a space.

  6. Next to the search button, click the down arrow and select Expression.
  7. Click the search icon.

Every object that references the string will appear in your search results. Once you have located all the references in your objects, update the field references. For updating field references in interfaces, we recommend updating the references in Expression Mode.

If you disable data sync on a record type, you will also need to review the objects in your applications and update any record field references associated with the record type. Any sync-enabled features, like relationships and custom record fields, will also be removed and made unavailable. For steps, see Troubleshooting Sync.

Refreshing your synced data

There are two ways to keep your synced data up-to-date with the source:

You can also trigger manual syncs at any time. However, manual syncs should only be used in development environments.

To view detailed information about all syncs that occurred for a specific record type, see the Sync History.

Immediate sync

An immediate sync will occur anytime Appian writes to the database table or Salesforce object that backs your record type. This means that whenever data is inserted, updated, or deleted from the source using a smart service or Salesforce integration, the data in the associated record type is also automatically synced to reflect the changes.

An immediate sync will only refresh the data that was changed by Appian. If more than 1,000 rows of data are changed by Appian, the smart service or integration will successfully write to the data source, but the immediate sync will fail. To prevent an immediate sync from failing, it's recommended to apply a limit to the associated smart service or integration so it does not update more than 1,000 rows at a time.

If Appian is the only system writing to your database table or to Salesforce, immediate sync allows you to be confident that you are always working with fresh data.

Other than a Salesforce object, Appian does not support immediate syncs for record types that use other web services as the data source.

Immediate sync for record types that use a database table

In a process model, you can configure Data Service smart services to write updates to a database table that’s connected to a data store entity. Once the database table is updated, Appian will trigger a sync to update the data in any synced records that use the database table as the source.

Without any additional configuration, you can use the following smart services to update the source of your synced data and trigger an immediate sync:

To use Data Service smart services, you will need to create a custom data type (CDT). Regardless of how your table data is structured in the CDT, immediate sync will occur when any table data that backs the record type is updated. The CDT is only necessary to write to the source.

To configure immediate syncs for your synced record:

  1. In your process model, configure the Data Services smart service node you need to perform a specific write operation.
  2. Connect the process model to a related action configured on your synced record.

Appian will not sync updates to synced records when the source updates are made using other smart services, plugins, or external systems. For example, the following services and tools will not trigger an immediate sync:

Immediate sync for record types that use a Salesforce object

Data sync is particularly helpful when using an external system, like Salesforce, as the source of your record type. To keep your synced data up-to-date with the source data in Salesforce, Appian triggers an immediate sync whenever you use a Salesforce integration to write to the Salesforce connected system.

You can update your source data and trigger an immediate sync by calling the Salesforce integration in a process model, a related action configured on the record type, or a save on an interface.

If the Salesforce integration fails to write to the source, the immediate sync will not occur. Likewise, if Appian does not receive a notification that the write to the source was successful, the sync will not occur.

Daily scheduled sync

After you enable data sync, it's recommended that you create a scheduled sync so that your data outside of Appian will automatically sync with your record type once a day. For quicker syncs, record types using Salesforce or other web services should have syncs scheduled outside of peak traffic or working hours.

To schedule a sync:

  1. In your record type with sync enabled, go to Data Sync.
  2. Under Sync Schedule, click Edit.
  3. Toggle Scheduled Sync to On.
  4. Under Sync Time, select the time and the timezone for your scheduled sync.
  5. Click OK.

You can edit your sync schedule here or on the Data Model page by opening the page's properties menu and selecting Change Sync Schedule.

Manual sync

Manual syncs allow you to sync your external data with the record type whenever you need it. This is useful when testing your application data in development environments, and when restoring your environment from a backup.

To trigger a manual sync, use one of the following methods:

  • In the record type, click SAVE CHANGES.
  • In the record type, on the left pane go to Data Sync, click START SYNC.
  • On the Monitoring View, in the top bar go to RECORD SYNC STATUS, click START SYNC.

Whenever your record type is deployed to a new environment, a sync will automatically occur. If you deploy additional packages that don't include the record type, but impact the synced data, you'll need to trigger a manual sync.

To trigger a sync when the additional packages are deployed, add recordType.<UUID of a Record Type>.forceSync=true to an import customization file. You can reference this property multiple times to trigger a sync on different record types.

Source and record type mapping

To sync your source data in Appian, the source schema and the record type schema must match. The source schema includes the names, types, and uniqueness constraints of all the fields in your source. When you enable data sync for your record type and configure your record fields, Appian automatically creates a mapping between the source schema and the record type schema.

If a change is made to the source schema without making a corresponding change to the record type, the record type will fail to sync. For example, if you try to refresh your synced data using an immediate sync, the smart service will write to the data source, but the sync will not occur.

When a record type fails to sync, it becomes unavailable until you resolve the schema mismatch. In addition, any calls to a!queryRecordType() for the synced record will fail.

For information on resolving a schema mismatch, see Troubleshooting Sync.

Monitor your data syncs

To ensure that your data is syncing successfully, you can check the status and history of all syncs associated with your record type.

Record sync status

In the Monitoring View of Appian Designer, the Record Sync Status tab displays the status of the most recent sync for all record types with sync enabled that you have access to view. This tab contains a grid that displays the record types with their source type, sync status, and the time of the last sync.

From this tab, you can click the link on the record type to view its Sync History, and access any errors or warnings that arise during the course of a sync. Learn more about the Record Sync Status tab.

Sync history

You can also view detailed information about the syncs that occurred for a specific record type. From either the Monitoring View of Appian Designer, or from the Data Sync tab of your selected record type, you can view a record type’s Sync History.

The record type’s sync history will display information on the past and currently running syncs, including the sync status, the times of the sync, and any errors that may have occurred.

A record type can have the following sync statuses:

Icon Status Meaning
Running A manual or scheduled sync is taking place, the record type is saving, or the record type is being imported to the environment.
Completed A manual or scheduled sync, record type save, or record type import has successfully completed.
Failed An error occurred while attempting to sync the record type and the record type is unavailable. For information on solving errors with your sync, see Troubleshooting Syncs.
Failed and Skipped An error occurred while attempting to sync the record type, but the record type is still available. Since this sync was skipped, queries will return data from the most recent successful sync. For information on solving errors with your sync, see Troubleshooting Syncs.
Failed and Retrying The sync initially failed and the system will retry a few times before failing. After failing, the record type will be unavailable. For information on solving errors with your sync, see Troubleshooting Syncs.
Approaching Limit The record type was able to sync but it's approaching the row limit.
Limit Reached The record type was able to sync but the row limit has been reached.

When a sync fails, the record type will either have a status of Failed or Failed and Skipped. You can click on these sync statuses to open the Sync Alerts dialog and view more information about what went wrong with the sync. System administrators and users with Administrator permission to the record type will also receive email alerts whenever a sync fails.

For more information on the different types of sync errors and steps to resolve these errors, see Troubleshooting Syncs.

Skip failed syncs

When a sync fails on a record type, all references to the record type will be temporarily unavailable. This will cause any interfaces, queries, or reports that reference your synced record type to also be temporarily unavailable. To give you the time and flexibility to troubleshoot sync failures, you can enable Skip Failed syncs.

When this setting is enabled, any failed manual syncs, scheduled syncs, or syncs from an import customization file will be skipped, so the record type will use data from the last successful sync. Since the record type is using data from the last successful sync, all references to the record type will remain available. By default, this setting is enabled on all new record types with data sync enabled.

To skip failed syncs on existing record types:

  1. On the record type, go to Data Sync.
  2. Under Skip Failed Sync, click Edit.
  3. In the Skip Failed Syncs dialog, choose On.

    /skip failed syncs

  4. Click OK.
  5. Click SAVE CHANGES.

When you have this setting enabled, it is still possible for a sync to fail and cause references to the record type to be temporarily unavailable. This can occur when the sync fails during an immediate sync, a sync caused by a record type save, or a sync caused by a deployment.

Open in Github Built: Mon, Nov 29, 2021 (03:22:32 PM)

On This Page

FEEDBACK