About Data Sync

Overview

Copy link to clipboard

Record types allow you to seamlessly connect to your enterprise data so you can build data-rich applications with ease. When you enable data sync on your record types, you can build powerful applications even faster.

This page explains the benefits of enabling data sync on your record types and provides tools and strategies you can use to leverage data sync, regardless of the size or location of your data.

What is data sync?

Copy link to clipboard

Data sync unlocks powerful features that can speed up your entire development process. These features make it easy to change your data model, access the data you need, and even allow Appian to configure parts of your applications for you.

See the table below for a full list of features that are only available when you enable data sync:

How does data sync work?

Copy link to clipboard

Data sync caches your source data in Appian. Think of it like a cache on your web browser: when you cache web content locally on your browser, your site content loads faster and improves the speed of your browsing. The same applies when you sync your source data—Appian only has to execute queries against your synced data instead of the external source, so your record data will load faster. In addition to faster queries, Appian also leverages the proximity to your synced data to provide the powerful features listed above.

As you build applications, you can keep your synced data up to date with the source by using smart services that automatically sync data in your process models, interfaces, and expression rules.

For example, let’s say you have a Customer record type that uses a database table as the source. If you use the Write Records smart service to add new customers, then that new customer data will also be synced and immediately reflected in the record type.

If there are other systems besides Appian that are writing to your source, you can also schedule a full sync or configure the Sync Records smart service to capture those changes.

If a smart service sync occurs at the same time as a scheduled full sync, all changes from the smart service sync will be immediately synced and available. Then, when the full sync completes, all other changes from the source will also be available.

How is my synced data secured?

Copy link to clipboard

Appian applies the same object-level security to your data across the platform. Whether it is being displayed in an interface, updated in a process, or synced in Appian, any user’s ability to view or edit data is determined by the permissions structure of your application.

You retain complete control over your business data, even when you sync it in Appian. Users can only build with and view synced data through the Appian design objects in your applications; they cannot access it directly. You can even specify who can view your synced data by applying record-level security on your record type.

If, at any time, you want to make your currently synced data unavailable to all users, you only need to disable data sync on that record type.

When to enable data sync

Copy link to clipboard

As a best practice, you should enable data sync on all new record types. This allows you to take advantage of powerful sync-enabled features and work with performant, up-to-date data.

If you have existing record types that do not have data sync enabled (unsynced record types), you can create new synced record types to use in Process HQ and in new development.

Appian does not recommend enabling data sync on an existing record type. If you enable data sync on an existing record type, it will create new record fields with new UUIDs. As a result, you will need to update any existing record field references with new record field references. By creating new synced record types, there’s no application re-work required.

Usage considerations

Copy link to clipboard

Before you enable data sync on your record type, consider the following:

• Row considerations:
  • Each record type with data sync enabled can sync up to 10 million rows from a source. If you’re working with an extra large data source or a growing data source, you may need to adapt your data model to access all the data you need.
• Field considerations:
  • A single primary key field is required.
  • Each synced record type can have up to 100 fields, including custom record fields.
  • Unique fields can have a maximum of 765 characters.
  • Text fields containing values longer than 4,000 characters will be truncated in Appian. Instead, set fields that store large amounts of data to type Extra Long Text, so you can store values up to 64,000 characters. Learn more.
  • Database columns must be a supported data type. Review the list of unsupported database column types.
• Salesforce API considerations:
  • Syncing 1,000 rows of data requires a single API call, so ensure the API limit on your Salesforce instance is large enough to support the number of rows you need to sync. Consider using sync filters to only sync the most relevant data in the record type.
• Write considerations:
  • Any data changed by select smart services will be automatically synced in Appian. If you're writing to a database table that is the source of multiple record types, consider syncing data independently from the writes to the source.

  • If you use other smart services or systems to write to your source, you can use the Sync Records smart service or a scheduled full sync to capture those changes.

  • If a smart service sync occurs at the same time as a scheduled full sync, all changes from the smart service sync will be immediately synced and available. Then, when the full sync completes, all other changes from the source will also be available.
• Additional database encryption:
  • You do not need to enable the database encryption capability offered by Appian Cloud. This is an additional layer of protection that can be enabled on top of the out-of-the-box disk encryption provided by Appian Cloud.

  Note:  This only applies to customers on Professional or Signature Success Plans.

Sync data from large data sources

Copy link to clipboard

Each record type with data sync enabled can sync up to 10 million rows of data from a source. While this is a significant amount of data, very large data sources or growing data sources may require you to adapt your data model so you can access all the data you need.

Depending on your business case, you can easily adapt your data model:

For example, let's say your Event History record type has 9.6 million rows of data and captures numerous events each day. Since you only need to display a timeline of the latest events that occur in your application, you could enable the Keep data available at high volumes option to dynamically sync the latest events.

Alternatively, you may have a database table with 7 million rows of support cases and you need to create an application that monitors all open cases. In this example, you could create a record type called Open Cases and configure sync filters to only sync cases that have a status of Open or In Progress.

You can also use a combination of synced and unsynced record types throughout your applications, depending on your application requirements. See an example here.

Use Extra Long Text fields to store large volumes of data

Copy link to clipboard

Text values with more than 4,000 characters will be truncated in Appian. While you can use the truncated data in your apps, it’s best to configure fields with large volumes of data as type Extra Long Text.

Extra Long Text fields can store up to 64,000 characters. This allows you to store and display unstructured data, like employee’s yearly performance reviews, in an interface. Extra Long Text fields can be queried in a!queryRecordType() or a!queryRecordByIdentifier(); however, to ensure optimal performance, you cannot use these fields in filters, sorts, aggregations, or records-powered components.

You can configure a field to be type Extra Long Text when you choose a data source or generate a database table. You can also change an existing Text field to type Extra Long Text at any point in development.

To change a Text field to type Extra Long Text:

1. In the record type, go to Data Model.
2. Click MAP RECORD FIELDS.

3. For Record Field Type, use the dropdown to select Extra Long Text.

   Tip:  You should only have a maximum of three Extra Long Text fields on a record type for optimal performance.

   

   

4. Click FINISH.
5. Click SAVE CHANGES.

API limit to sync rows from Salesforce

Copy link to clipboard

When your record type uses Salesforce as the source, syncing 1,000 rows of data requires a single API call. For example, if your record type has 10 million rows, the API limit on your Salesforce instance must be at least 10,000 per day.

If you have multiple record types that use a Salesforce object with data sync enabled, ensure your API limit is large enough to support all your record types. For example, say you have five record types that total to 400,000 rows, and each record type has a scheduled full sync. To successfully sync each record type, you would need at least 120,000 API calls per month.

If the API limit is not high enough, the sync will fail and the record data will be unavailable.

Consider applying sync filters to only sync the most relevant data in your record type.

Syncing changes from the same database table in multiple record types

Copy link to clipboard

When working with large datasets, you may create multiple record types that point to the same database table, and use source filters to sync different portions of data in each record type.

Whenever you use a smart service that automatically syncs data to update data in that database table, Appian will start a sync on each of the record types to capture the changes. After configuring multiple record types that sync to the same database table, you can expect a proportional increase in sync requests.

For applications requiring a high rate of writes, you can use this pattern to sync data independently from the writes to the source.

Unsupported database columns

Copy link to clipboard

There are some databases that have unique data types that are not supported in synced record types. In general, Appian does not support binary large object (BLOB), spatial, or XML types, or any user-defined types.

For binary types, you may still be able to manually map a field in your record type to that column, but it is not guaranteed to work correctly and may fail completely with a future version of Appian.

The following table provides a reference of unsupported data types and may not be comprehensive. Vendor-specific and user-defined types are generally not supported and may cause record types to not sync or other issues.

Feedback

Copy link to clipboard