About Data Sync

Overview

Record types allow you to seamlessly connect to your enterprise data so you can build data-rich applications with ease. When you enable data sync on your record types, you can build powerful applications even faster.

This page explains the benefits of enabling data sync on your record types and provides tools and strategies you can use to leverage data sync, regardless of the size or location of your data.

What is data sync?

Data sync unlocks powerful features that can speed up your entire development process. These features make it easy to change your data model, access the data you need, and even allow Appian to configure parts of your applications for you.

See the table below for a full list of features that are only available when you enable data sync:

Feature	Description
Record type relationships	Establish relationships between record types to easily reference related record data throughout your applications. Build relationships between data from different sources, and easily create comprehensive reports and queries using more of your enterprise data.
Custom record fields	Extend your data by transforming and manipulating existing record data into new custom record fields. Using predefined templates or expression mode, you can perform operations that calculate new values, group your data, manipulate text, and more. These values are then saved as new fields in the record type.
No-code security configurations	Specify, in plain language, who can view which records, views, and actions on a record type. Appian will automatically enforce your security configuration anywhere the record type is used in your app.
Record events	Seamlessly track what happens in your applications and when, with minimal configuration on your part. You can use this feature to provide application users with a timeline of events, enable collaboration amongst team members, and improve your business processes using process insights.
Auto-generate elements of the record type	Automatically generate and configure core elements of the record type, like user filters, record views, and record actions.
Generate sample data	Automatically create realistic record data for existing record types and their related record types all at once.
Data fabric insights	Allow business users to build their own custom reports and dashboards using your synced record data in Process HQ.
Automatic performance tuning	Synced data is automatically optimized so your applications are overall more performant---No manual performance tuning necessary.

How does data sync work?

Data sync caches your source data in Appian. Think of it like a cache on your web browser: when you cache web content locally on your browser, your site content loads faster and improves the speed of your browsing. The same applies when you sync your source data—Appian only has to execute queries against your synced data instead of the external source, so your record data will load faster. In addition to faster queries, Appian also leverages the proximity to your synced data to provide the powerful features listed above.

As you build applications, you can keep your synced data up to date with the source by using smart services that automatically sync data in your process models, interfaces, and expression rules.

For example, let’s say you have a Customer record type that uses a database table as the source. If you use the Write Records smart service to add new customers, then that new customer data will also be synced and immediately reflected in the record type.

If there are other systems besides Appian that are writing to your source, you can also schedule a full sync or configure the Sync Records smart service to capture those changes.

If a smart service sync occurs at the same time as a scheduled full sync, all changes from the smart service sync will be immediately synced and available. Then, when the full sync completes, all other changes from the source will also be available.

How is my synced data secured?

Appian applies the same object-level security to your data across the platform. Whether it is being displayed in an interface, updated in a process, or synced in Appian, any user’s ability to view or edit data is determined by the permissions structure of your application.

You retain complete control over your business data, even when you sync it in Appian. Users can only build with and view synced data through the Appian design objects in your applications; they cannot access it directly. You can even specify who can view your synced data by applying record-level security on your record type.

If, at any time, you want to make your currently synced data unavailable to all users, you only need to disable data sync on that record type.

When to enable data sync

As a best practice, you should enable data sync on all new record types. This allows you to take advantage of powerful sync-enabled features and work with performant, up-to-date data.

If you have existing record types that do not have data sync enabled (unsynced record types), you can create new synced record types to use in Process HQ and in new development.

Appian does not recommend enabling data sync on an existing record type. If you enable data sync on an existing record type, it will create new record fields with new UUIDs. As a result, you will need to update any existing record field references with new record field references. By creating new synced record types, there’s no application re-work required.

Usage considerations

Before you enable data sync on your record type, consider the following:

• Row considerations:
  • Each synced record type can sync up to 4 million rows from a source. Consider the different ways to sync data from large data sources.
• Field considerations:
  • A single primary key field is required.
  • Each synced record type can have up to 100 fields, including custom record fields.
  • Text columns containing strings longer than 4,000 characters will be truncated in Appian. Consider removing fields that exceed this character limit.
  • Database columns must be a supported data type. Review the list of unsupported database column types.
• Salesforce API considerations:
  • Syncing 1,000 rows of data requires a single API call, so ensure the API limit on your Salesforce instance is large enough to support the number of rows you need to sync. Consider using source filters to only sync the most relevant data in the record type.
• Additional database encryption:
  • You do not need to enable the database encryption capability offered by Appian Cloud. This is an additional layer of protection that can be enabled on top of the out-of-the-box disk encryption provided by Appian Cloud.

  Note:  This only applies to customers on Advanced or Enterprise Support.

Sync data from large data sources

Each record type with data sync enabled can sync up to 4 million rows of data from a source. While this is a significant amount of data, very large data sources or growing data sources may require you to adapt your data model so you can access all the data you need.

Depending on your business case, you can easily adapt your data model:

If…	Then…
You only need the most recent data from a large database table.	Enable Keep data available at high volumes on your record type. This sync option will dynamically sync the latest 4 million rows of data from your source.
You only need a portion of the data from a large data source.	Configure source filters to choose which data to sync in the record type. You can create multiple record types using the same data source and use source filters to sync different portions of data in each record type.
You need all data from the large data source.	Create an unsynced record type to store the entire dataset from a table, view, or web service. Create synced record types to store subsets of data to address specific scenarios in your applications Depending on your use case, you may need to create additional indexes, views, caches, or other tuning to optimize query performance in the source system. By syncing subsets of the data, you can take advantage of automatic performance tuning and sync-enabled features in targeted areas of your application.

For example, let's say your Event History record type has 3.8 million rows of data and captures numerous events each day. Since you only need to display a timeline of the latest events that occur in your application, you could enable the Keep data available at high volumes option to dynamically sync the latest 4 million events.

Alternatively, you may have a database table with 6 million rows of support cases and you need to create an application that monitors all open cases. In this example, you could create a record type called Open Cases and configure source filters to only sync cases that have a status of Open or In Progress.

You can also use a combination of synced and unsynced record types throughout your applications, depending on your application requirements. See an example here.

Field character limit

Any field values with more than 4,000 characters will be truncated in Appian. While you can use the truncated data throughout your applications, it's a best practice to remove fields that exceed this character limit.

Unique fields have a lower character limit than standard fields and cannot exceed 765 characters.

To remove fields that exceed the character limit:

1. In the record type, go to Data Model.
2. Click MAP RECORD FIELDS.

3. Deselect the checkbox next to the field that exceeds the field value character limit.

   In the following example, LONG_DESCRIPTION is a field that can contain more than 4,000 characters, so it is deselected and will not be included in the record type.

   

   

4. Click FINISH.
5. Click SAVE CHANGES.

API limit to sync rows from Salesforce

When your record type uses Salesforce as the source, syncing 1,000 rows of data requires a single API call. If your record type has the maximum number of rows (4 million), the API limit on your Salesforce instance must be at least 1,000 per day. If you have multiple record types that use a Salesforce object with data sync enabled, ensure your API limit is large enough to support all your record types.

For example, say you have five record types that total to 400,000 rows, and each record type has a scheduled full sync. To successfully sync each record type, you would need at least 120,000 API calls per month. If the API limit is not high enough, the sync will fail and the record data will be unavailable.

Consider applying source filters to only sync the most relevant data in your record type.

Writes to your source

Appian will automatically sync any data changed by select smart services. If you need to capture changes made by other smart services, or even other systems, you can use the Sync Records smart service to sync those specific rows of data.

Additionally, you should schedule a full sync to occur once each day. This will purge your existing synced data and replace it with the latest data from your source.

If a smart service sync occurs at the same time as a scheduled full sync, all changes from the smart service sync will be immediately synced and available. Then, when the full sync completes, all other changes from the source will also be available.

Unsupported database columns

There are some databases that have unique data types that are not supported in synced record types. In general, Appian does not support binary large object (BLOB), spatial, or XML types, or any user-defined types.

For binary types, you may still be able to manually map a field in your record type to that column, but it is not guaranteed to work correctly and may fail completely with a future version of Appian.

The following table provides a reference of unsupported data types and may not be comprehensive. Vendor-specific and user-defined types are generally not supported and may cause record types to not sync or other issues.

Database Vendor	Unsupported types
IBM DB2	BINARY BLOB CLOB DBCLOB GRAPHIC
Maria DB and MySQL	BLOB BINARY LONGBLOB MEDIUMBLOB MULTILINESTRING MULTIPOINT MULTIPOLYGON POINT POLYGON TINYBLOB VARBINARY YEAR
Microsoft SQL Server	binary datetimeoffset geography geometry hierarchyid image rowversion sql_variant varbinary
Oracle	BFILE BINARYDOUBLE BINARYFLOAT BLOB CLOB INTERVAL_DAY_TOSECOND INTERVAL_YEAR_TOMONTH NCLOB ROWID TIMESTAMP_WITH_LOCALTZ TIMESTAMP_WITH_TZ UROWID VARBINARY
PostgreSQL	BIT BIT_VARYING BOX BYTEA CIDR CIRCLE INET INTERVAL JSON JSONB LINE LSEG MACADDR MACADDR8 MONEY PATH PG_LSN PG_SNAPSHOT POINT POLYGON TSQUERY TSVECTOR TXID_SNAPSHOT UUID XML

Feedback