Data Service

Appian’s data service is a custom-built distributed data system which is optimized to handle both transactional and analytical workloads without any manual database tuning or administration required. The data service guarantees ACID-compliant transactions and is designed for high availability and resiliency.

Data for record types with data sync enabled is stored in the data service. Through Appian’s data sync capabilities, the data service simplifies application development and maintenance by eliminating many of the complex data management and database administration tasks that are required to optimize performance for various operations and usage patterns.

Additionally, the data service can dynamically scale up and down in response to changes in user and process activity. This dynamic scaling ensures consistent query performance, optimizes resource efficiency, and further reduces the need for manual scaling.

Architecture

The data service consists of the following:

• A historical store component for writing data.
• Multiple real-time store components for querying data.
• A bulk ingestion component for efficiently processing full syncs.
• An appender component for performing background operations.
• A data client that runs in the application server and provides an interface for internal requests to the data service.
• A watchdog component that monitors the health of all components and recovers a component in the event of an isolated failure.

Historical store

The historical store (or hs) component executes write requests to the data service. After validating a write request, the historical store commits the transaction and distributes the data to other data service components by forwarding the request to the Internal Messaging Service, which serves as a transaction log for the data service. When restarting the data service for scheduled maintenance or as a result of system failure, transactions from the transaction log are replayed ensuring data consistency and optimizing recovery.

Real-time store

Each real-time store (or rts) component processes and executes query requests to the data service. When query requests are made from the application server, they are load-balanced across the real-time stores.

For Appian Cloud, the default number of real-time stores varies based on the amount of memory available on each node as shown below.

When you enable dynamic scaling for the data service, the real-time store count represents the number of real-time stores that will be statically provisioned on each node for your site. As a best practice, sites should run at least four real-time store components per node. Learn more about how dynamic scaling for the data service can maximize site resource efficiency.

Available Memory	Default RTS Count
Less than 32GB	2
Between 32GB - 127GB	4
Between 128GB - 384GB	8
Greater than or equal to 384GB	12

Bulk ingestion

The bulk ingestion component is used to efficiently perform full syncs into the data service.

The bulk ingestion component bypasses the internal messaging service and all unnecessary data validations. By streamlining full syncs, the data service is able to rapidly ingest data without any significant resource contention across other components, ensuring consistent query and write performance.

Appender

The appender component performs background operations designed to automatically optimize queries and limit unnecessary resource usage in the data service. This includes periodically flushing data from memory to disk, generating new data snapshots, and performing garbage collection.

Data client

The data client runs in the application server. It provides a simple interface for the rest of Appian to communicate with the data service.

Watchdog

The watchdog component runs a Java process that monitors the health of each data service component. In the event of an isolated failure of any other component, the watchdog will attempt to heal the failed component. If the watchdog component itself fails, it will self-heal.

Dynamic scaling for queries

For Appian Cloud environments, you can choose to automatically scale the data service up and down to match query throughput demand during peak processing times. With dynamic scaling, you can optimize resource efficiency during periods of low usage and eliminate the need to manually size the data service to account for peak data fabric usage.

How does dynamic scaling work?

The data service uses Kubernetes to automatically provision and remove data service pods, providing a serverless experience for data fabric queries on Appian Cloud sites.

Every Appian Cloud site has one or more site nodes, and each node has one data service pod that is always available. With dynamic scaling enabled, when your data fabric query usage exceeds the capacity of your site nodes, Appian automatically scales data service pods up and down as needed to meet demand. These dynamically provisioned pods are deployed to Appian Cloud-managed Kubernetes clusters, and do not use compute or memory resources from your site nodes.

Each dynamically scaled pod includes four real-time store components and one appender component.

Usage considerations

Scaling triggers

The data service utilizes scaling triggers to initiate scale up in response to increased process and user activity.

There are a few configurable triggers, which are optimized for query performance and resource efficiency:

• Real-time store utilization trigger: The data service proactively triggers scale up when real-time store component utilization exceeds 50%.
• Query routing wait time trigger: The data service reactively triggers scale up when query queue time exceeds 100ms.
• Autoscaled process activity trigger: The data service scales to accommodate high throughput workloads in autoscaled processes by scaling up when process executors are scaled up.

Maximizing site resource efficiency

When running with dynamic scaling for the data service enabled, the number of real-time stores deployed to your site nodes can be adjusted to maximize resource efficiency.

Real-time stores deployed on your site nodes will not be dynamically removed, so they are available to serve queries at all times. Real-time stores that are dynamically provisioned will be automatically deployed to new Appian Cloud nodes and will not increase the amount of resources utilized within your site nodes.

It's recommended that sites run with at least four real-time store components per node to avoid impacting the availability of resources as process and user activity ramps up.

Usage limitations

By default, the data service is configured to dynamically provision a maximum of 10 additional data service pods.

Enable dynamic scaling

Open a support case to request changes to your data service configurations.

Feedback