Data Service

What is the data service?

Appian’s data service is a custom-built distributed data system which is optimized to handle both transactional and analytical workloads without any manual database tuning or administration required. The data service guarantees ACID-compliant transactions and is designed for high availability and resiliency.

The data service supports the following functionality:

• Stores and optimizes synced data: Stores data for synced record types and automatically optimizes it. This eliminates many of the complex data management and database administration tasks required to manually optimize performance for various operations and usage patterns.

• Powers federated queries: Allows you to query across both synced and unsynced record types. This enables you to apply platform features like custom record fields, record-level security, and record relationships across your entire data landscape.

• Scales dynamically: Dynamically adjusts resources up and down in response to changes in user and process activity. This ensures consistent query performance and optimizes resource efficiency without manual intervention.

Architecture

The data service consists of the following:

• A historical store component for writing data.
• Multiple real-time store components for querying data.
• A bulk ingestion component for efficiently processing full syncs.
• An appender component for performing background operations.
• A data client that runs in the application server and provides an interface for internal requests to the data service.
• A watchdog component that monitors the health of all components and recovers a component in the event of an isolated failure.

Historical store

The historical store (or hs) component executes write requests to the data service. After validating a write request, the historical store commits the transaction and distributes the data to other data service components by forwarding the request to the Internal Messaging Service, which serves as a transaction log for the data service. When restarting the data service for scheduled maintenance or as a result of system failure, transactions from the transaction log are replayed ensuring data consistency and optimizing recovery.

Real-time store

Each real-time store (or rts) component processes and executes query requests to the data service. When query requests are made from the application server, they are load-balanced across the real-time stores.

When you enable dynamic scaling for queries, the real-time store count represents the number of real-time stores that will be statically provisioned in your environment. With dynamic scaling for queries, it's recommended that you run at least four real-time store components to maximize resource efficiency.

For Appian Cloud, the default number of real-time stores varies based on the amount of memory available on each node:

Available Memory	Default RTS Count
Less than 32GB	2
Between 32GB - 127GB	4
Between 128GB - 384GB	8
Greater than or equal to 384GB	12

Bulk ingestion

The bulk ingestion component is used to efficiently perform full syncs into the data service.

The bulk ingestion component bypasses the internal messaging service and all unnecessary data validations. By streamlining full syncs, the data service is able to rapidly ingest data without any significant resource contention across other components, ensuring consistent query and write performance.

Appender

The appender component performs background operations designed to automatically optimize queries and limit unnecessary resource usage in the data service. This includes periodically flushing data from memory to disk, generating new data snapshots, and performing garbage collection.

Data client

The data client runs in the application server. It provides a simple interface for the rest of Appian to communicate with the data service.

Watchdog

The watchdog component runs a Java process that monitors the health of each data service component. In the event of an isolated failure of any other component, the watchdog will attempt to heal the failed component. If the watchdog component itself fails, it will self-heal.

Dynamic scaling for queries

For Appian Cloud environments, you can choose to automatically scale the data service up and down to match query throughput demand during peak processing times. With dynamic scaling, you can optimize resource efficiency during periods of low usage and eliminate the need to manually size the data service to account for peak data fabric usage.

How does dynamic scaling work?

The data service uses Kubernetes to automatically provision and remove data service pods, providing a serverless experience for data fabric queries on Appian Cloud sites.

Every Appian Cloud site has one or more site nodes, and each node has one data service pod that is always available. With dynamic scaling enabled, when your data fabric query usage exceeds the capacity of your site nodes, Appian automatically scales data service pods up and down as needed to meet demand. These dynamically provisioned pods are deployed to Appian Cloud-managed Kubernetes clusters, and do not use compute or memory resources from your site nodes.

Each dynamically scaled pod includes four real-time store components and one appender component.

Scaling triggers

The data service uses scaling triggers to initiate scale up in response to increased process and user activity.

There are three scaling triggers, which are optimized for query performance and resource efficiency:

• Real-time store utilization trigger: The data service proactively triggers scale up when real-time store component utilization exceeds 50%.
• Query routing wait time trigger: The data service reactively triggers scale up when query queue time exceeds 100ms.
• Autoscaled process activity trigger: The data service scales to accommodate high throughput workloads in autoscaled processes by scaling up when process executors are scaled up.

Maximizing resource efficiency

When dynamic scaling for the data service is enabled, the number of real-time stores deployed in your environment can be adjusted to maximize resource efficiency.

Real-time stores deployed in your environment are not dynamically removed, so they are available to serve queries at all times. Dynamically provisioned real-time stores do not increase the amount of resources utilized within your environment.

It's recommended that you run at least four real-time store components to avoid impacting the availability of resources as process and user activity ramps up. By default, the data service can dynamically provision a maximum of 10 additional data service pods.

Enable dynamic scaling

To enable dynamic scaling for queries, open a support case.

Feedback