Configure the Data Service

This page provides information for self-managed customers configuring the data service.

Why set up the data service?

The data service setup is required for running an Appian environment. If the data service is unreachable when Appian starts up or if Appian is started before the data service is running, the application server will not start.

Additional configurations can be used to scale the data service to support higher scale usage of synced record types.

Monitoring and troubleshooting

Starting and stopping

To stop or restart the data service, you must stop and restart Appian.

Logging

Each component of the data service writes logs:

• Historical Store: hs-gateway.log
• Real-time Store: rts-gateway-*.log
• Appender: appender-gateway.log
• Bulk Ingestion: binge-gateway.log
• Data Client: client.log
• Watchdog: watchdog.log

The log files contain important information about startup and shutdown, process execution, configuration, and errors. In the event of a system issue, these files should be shared with Appian Support.

Note that for the real-time store components, the logs are enumerated as rts-engine-0.log, rts-engine-1.log, etc. for each real-time store component.

To access these logs, get into the data server pod (kubectl exec) and navigate to /usr/local/appian/ae/logs.

The data service also logs other data, including performance metrics and traces. See Logging for a more comprehensive overview of Appian logs.

Recovery and monitoring

Watchdog continuously monitors each component of the data service and restores functionality of each component in the event of an isolated failure.

To validate that the data service is running correctly:

1. Exec onto the data server pod:

   1
   kubectl exec
   

2. Execute the health.sh script.

The following information is displayed after executing the health script:

For the data service cluster:

• node_count: Number of nodes in the cluster.
• healthy: true if the data service is functioning normally, otherwise false.

For each node in the data service cluster:

• hostname: Host name of the node.
• ip: IP address of the node.
• healthy: true if the data service is functioning normally on this node, otherwise false.

Transaction Log

The data service uses the Kafka Topic ads2_tx_effects to persist and distribute transactions. The default retention time is one hour.

In self-managed environments, system administrators can change the retention time by setting the following custom property in the Appian CR configuration file:

1
2
3
spec:
  customProperties:
    kfk.trunc.buffer.seconds: "<seconds>"

File system

The <APPIAN_HOME>/ae/data-server/ directory stores the data service binaries, scripts, configuration details, and data.

The data files are located in the <APPIAN_HOME>/ae/data-server/data/ directory. The ss folder contains all snapshot files. Since access to data service is latency-sensitive, it is recommended that the data is hosted locally on the machine, rather than a shared drive or an external drive, such as shared network-attached storage (NAS). This is true in High Availability (HA) topologies as well, since each data service node stores its own version of the data.

For disaster recovery purposes, the <APPIAN_HOME>/data-server/data/ directory and the Kafka logs should be backed up regularly.

Troubleshooting

If the data service cannot start and the watchdog.log indicates an issue with the security token, check the watchdog.log for troubleshooting information.

If the data service stops running while the application server is running, any synced record types will be temporarily inaccessible. See Troubleshooting Data Sync for more information.

Sizing guidance

After configuring the data service, the amount of disk space and memory consumed by the data service will vary based on data volumes and usage patterns.

Disk space

After it is started for the first time in your environment, the data service is expected to take approximately 50 MB of disk space by default. If a site is not syncing record data, the data service will not occupy additional disk space.

Any additional disk space usage by the data service is expected to be proportional to the total amount of data synced into Appian. Data synced into the data service is compressed to optimize for storage efficiency. The exact disk usage will vary depending on the compression ratio and the current state of data sync activities and data service background processes running on the site.

Memory

In total, all data service components require a minimum of approximately 1GB of memory to run. Additional memory usage will vary based on the current activity on the site. For both Appian Cloud and self-managed environments, memory is bounded by a configurable container memory limit.

During periods of high write load, memory for each real-time store and appender component will increase proportionally to the amount of data written to the data service. Background processing is triggered automatically based on write usage to ensure memory stays within the configured container limit.

Memory spikes are expected for real-time store components during query execution. The magnitude of the spikes will vary based on the complexity of query operations and the volume of data being queried. The maximum memory usage for individual real-time store components varies based on the amount of available memory for a site. See configuring the real-time store for more detailed information about memory and compute configurations.

Configuring the real-time store

In the Appian CR configuration file, the .spec.dataServer.topology.rtsCount field specifies the number of real-time stores in the data service. The real-time store component is responsible for processing queries to the data service. The rtsCount property is set to 2 by default, but can be increased as needed to support higher query throughput. The maximum recommended rtsCount varies based on the amount of memory available as shown above.

In addition to configuring the number of real-time stores, you can use custom properties to configure limits on the amount of memory (ads.adb.systemConfig__.rts.memory_limit) and number of threads (ads.adb.systemConfig__.rts.threads) that each real-time store uses when processing queries. Increasing the amount of memory and number of threads will improve the performance of queries against synced record types, but may result in higher resource usage during query execution.

By default, the ads.adb.systemConfig__.rts.memory_limit property is unenforced and the ads.adb.systemConfig__.rts.threads property is set to 1. In addition to the guidelines based on available memory resources listed below, it is recommended that the ads.adb.systemConfig__.rts.threads * rtsCount be set less than or equal to the number of available CPU cores.

Data service connection restrictions

Appian restricts connections to the data service's HTTP endpoints by authenticating requests with a security token. Unauthenticated requests could be made after an unauthorized network intrusion or by a security application that scans the HTTP endpoints. When a unique security token is used, only trusted systems are permitted to make calls to the data service.

Feedback