Data Server

The data server is a storage layer designed for the Appian platform. It tracks the design objects added to the application patches for export. In future versions of Appian, the data server will be used as a data persistence layer for an increasing number of functionalities and capabilities. It provides better performance, more reliability, and increased security for the application data.

Components

Following are the major components of the data server:

Historical Store

The historical store is a single kdb+ database optimized for data writes. The historical store (also referred to as hs) and the historical store gateway (also referred to as hs-gw) form the writer in the data server. Every write made to the data server goes to the historical store.

Real-time Store

The real-time store consists of kdb+ databases and serves all query requests. In the current version of Appian, there can only be one real-time store. It consists of "snapshot tables" which are read-optimized tables that serve queries (reads) in conjunction with in-memory, real-time tables. The real-time store (also referred to as rts) and the real-time store gateway (also referred to as rts-gw) form the reader in the data server.

Watchdog

Watchdog is responsible for starting all of the data server components and monitoring the health of each component. The watchdog also heals (restarts) a component in case of an isolated failure.

Background Engine

The background engine (bg) is responsible for producing the optimized snapshot views that are used by the real-time store.

Licensing

A valid license (k4.lic) is required to run the data server. Please refer Requesting and Installing a License page to obtain and install the license.

Configuring the Data Server

Currently, the data server can only be configured to run on a single node (machine). The configuration has to be specified in data-server-cluster element in the appian-topology.xml file. For example,

1
2
3
4
5
6
<topology>
    ...
    <data-server-cluster>
        <data-server host="machine1.domain.tld" port="5400" rts-count="1"/>
    </data-server-cluster>
</topology>

Host

This attribute in data-server element represents the name of the machine hosting the data server. If host is not provided, the default is localhost.

Port

Port element can be defined on both, the data-server-cluster and the data-server elements. If defined on both, the data-server element takes precedence. If not supplied, the default port number is 5400.

Real-time Store Count

The rts-count parameter determines the number of real-time stores in the data server. Multiple real-time stores provide horizontal read scalability. The rts-count can only be set to 1 in the current version.

If data-server-cluster configuration is not specified in appian-topology.xml, it will run on the first node of Kafka listed in the kafkaCluster configuration. The appian-topology.xml file must be copied from <APPIAN_HOME>/conf/ into <APPIAN_HOME>/data-server/conf/ before starting the data server. The two copies of the topology file in both the locations must always be in sync, irrespective of the configurations specified.

Data Location

The data files are located in <APPIAN_HOME>/data-server/data/. The hs folder contains the historical store data and the ss folder contains the real-time store (snapshot) data written on the disk. The <APPIAN_HOME>/data-server/data/ directory should be backed up as a part of your regular backup procedures.

See also:

Sizing

RAM

The Java Virtual Machine (JVM) and Go processes executed by the data server utilize approximately 200MB of RAM without any application data. On top of that, there is a negligible usage of RAM for application data by the data server. The background engine runs every hour and writes application data from memory to disk. Application patches and patch contents are the only functional objects stored in the data server in the current version.

Disk Usage

The data server may take up to 40MB of disk space by default after the application is started. In addition to that, disk space will be occupied based on the number of application patches created on the site. Following table gives an estimation of the amount of disk space utilized by patches on the data server:

O = number of objects per patch; P = number of patches created in the environment; S = size on disk per object

O x P x S Disk space used by patches (approximate)
25 x 1,000 x ~1KB 25MB
25 x 100,000 x ~1KB 2.5GB

Patches continue to reside on disk for audit purposes even after they are cleared.

Starting and Stopping

To start or stop the data server, please refer Starting and Stopping Appian page.

Note that when logging out of Windows, the data server process started by the user using the script will stop. Instead, the data server can be installed as a Windows service and started and stopped using the Windows service management console. For instructions on controlling the data server as a Windows service see Installing data server as a Windows Service.

Monitoring and Recovery

Logs generated by the data server reside in <APPIAN_HOME>/logs/data-server/ folder. Following logs can be found in this directory:

  • data-server.log - The log containing the standard output from all components running on the server
  • hs-gw-<port>.log - The historical store log
  • rts-gw-<port>.log - The real-time store log
  • wd-<port>.log - The watchdog log
  • data-server-client.log - The log containing error messages at the data client level

To check the status or health of the data server, Execute <APPIAN_HOME>/data-server/bin/status.sh. (status.bat on Windows). The meaning of various machine status levels are as follows:

  • Green: Reads and writes can be performed. The writer(hs/hs-gw) and the reader(rts-0/rts-gw-0) are both up and running.
  • Red: Reads and/or writes can no longer be performed. This means either the writer (hs/hs-gw) is down or the reader(rts-0/rts-gw-0) is down.

Watchdog continuously monitors each component and ensures that system is up again in case of an isolated failure.

Other Error Scenarios

If the data server is unreachable when the application starts up or if the application is started before the data server is running, the application server will come up successfully and the following error message will be logged in the application-server log:

Failed to connect to data server. The ability to store and retrieve application patches will not be available until the connection is reestablished. Check the status of the data server.

Creating Application patches is the only functionality that will be affected if the data server is not available. The user will not be able to add objects to a patch if the data server is down. Following error message is displayed on the UI when a user attempts to do so:

Your patch cannot be reached at this time. Check that the data server is running. As a workaround, add your objects to an application and export the full application package.

FEEDBACK