Data Server

The data server is a storage layer designed for the Appian platform. It tracks the design objects added to the application patches for export. In future versions of Appian, the data server will be used as a data persistence layer for an increasing number of functionalities and capabilities. It provides better performance, more reliability, and increased security for the application data.

Components

Following are the major components of the data server:

Historical Store

The historical store is a single kdb+ database optimized for data writes. The historical store (also referred to as hs) and the historical store gateway (also referred to as hs-gw) form the writer in the data server. Every write made to the data server goes to the historical store.

Real-time Store

The real-time store consists of kdb+ databases and serves all query requests. It consists of "snapshot tables" which are read-optimized tables that serve queries (reads) in conjunction with in-memory, real-time tables. The real-time store (also referred to as rts) and the real-time store gateway (also referred to as rts-gw) form the reader in the data server. For on-premises installation, only a single real-time store is currently supported for the data server.

Watchdog

Watchdog is responsible for starting all of the data server components and monitoring the health of each component. The watchdog also heals (restarts) a component in case of an isolated failure.

Licensing

A valid license (k4.lic) is required to run the data server. Please refer Requesting and Installing a License page to obtain and install the license.

Configuring the Data Server

The configuration for the data server has to be specified in data-server-cluster element in the appian-topology.xml file. For example, for a single instance of data server:

1
2
3
4
5
6
<topology>
    ...
    <data-server-cluster>
        <data-server host="machine1.example.com" port="5400" rts-count="1"/>
    </data-server-cluster>
</topology>

For a high availability configuration, specify three instances of data server on different machines:

1
2
3
4
5
6
7
8
<topology>
    ...
    <data-server-cluster>
        <data-server host="machine1.example.com" port="5400" rts-count="1"/>
        <data-server host="machine2.example.com" port="5400" rts-count="1"/>
        <data-server host="machine3.example.com" port="5400" rts-count="1"/>
    </data-server-cluster>
</topology>

If data-server-cluster configuration is not specified in appian-topology.xml, it will run on the first node of Kafka listed in the kafkaCluster configuration. The appian-topology.xml file must be copied from <APPIAN_HOME>/conf/ into <APPIAN_HOME>/data-server/conf/ before starting the data server. The two copies of the topology file in both the locations must always be in sync, irrespective of the configurations specified.

Host

This attribute in data-server element represents the name of the machine hosting the data server. If host is not provided, the default is localhost. However, this behavior has been deprecated and will be removed in the future. In future versions of Appian, host needs to be explicitly specified for the data server. If the host is not specified, the following warning is printed in watchdog.log:

WARN com.appian.data.TopologyFactory - DataServer host is empty, defaulting to 'localhost'. This behavior is deprecated and will be removed in the future, and this topology will be considered invalid. You should not leave the host blank.

Port

Port element can be defined on both, the data-server-cluster and the data-server elements. If defined on both, the data-server element takes precedence. If not supplied, the default port number is 5400.

Real-time Store Count

The rts-count parameter determines the number of real-time stores in the data server. Multiple real-time stores will provide horizontal read scalability in future. Currently, the rts-count should only be set to 1 for on-premises installation.

Making changes to the topology

Changes to data-server-cluster configuration in appian-topology.xml are needed for reasons like adding or removing data server nodes, changing the host name, or changing the port. In such scenarios, take the following steps:

  1. Stop the data server on all the servers by exexuting <APPIAN_HOME>/data-server/bin/stop.sh on Linux or <APPIAN_HOME>\data-server\bin\stop.bat on Windows
  2. Delete <APPIAN_HOME>/ae/data-server/node/election directory from all the servers. This directory contains runtime data that needs to be deleted when certain topology changes are made.
  3. Make the required topology change on all the servers
  4. Start the data server on all the servers in any order by exexuting <APPIAN_HOME>/data-server/bin/start.sh on Linux or <APPIAN_HOME>\data-server\bin\start.bat on Windows

Data Location

The data files are located in <APPIAN_HOME>/data-server/data/. The hs folder contains the historical store data and the ss folder contains the real-time store (snapshot) data written on the disk. The <APPIAN_HOME>/data-server/data/ directory should be backed up as a part of your regular backup procedures.

See also:

Sizing

RAM

The Java Virtual Machine (JVM) and Go processes executed by the data server utilize approximately 200MB of RAM without any application data. On top of that, there is a negligible usage of RAM for application data by the data server. The background engine runs every hour and writes application data from memory to disk. Application patches and patch contents are the only functional objects stored in the data server in the current version.

Disk Usage

The data server may take up to 40MB of disk space by default after the application is started. In addition to that, disk space will be occupied based on the number of application patches created on the site. Following table gives an estimation of the amount of disk space utilized by patches on the data server:

O = number of objects per patch; P = number of patches created in the environment; S = size on disk per object

O x P x S Disk space used by patches (approximate)
25 x 1,000 x ~1KB 25MB
25 x 100,000 x ~1KB 2.5GB

Patches continue to reside on disk for audit purposes even after they are cleared.

Starting and Stopping

To start or stop the data server, please refer Starting and Stopping Appian page.

Note that when logging out of Windows, the data server process started by the user using the script will stop. Instead, the data server can be installed as a Windows service and started and stopped using the Windows service management console. For instructions on controlling the data server as a Windows service see Installing data server as a Windows Service.

Monitoring and Recovery

To check the status or health of the data server, execute <APPIAN_HOME>/data-server/bin/health.sh (health.bat on Windows).

The following information is contained in the output after executing the health script:

For the data server cluster,

  • node_count: Number of nodes in the cluster.
  • healthy: "true" if the data server is functioning normally, "false" otherwise.

For each node in the data server cluster,

  • hostname: Host name of the node.
  • ip: IP address of the node.
  • healthy: "true" if the data server is functioning normally on this node, "false" otherwise.

Watchdog continuously monitors each component of the data server and ensures that system is up again in case of an isolated failure.

Logs generated by the data server reside in <APPIAN_HOME>/logs/data-server/ folder. Logs are generated for each process in the data server.

Other Error Scenarios

If the data server is unreachable when the application starts up or if the application is started before the data server is running, the application server will not start.

If the data server stops running while the application server is running, creating Application patches is the only functionality that will be affected. The user will not be able to add objects to a patch if the data server is down.

FEEDBACK