TAS Operations Runbook

This article describes recommended operational procedures for maintaining and supporting the Team Assistant (TAS) platform in on-premise or customer-managed infrastructure. It covers backup and restore, Docker Swarm service management, updates, monitoring, working with logs, and basic procedures for handling errors.

TAS version: 5.7–5.15

Target audience: infrastructure administrator, partner, or customer who does not know TAS in depth but is responsible for operations, backup, monitoring, and basic environment maintenance.

1. System overview

A typical TAS installation consists of the following parts:

In the standard deployment, TAS runs as a Docker Swarm. Docker serves here as the main mechanism for managing and controlling individual services. Services can be centrally started, stopped, restarted, and inspected using Docker commands.

2. Backup

2.1 Minimum backup scope

The minimum backup is the smallest set of data needed to restore TAS to an operational state. It must contain:

1. the TAS application database,
2. DMS storage — attachments, documents, and files in cases,
3. TAS and deployment configuration files,
4. application files stored in the deployment — especially assets, templates, uploads, plugins, and other files stored outside the Docker image,
5. SSL/TLS certificates, if stored locally on the server and not managed by another internal customer mechanism.

If the database is restored without a corresponding DMS, TAS may contain references to files that do not exist on disk. If the DMS is restored without a corresponding database, files may exist on disk that TAS is unaware of.

2.2 Deployment and configuration file locations

The specific file locations depend on how the partner or customer performed the installation. Before setting up backups, the actual paths on the specific server must be verified.

The recommended location of the entire deployment according to the installation guide is:

/srv/tas/stack/<stack_name>/deployment

Where <stack_name> is the name of the specific Docker Swarm stack / instance. The name differs depending on the specific implementation, for example prod, test, example, tas-prod, etc.

Typical structure relevant for backup:

/srv/tas/stack/<stack_name>/deployment
├── common
│   ├── <service_name>.yml
│   ├── arango.yml
│   ├── elastic.yml
│   ├── redis.yml
│   ├── tas.yml
│   └── tika.yml
└── instance
    └── <stack_name>
        ├── .env
        ├── instance.yml
        ├── config
        │   ├── local.js / config.js
        │   └── tas.js
        └── storage
            ├── backend
            │   ├── assets
            │   ├── cache
            │   ├── plugins
            │   └── tmp
            ├── frontend
            │   └── assets
            └── dms

The following files and folders are considered key:

DMS storage

The default DMS location in the recommended deployment is:

/srv/tas/stack/<stack_name>/deployment/instance/<stack_name>/storage/dms

This only applies if the mapping in instance.yml has not been changed.

If the DMS source folder is mapped in instance.yml from a different path, for example /srv/dms-data or /srv/dms, then this external path must be backed up, not just the DMS folder inside the deployment.

In practice, there are therefore two common backup variants:

SSL/TLS certificates

Certificates are the responsibility of the partner or customer who manages the infrastructure and reverse proxy. If certificates and private keys are stored locally on the server, they must be included in the backup policy according to the customer's internal security rules.

Typical locations may be, for example, /etc/httpd/ssl, /etc/apache2/ssl, or /etc/nginx/ssl. The specific path depends on the server administrator and the reverse proxy used.

2.3 Recommended backup scope

2.4 Recommended backup policy

Database

• Complete backup at least once a day.
• Minimum retention of 7 days, unless the customer's internal policy requires stricter rules.
• The backup must be stored outside the same disk / server on which TAS is running.
• For production environments, it is also advisable to schedule regular restore tests.

Deployment and configuration

• Back up after every configuration or deployment change.
• It is recommended to back up the entire /srv/tas/stack/<stack_name>/deployment folder.
• If the DMS is outside the deployment, back up both the deployment and the external DMS path.

DMS storage

• Daily backup or snapshot.
• Retention should be aligned with the database backup.
• For larger installations, size growth and file count need to be monitored.
• During restore, the link between files in the DMS and records in the database must be taken into account.

Docker volumes of individual services

For services running in Docker Swarm, their Docker volume can be backed up as needed. If the installation was performed according to the installation guide with the Docker namespace, Docker volumes are by default located here:

/var/lib/docker/500000.500000/volumes

If Docker has been configured differently, the path may be different.

Example of a general procedure for backing up a Docker volume:

# 1) Verify the exact service name
docker service ls

# 2) Stop the specific service
docker service scale <service_name>=0

# 3) Back up the corresponding volume according to the actual name and path
sudo tar -czf /srv/backup/<service_name>_volume_$(date +%F).tgz \
  -C /var/lib/docker/500000.500000/volumes/<volume_name>

# 4) Start the service again
docker service scale <service_name>=1

The value after = sets the desired number of replicas. The value 0 stops the service. The value 1 starts it with one replica. Some environments may have a higher replica count; in that case, the original value must be restored.

2.5 Responsibility for backups

Unless contractually agreed otherwise:

• the customer or partner is responsible for infrastructure backups,
• the customer or partner is responsible for backing up certificates, private keys, reverse proxy configuration, and external storage,
• TAS support provides application guidance, a description of recommended paths, and information about which parts are important for recovery,
• TAS does not, by default, ship universal backup scripts for all environments, because specific paths and technologies differ across customers.

3. Working with services in Docker Swarm

3.1 The role of Docker in TAS

Within TAS, Docker serves as the main mechanism for managing the deployment of individual services. Each main component runs as a separate Docker service.

Typical services:

<INSTANCE_NAME>_frontend
<INSTANCE_NAME>_backend
<INSTANCE_NAME>_backend-cron
<INSTANCE_NAME>_tika
<INSTANCE_NAME>_redis
<INSTANCE_NAME>_elasticsearch
<INSTANCE_NAME>_arango
<INSTANCE_NAME>_database      # only if DB is part of the stack

The exact service names must always be verified using docker service ls.

3.2 Checking service status

The basic command for checking all deployed services:

docker service ls

This command lists the services, replica count, image, and ports.

For a more detailed inspection of a specific service:

docker service ps <service_name>

To check running containers:

docker ps

To check Docker Swarm stacks:

docker stack ls

3.3 Stopping and starting a service

An individual service can be stopped by scaling the replica count to 0:

docker service scale <service_name>=0

The service can be started again by scaling to the desired replica count:

docker service scale <service_name>=1

Example:

docker service scale prod_backend=0
docker service scale prod_backend=1

3.4 Recommended order of stopping and starting services

During a planned operation, it is recommended to follow the service order to minimize the risk of an inconsistent state or errors in dependent components.

Recommended stop order

1. backend-cron
2. backend
3. frontend
4. arango
5. elasticsearch
6. redis
7. tika
8. database — only if the database is part of the stack

Example:

docker service scale <INSTANCE_NAME>_backend-cron=0
docker service scale <INSTANCE_NAME>_backend=0
docker service scale <INSTANCE_NAME>_frontend=0
docker service scale <INSTANCE_NAME>_arango=0
docker service scale <INSTANCE_NAME>_elasticsearch=0
docker service scale <INSTANCE_NAME>_redis=0
docker service scale <INSTANCE_NAME>_tika=0
docker service scale <INSTANCE_NAME>_database=0

Recommended start order

1. database — only if the database is part of the stack
2. tika
3. redis
4. elasticsearch
5. arango
6. frontend
7. backend
8. backend-cron

Example:

docker service scale <INSTANCE_NAME>_database=1
docker service scale <INSTANCE_NAME>_tika=1
docker service scale <INSTANCE_NAME>_redis=1
docker service scale <INSTANCE_NAME>_elasticsearch=1
docker service scale <INSTANCE_NAME>_arango=1
docker service scale <INSTANCE_NAME>_frontend=1
docker service scale <INSTANCE_NAME>_backend=1
docker service scale <INSTANCE_NAME>_backend-cron=1

If the database is not part of the Docker Swarm stack, the database step is skipped.

3.5 DMS permissions and the dkr-tas user

In an installation following the standard guide, the dkr-tas user is created on the host server. Thanks to Docker namespaces, this user is mapped into the TAS containers.

From an operational point of view, it is important that the DMS folder, its subfolders, and files are set up so that the dkr-tas user can write to them. This is particularly important after:

• a manual restore of the DMS from a backup,
• manually copying files to the server,
• a change of DMS mapping in instance.yml,
• migrating the DMS to a different disk or mount.

Example check:

ls -la /srv/tas/stack/<stack_name>/deployment/instance/<stack_name>/storage
ls -la /srv/tas/stack/<stack_name>/deployment/instance/<stack_name>/storage/dms

Example of setting ownership after a DMS restore:

sudo chown -R dkr-tas:dkr-tas /srv/tas/stack/<stack_name>/deployment/instance/<stack_name>/storage/dms

If the DMS is mapped outside the deployment, the real path must be used, for example:

sudo chown -R dkr-tas:dkr-tas /srv/dms-data

4. Data restore

4.1 General restore principle

Restore the database, DMS, or configuration files while TAS is stopped. Stopping the services is important so that the application does not write to the database or DMS during the restore.

Before restoring, the following should be clarified:

• the timestamp of the database backup,
• the timestamp of the DMS backup,
• whether the DMS folder was inside the deployment or outside at the time of the backup,
• whether the path mapping in instance.yml has changed since the backup,
• whether configuration files have changed,
• whether Docker volumes of auxiliary services also need to be restored.

4.2 Database restore

Recommended procedure:

1. Inform users about the downtime.
2. Stop TAS services according to the recommended order in chapter 3.4.
3. Restore the database from the selected backup.
4. Verify database access and settings.
5. In particular, verify:
   • availability of the database server,
   • the correct database user,
   • the database schema,
   • collation,
   • READ_COMMITTED_SNAPSHOT or other required settings depending on the DB used,
   • network reachability of the DB from the backend container.
6. Start the services according to the recommended order in chapter 3.4.
7. Perform validation tests:
   • user login,
   • loading the case list,
   • opening an existing case,
   • opening a task,
   • performing a basic action on a task,
   • checking access to attachments,
   • checking the backend logs.

4.3 DMS restore

Recommended procedure:

1. Inform users about the downtime.
2. Stop TAS services according to chapter 3.4.
3. Verify the actual DMS path:
   • by default instance/<stack_name>/storage/dms,
   • or the external path according to instance.yml.
4. Replace the DMS folder contents with those from the backup.
5. Set ownership and permissions so that the dkr-tas user can write to the DMS.
6. Start the services according to chapter 3.4.
7. Open an existing case with an attachment and verify that the file can be displayed or downloaded.
8. Check the backend logs for errors such as file not found, permission denied, or similar.

Example of fixing permissions:

sudo chown -R dkr-tas:dkr-tas /path/to/dms

4.4 Configuration file restore

Restore configuration files carefully, as they may contain path settings, image versions, DB connections, secrets, SMTP, authentication, or volume mappings.

Recommended procedure:

1. Stop the affected services or, if needed, the entire TAS.
2. Verify which configuration files should be restored.
3. Restore the files from the backup.
4. Check the differences against the current state, in particular:
   • DMS paths,
   • secret names,
   • DB hostname and port,
   • image versions,
   • frontend/backend ports,
   • extra hosts,
   • replica count.
5. Redeploy or restart the affected services depending on the type of change.
6. Verify the application is working.

If only configuration changes and no database migrations are involved, the procedure typically follows the installation guide for applying configuration changes without the migration parameter.

4.5 Restore test

It is recommended to test a complete restore — or at least a restore on a test environment — at least once a year.

A restore test should verify:

• database restore,
• DMS restore,
• restore of the deployment/config folder,
• DMS permissions,
• user login,
• opening a case,
• opening an attachment,
• running a task,
• availability of frontend/backend status endpoints,
• basic logs with no critical errors.

5. Server and package updates

5.1 General principle

Security updates of the server are recommended. However, major infrastructure changes must be coordinated with the application administrator or TAS support. The reason is that changes at the database, storage, reverse proxy, or Docker level can affect application availability, access to DMS, authentication, or deployment behaviour.

5.2 Updates that can usually be performed without coordination

Unless the customer's internal policy states otherwise, the following can typically be performed:

• OS security patches,
• minor Docker updates,
• updates of monitoring agents,
• antivirus updates,
• routine OS maintenance without architectural changes.

5.3 Changes requiring coordination

Coordination with the application administrator or TAS support is required especially for:

• major database upgrade,
• changing database collation,
• migrating between database engines,
• major PostgreSQL upgrade,
• changing storage paths,
• changing DMS mapping,
• reverse proxy configuration changes,
• certificate changes, if they may affect application availability,
• changing the Docker namespace or Docker root path,
• relocating Docker volumes,
• changes to network topology, firewall, or DNS,
• changes to authentication integration.

5.4 TAS version update

Upgrading TAS to a newer version must follow the official installation or upgrade guide and be coordinated with the TAS vendor / support, unless agreed otherwise.

The TAS release strategy typically distinguishes:

• Release Candidate (RC),
• Stable release,
• Patch update.

Before a production update, it is advisable to have:

• a database backup,
• a DMS backup,
• a backup of the deployment/config folder,
• a verified test environment,
• a rollback plan,
• an agreed maintenance window.

6. Service restart

6.1 Difference between restart and stop/start

When restarting a service in Docker Swarm, a fixed order usually does not need to be followed. Docker handles the restart of the given service and its integration with the other components.

Order matters mainly during a full manual stop and subsequent start of services, for example during data restore, storage operations, or maintenance, when services are stopped for an extended period.

6.2 Restart using Docker Swarm

To restart a service in the standard Docker Swarm deployment, use:

docker service update --force <service_name>

Examples for individual services:

docker service update --force <INSTANCE_NAME>_frontend
docker service update --force <INSTANCE_NAME>_backend
docker service update --force <INSTANCE_NAME>_backend-cron
docker service update --force <INSTANCE_NAME>_tika
docker service update --force <INSTANCE_NAME>_redis
docker service update --force <INSTANCE_NAME>_elasticsearch
docker service update --force <INSTANCE_NAME>_arango
docker service update --force <INSTANCE_NAME>_database

Use the <INSTANCE_NAME>_database service only if the database is part of the Docker Swarm stack. The order listed does not imply a mandatory restart order — services can be restarted individually as needed.

6.3 Post-restart check

After a restart, verify:

• that the service is running in docker service ls,
• that a new container has been created and is not in a restart loop,
• that the HTTPS endpoint responds,
• that you can log in,
• that the case list loads,
• that a case and task can be opened,
• that there are no immediate HTTP 500 errors in the backend log,
• that scheduled jobs are running (backend-cron),
• that status routes return the expected state.

Example commands:

docker service ls
docker service ps <INSTANCE_NAME>_backend
docker service logs <INSTANCE_NAME>_backend --tail 200
docker service logs <INSTANCE_NAME>_backend-cron --tail 200

7. Monitoring

7.1 Infrastructure monitoring

It is recommended to monitor at least:

7.2 Database monitoring

It is recommended to monitor:

• number of active connections,
• deadlocks,
• long-running queries,
• growth of the database size,
• growth of the transaction log,
• availability of the DB from the backend container,
• CPU/RAM/disk load on the database server,
• errors in the database logs.

7.3 Application monitoring

It is recommended to monitor:

• HTTP 500 error rate,
• backend response time,
• frontend availability,
• backend availability,
• container restart count,
• backend-cron status,
• completion of scheduled jobs,
• Elasticsearch availability,
• ArangoDB availability,
• Redis availability,
• errors in Docker service logs.

7.4 DMS monitoring

It is recommended to monitor:

• DMS size growth,
• available disk space,
• disk performance,
• permissions on the DMS folder,
• errors such as permission denied or file not found in the backend log.

7.5 Backend status routes

The TAS backend implements a mechanism for safe shutdown and provides dedicated HTTP endpoints for application health monitoring. This allows the orchestrator or load balancer to precisely control the flow of traffic.

A more detailed description is in the Status routes documentation.

GET /status/liveness

This endpoint answers the question of whether the application is "alive" — that is, not stuck and able to respond.

Usage:

• suitable for liveness probes,
• the orchestrator uses it to decide whether to consider the container alive,
• should not be used as the sole signal for sending traffic.

Principle:

• liveness = true after successful application startup,
• liveness = false only at the end of graceful shutdown, after the HTTP server and DB connections have been closed.

GET /status/readiness

This endpoint answers the question of whether the application is ready to accept operational requests.

Usage:

• suitable for readiness probes,
• the load balancer or orchestrator uses it to decide whether traffic can be sent to the backend,
• during graceful shutdown, readiness is set to false immediately at the start so that new requests stop being accepted as soon as possible.

Principle:

• readiness = true after successful startup completion,
• readiness = false immediately when graceful shutdown begins.

Monitoring recommendations

• Use readiness to decide whether to send traffic to the backend.
• Use liveness to detect a "stuck" or unresponsive application.
• Alert if readiness or liveness returns 503 outside of planned downtime or deploy.
• During restart or deploy, expect a brief 503 state, which may be expected.

Example of a manual check:

curl -i https://<tas-domain>/api/status/liveness
curl -i https://<tas-domain>/api/status/readiness

8. Error handling procedures

8.1 Level 1 — functional error

Typical symptoms:

• a specific case fails,
• a specific task fails,
• the UI displays an error message,
• the error relates to a specific process or template.

Recommended procedure:

1. Verify whether the error is reproducible.
2. Identify the specific case, task, user, and time of the error.
3. Check the backend logs.
4. Check whether the error is related to user permissions.
5. If the error recurs, escalate to TAS support with logs and the reproduction steps.

8.2 Level 2 — system slowdown

Typical symptoms:

• slow case loading,
• slow loading of overviews,
• delayed task hand-off,
• long-running scheduled jobs,
• increased response times.

Recommended procedure:

1. Check the application server CPU/RAM.
2. Check the disk I/O.
3. Check the DB performance and long-running queries.
4. Check the number of active DB connections.
5. Check the Elasticsearch state.
6. Check the backend-cron state.
7. Check the backend logs for errors and timeouts.
8. If the issue persists, escalate to TAS support.

8.3 Level 3 — service unavailable

Typical symptoms:

• TAS is unavailable,
• users see 502 / 503 / timeout,
• login is not possible,
• the frontend or backend is unresponsive,
• a Docker service is in a restart loop.

Recommended procedure:

1. Verify server availability.
2. Verify reverse proxy status.
3. Verify the status of Docker services: docker service ls
4. Verify the details of a specific service: docker service ps <service_name>
5. Check the backend logs: docker service logs <INSTANCE_NAME>_backend --tail 200
6. Check the frontend logs: docker service logs <INSTANCE_NAME>_frontend --tail 200
7. Verify database connectivity.
8. Verify the status routes, if the backend responds.
9. Restart the affected service, if it is safe to do so.
10. If the issue persists, escalate to TAS support.

9. Logs

9.1 Log types

In TAS operations, the following logs are typically used:

• backend application logs,
• frontend / web server logs depending on the image type and configuration,
• backend-cron logs,
• Docker container logs,
• Docker service logs,
• reverse proxy logs,
• database logs,
• Elasticsearch logs,
• ArangoDB logs,
• system logs of the server.

9.2 Accessing the logs

Logs of a specific container:

docker logs <container_name>

Logs of a specific Docker Swarm service:

docker service logs <INSTANCE_NAME>_backend
docker service logs <INSTANCE_NAME>_frontend
docker service logs <INSTANCE_NAME>_backend-cron

Practical use with a line limit:

docker service logs <INSTANCE_NAME>_backend --tail 200

Following the log in real time:

docker service logs <INSTANCE_NAME>_backend -f

Reverse proxy logs depend on the technology used. Examples for Nginx: /var/log/nginx/. Examples for Apache / httpd: /var/log/httpd/ or /var/log/apache2/. Database logs depend on the DB engine used and its configuration.

9.3 What to look for in the logs

When diagnosing, look in particular for:

• ERROR,
• Unhandled,
• Timeout,
• Deadlock,
• Connection refused,
• ECONNREFUSED,
• permission denied,
• file not found,
• Cron failure,
• HTTP 500,
• database connection errors,
• Elasticsearch connection errors,
• ArangoDB connection errors,
• DMS handling errors.

10. Security recommendations

• HTTPS must be enforced on production environments.
• Private keys and certificates must be protected according to the customer's internal security policy.
• Database and DMS backups must be protected at least to the same level as production data.
• Access to the server must be restricted to authorized persons only.
• Access to Docker is highly privileged and should be governed by the internal security policy.
• Regular OS security patches are recommended.
• Major infrastructure changes must be coordinated.
• Short-lived access tokens are preferable from a security standpoint to long-lived tokens.
• Do not store secrets and passwords in commonly accessible files outside the designated mechanism.

11. Escalation and support

In the event of a critical issue, prepare the following information before escalating:

• environment name,
• environment URL,
• TAS version,
• database type and version,
• operating system,
• whether the DB is part of the Docker stack or runs outside of it,
• description of the problem,
• time the problem occurred,
• steps to reproduce,
• relevant logs,
• information about recent infrastructure changes,
• information about the last deploy or restart,
• status of Docker services from docker service ls,
• optionally the output of status routes.

Example of useful commands before escalating:

docker service ls
docker stack ls
docker service ps <INSTANCE_NAME>_backend
docker service logs <INSTANCE_NAME>_backend --tail 200
docker service logs <INSTANCE_NAME>_backend-cron --tail 200
curl -i https://<tas-domain>/api/status/liveness
curl -i https://<tas-domain>/api/status/readiness

12. Note on environment differences

This document describes the recommended standard operational model of TAS. A specific environment may differ, for example, by:

• a different stack name,
• a different deployment path,
• externally mapped DMS,
• a database outside Docker Swarm,
• a different reverse proxy,
• different certificate paths,
• customer security policy,
• a custom backup tool,
• different monitoring.

13. Quick command cheat sheet

# List services
docker service ls

# List stacks
docker stack ls

# Services in a stack
docker stack services <stack_name>

# Service details
docker service ps <service_name>

# Stop a service
docker service scale <service_name>=0

# Start a service with one replica
docker service scale <service_name>=1

# Restart a service
docker service update --force <service_name>

# Service logs
docker service logs <service_name> --tail 200

# Follow service logs
docker service logs <service_name> -f

# Running containers
docker ps

# Container logs
docker logs <container_name>