Upgrading TAS 5.17 to 5.19 – Technical Part (DevOps)

Technical guide for upgrading a customer TAS instance from v5.17 to v5.19. Intended for DevOps engineers and administrators who plan and carry out the upgrade: infrastructure, configuration, database migrations, deployment procedure and rollback.

TL;DR – what is most likely to trip you up

Checklist

Complete this before you touch anything else.

Migration data prerequisites

Run against a copy of the production DB. Every row in the result means the migration will fail.

(A) JS_SCRIPTS.JS_TYPE – newly NOT NULL + check in ('C','F','R'), no backfill. The legacy default was 'B'.

SELECT "JS_ID", "JS_TYPE" FROM "JS_SCRIPTS"
WHERE "JS_TYPE" IS NULL OR "JS_TYPE" NOT IN ('C','F','R');

(B) TEMPLATE_TASKS – new CHECK constraints.

SELECT DISTINCT "TTASK_TYPE"                FROM "TEMPLATE_TASKS";  -- allowed: A,S,P,E,N,W,C
SELECT DISTINCT "TTASK_ASSESMENT_METHOD"    FROM "TEMPLATE_TASKS";  -- S,U,T,L,W,C,A,V,P
SELECT DISTINCT "TTASK_ASSESMENT_HIERARCHY" FROM "TEMPLATE_TASKS";  -- G,C,D,P,A,S,L
SELECT DISTINCT "TTASK_INVOKE_EVENT"        FROM "TEMPLATE_TASKS";  -- I,B
SELECT DISTINCT "TTASK_ENOT_TGT_TYPE"       FROM "TEMPLATE_TASKS";  -- O,G,T,P,S,U,R
--   same for TTASK_ENOT_COPY_TYPE, TTASK_ENOT_BLIND_TYPE, TTASK_ENOT_REPLY_TYPE
SELECT DISTINCT "TTASK_CAN_BE_BUTTON", "TTASK_GEN_HISTORY",
       "TTASK_IS_BULK_COMPLETABLE", "TTASK_MULTIINSTANCE_FLAG",
       "TTASK_SUFFICIENT_END"               FROM "TEMPLATE_TASKS";  -- Y,N only

(C) TEMPLATE_GRAPH

SELECT DISTINCT "TGRAPH_COLOR"     FROM "TEMPLATE_GRAPH";
--   allowed: fffacd, e6e6fa, ffe4e1, f0ffff, ffe4b5, ffffff, '' 
--   (the migration lowercases the values first)
SELECT DISTINCT "TGRAPH_BPMN_TYPE" FROM "TEMPLATE_GRAPH";
--   allowed: bpmn:UserTask, bpmn:SendTask, bpmn:ReceiveTask, bpmn:ServiceTask,
--            bpmn:SubProcess, bpmn:IntermediateThrowEvent, bpmn:IntermediateCatchEvent,
--            bpmn:ScriptTask, bpmn:Task, bpmn:Participant, bpmn:Lane

(D) Orphan processes – the migration adds an FK INSTANCE_PROCESSES → TEMPLATE_PROCESSES. It sets TPROC_ID = NULL on orphans by itself; determine the extent in advance.

SELECT count(*) FROM "INSTANCE_PROCESSES" ip
WHERE ip."TPROC_ID" IS NOT NULL
  AND NOT EXISTS (SELECT 1 FROM "TEMPLATE_PROCESSES" tp WHERE tp."TPROC_ID" = ip."TPROC_ID");
--   same for ARCH_INSTANCE_PROCESSES

(E) Duration estimate of the heaviest steps

SELECT count(*) FROM "TEMPLATE_TASK_LINKS";            -- migration #18 performs row-by-row UPDATEs
SELECT count(*) FROM "INSTANCE_PROCESS_HISTORY";       -- migration #22 casts date -> timestamp
SELECT count(*) FROM "ARCH_INSTANCE_PROCESS_HISTORY";  -- on MSSQL via an intermediate nvarchar(max) step
SELECT count(*) FROM "INSTANCE_TASKS";                 -- column drops + CHECK validation
SELECT count(*) FROM "ARCH_INSTANCE_TASKS";

Environment blockers

Backup

Migrations irreversibly delete data. A full DB backup is required – the migrations' down() restores only the schema, not the content.

Objects removed by the upgrade:

• CRON_RUNS – cron run history (moved to Elasticsearch, old data is not transferred),
• GUIDES, HEALTH_STATUS_PARAM,
• INSTANCE_TASK_INVITATIONS, TEMPLATE_TASK_INVITATIONS,
• INSTANCE_TASK_LINKS, INSTANCE_LINK_CONDITIONS,
• INSTANCE_GRAPH, ARCH_INSTANCE_GRAPH,
• TEMPLATE_TASK_CALCULATIONS,
• columns: ITASK_SEEN, ITASK_DISC_FLAG, TTASK_DISC_FLAG, TTASK_ENOT_BODY, TTASK_IS_PROC_ID, TTJSCALC_TITLE, ITJSCALC_TITLE, TTASKLINK_IS_MANDATORY, IGRAPH_ITASKCON_ID, CRON_ALIAS, CRON_IS_CLONE, CRON_TIMEOUT, CRON_HELP.

Infrastructure

Runtime and dependencies

Neither package.json has an engines field – the Node version is determined solely by the base image.

New backend dependencies: @modelcontextprotocol/sdk, @fastify/sse, fastify-plugin, tar-stream, tmp, tsx (dev).

Removed: arangojs, typedi (replaced by a custom DI in backend/src/infrastructure/di/service.ts), reflect-metadata, systeminformation, docx-templates, docxtemplater, pizzip (moved to the docxGenerator plugin).

Frontend: ESLint + Prettier → Biome 2.4.x; added Vitest 4 + Testing Library + jsdom; react-beautiful-dnd → @dnd-kit; removed hopscotch (discontinued Guides) and npm as a frontend dependency. Security bumps: axios 1.16.1, sanitize-html 2.17.4, dompurify 3.4.5, express 4.22.2, lodash 4.18.1, overview-components 1.1.184. React / MUI / webpack major versions unchanged.

Dockerfile

Backend (backend/Dockerfile)

• ARG TAS_APP_VERSION → ENV TAS_APP_VERSION baked into every stage. CI passes it from the git tag (TAG_NAME_CLEANED, including the prerelease suffix).
• npm ci --legacy-peer-deps (because of the openapi-typescript peer dep), in both the prod and dev stages.
• npm_config_allow_remote=root, npm_config_allow_git=root – npm 12 would otherwise reject the xlsx CDN tarball and the annotpdf git dependency.
• The plugin source is no longer in the backend build context (backend/.dockerignore); plugin_builder receives it via a named build context. plugin_builder now inherits FROM test (previously FROM linter).
• Unit tests are no longer run during the build (they run in a separate CI job).
• The set of apk packages (LibreOffice, OpenJDK17, Chromium, msttcorefonts, openssl) is unchanged.
• Production target names are unchanged: prod, prod_obfuscated (default), prod_devlicense, prod_obfuscated_devlicense.

If you build the image manually, you must use:

docker buildx build --build-context plugin_source=./backend/plugin --target plugin_archives ...

Frontend (frontend/Dockerfile) – check your deployment pipeline

The build has been split by license. 5.17 had source → prod / prod_obfuscated. 5.19 has:

source_base → source_prodlicense → prod, source_obfuscated_prodlicense → prod_obfuscated (default)
            → source_devlicense  → prod_devlicense, source_obfuscated_devlicense → prod_obfuscated_devlicencse

The build patches developmentBuild in frontend/src/components5.0/zustand/loggedUserStore.ts according to the license.

New and changed env variables

Newly mandatory

Removed

New (all optional)

Dead, but still declared

TAS_LOGGER_FILE_PATH remains in envVariables.ts, but staticConfigHandler.ts no longer reads it (logging.fallbackfilePath has disappeared from globalConfig.ts). It has been replaced by TAS_LOGGER_SIEM_FILE_PATH.

Stricter validation of the static configuration – it may reject a local override that has worked so far.

Frontend

Dynamic (GUI) configuration

Database migrations

24 new logical migrations, each as an MSSQL + PostgreSQL pair (none of them is dialect-neutral). Plus regenerated snapshots (.snapshot-TAS.json for both dialects).

Migration overview

Certificate migration to Vault (#3)

The migration walks through the directory process.env.TAS_CERTIFICATES_STORAGE_PATH (fallback /app/tas/storage/certificates), parses every file in it as X.509, encrypts it and inserts it as a secret_store row with kind = certificate.

Before running the migration, verify:

• that the variable points to the correct directory and that the directory is available in the migration's runtime environment (volume mounted),
• that the directory contains only certificates – the migration will try to import anything else (README files, keys, temp files),
• that globalThis.container (database encryption key) is initialized and that a user with id = 1 exists.

secret_store.valid_from divergence between 5.17 and 5.19

5.17 added the column via the migration Migration20260528120000-secret-store-valid-from. This migration does not exist in 5.19 – the column is added by the earlier Migration20260424112724 (MSSQL) / …725 (PG).

Consequences for a DB that has already gone through 5.17:

• An orphaned record Migration20260528120000 remains in the migration table.
• Migration20260424112724/25 will run “out of order”. Both are written defensively (MSSQL if not exists (select 1 from sys.columns …), PG add column if not exists), so they will not fail – this is the tas-3673 fix; previously this ended with MSSQL error 2705.
• PostgreSQL – type mismatch. 5.17 created valid_from as timestamp, whereas 5.19 would create it as timestamptz. The migration skips the column, so it stays timestamp and does not match the 5.19 snapshot/entity. Functionally this does not matter for now, but the next migration generated from the snapshot may try to address it. Verify after the upgrade and align manually if needed.
• breakpointCheck (npm run mig) may flag the orphaned record – use npm run mig:nocheck or the new tools below.

New CLI for troubleshooting migrations

npm run db:migrations                 # JSON dump of the migration table: name, time, what is pending
npm run db:migrations:set-state -- name=<Migration…> state=executed   # marks it as executed (the SQL is not run)
npm run db:migrations:set-state -- name=<Migration…> state=pending    # deletes the row → it will run again

Other new commands: npm run db:report (index fragmentation report + heavy queries), npm run db:rebuild-indexes.

Post-migration step outside the migration table

Every migrate up now runs cron parameter normalization against the parametersSchema of the respective cron class:

• type mismatches are examined and coerced ("5" → 5),
• properties not allowed by the schema are removed,
• defaults are not filled in,
• a row that still does not validate after normalization (or could not be parsed) is left untouched and a warning is logged.

Recommended upgrade procedure

1. Announce the downtime. Estimate the window using query (E) – migrations #14, #18, #19 and #22 scale linearly with the number of rows.
2. FULL DB BACKUP + snapshot of the storage volume (certificates, plugins, DMS).
3. Pre-flight check of the data prerequisites and environment blockers. Fix data defects NOW.
4. Stop operations:
   • first the cron workers (required – the cron overhaul drops columns in a single phase),
   • then the web backends,
   • put the frontend / LB into maintenance.
5. Prepare the configuration for 5.19:
   • TAS_APP_VERSION (mandatory),
   • remove TAS_LOGGER_ARANGO_* and TAS_LOGGER_ELASTIC_ENABLE,
   • verify TAS_CERTIFICATES_STORAGE_PATH for the migration,
   • optionally TAS_PLUGIN_STORE_URL on the frontend.
6. Deploy the 5.19 images (backend + frontend), but DO NOT START the backends yet.
7. Run the migrations from a single instance: npm run mig. Watch the log; if it fails, do not retry blindly – see the new CLI.
8. Check the migration log:
   • warnings from cron parameter normalization,
   • lines tagged [tas-3432] = configuration of the deleted EWS crons → save it aside,
   • the result of the certificate migration into secret_store.
9. Deploy / update the plugins to the 5.19 versions. Without this they will not load.
10. Start the web backends → verify /status/liveness, /status/readiness, /status/startup.
11. Start the cron workers.
12. Perform the post-upgrade configuration.
13. Go through the verification checklist.
14. Turn off maintenance mode.

Post-upgrade configuration

Plugins – mandatory

Manifests in 5.19 have minimalCompatibleTasVersion: "5.19.0" and maximalCompatibleTasVersion: "5.19.999". Because application.version now contains the actual version (previously it was masked by an outdated manifest.json with 5.18.16), an incompatible plugin is silently skipped.

Plugin versions in 5.19:

New plugin management options:

• Administration → Plugins → Plugin Store tab (only when TAS_PLUGIN_STORE_URL is set),
• POST /plugins/install – uploads the archive to the plugin volume; this does not load the plugin,
• DELETE /plugins/:sysname – uninstall (deletes the folder, restartRequired: true),
• POST /plugins/kill – broadcast via Redis; each backend (web and cron) sends SIGTERM to its own graceful-shutdown handler and the orchestrator starts it again,
• a plugin with a broken manifest or a capability that cannot be loaded no longer crashes backend startup – it is logged and skipped (tas-3545).

Crons – requires attention

Complete overhaul (tas-3500). What to do after the upgrade:

1. Go through the entire list of crons in Administration → Crons. The column renames (CRON_NAME→DISPLAY_NAME, CRON_FILE→TECHNICAL_NAME, CRON_SYNTAX→TIMING, CRON_STATUS→boolean IS_ACTIVE) have been applied to the data, but verify that the right crons are active.
2. Run history has been lost – CRON_RUNS is dropped; new history goes to the Elasticsearch index <integrations.elastic.index>-cron-runs (created automatically via iniESIndex; no-op when ES indexing is disabled).
3. Cloned crons no longer exist (CRON_ALIAS, CRON_IS_CLONE dropped). The alias has been migrated into the user-facing name.
4. The per-cron timeout no longer exists (CRON_TIMEOUT), and neither does the POST /cron-runs/kill endpoint or the “kill run” action. Crash recovery via stale heartbeat and the “running much longer than usual” indicator remain.
5. Cron help no longer exists – parameter documentation is now in parametersSchema and is rendered by JsonForms.
6. Set application.crons.consecutiveFailureThreshold (default 5) – after N consecutive failures, a notification e-mail is sent to the cron's error addresses.
7. Consider activating the new crons (they are created manually and are inactive by default): DatabaseHealthReportCron, DatabaseIndexRebuildCron.
8. The DeleteLogs cron has a new default name and description (“Remove old logs.” – the word “arango” has been dropped).
9. “Factory settings” on the cron detail now resets unconditionally (timing, description, parameters and active state; the user-facing name is kept).
10. Crons with parameters that do not match the schema can no longer be saved – POST /crons/create and POST /crons return 400 INVALID_CRON_PARAMETERS.
11. Behavior change: MailReportsCron treats a task as overdue when deadline < now (previously < today's midnight).

Zabbix – push → pull, reconfiguration required

Steps: generate an API token with admin scope → import tas_template.yaml → set the macro → verify LLD discovery. The scrape endpoint deliberately bypasses the isAlive/inMaintenance hooks, so monitoring keeps working even during draining/maintenance.

The keys tas.crons and tas.cronStatus and the /health-status crons actions (cronStatus/crons/cronsGui/cronsHealth) have kept their field shape – existing monitoring based on them keeps working. Removed: scheduledTasksGui, the tasZombieCron metric.

Appstatus page removed, new System Health page

The legacy Appstatus page is gone. It is replaced by Administration → System Health (/administration/system-health) with probes for: application, database, database-activity, Elasticsearch, Redis, Tika, LibreOffice, e-mail, Firebase, crons, host resources and business KPIs.

• Fleet-aware: each instance is sampled every 60 s into Redis (HEALTH_FLEET hash + per-instance series, 48h window / 49h TTL). Without Redis it degrades to the local instance only.
• Database Activity reads pg_stat_activity / sys.dm_exec_*. The MSSQL user needs VIEW SERVER STATE, otherwise the card is shown as “unknown” (previously it would print the entire failing SQL).
• New dashboard widget “System health” (admin only).

E-mail

• New msgraph transport (app-only / Client Credentials) alongside basic, oauth2 and sendmail.
• Custom X-headers in notifications with placeholder support + allowlist (TAS_MAIL_ALLOWED_CUSTOM_HEADERS).
• MS Graph mail crons: clientSecret can be a {{vault:name}} reference to a Vault secret; new “Discover folders” button (POST /ms-graph/discover-folders) and mailbox duplication.
• Newly created “Create Processes from MS Graph mails” crons have useEmailObjectInDataHolder = false (so that field mapping works). Existing crons keep their stored parameters.

Passwords and authentication

• Hashing has switched to argon2id with lazy migration from bcrypt – a user's password is rehashed on their next login. The global pepper has been removed, including the related config.
• USERS.USER_PASSWORD extended to 255 characters.
• Fallback to another authority on the default login path (without an explicit auth_id).
• Locking a user now invalidates the cache (USER2-<id>) → existing access tokens stop working immediately.
• GET /authorization/config returns paramsSchema per authority (the frontend renders forms from the JSON schema).

Breaking changes for integrations and API

Removed endpoints

Changed contracts

New endpoints

• POST /mcp/:scenario? – stateless MCP Streamable HTTP endpoint for external clients (Claude Desktop / Claude Code). Scenarios: case-creation, case-editing, case-lookup, knowledge, account, administration; a bare /mcp exposes everything. API token only (tokenPayload.kind === "api"); a session token returns UNAUTHORIZED. Every call is audited into AI_LLM_TOOL_CALL with source = mcp.
• GET /system-health, /system-health/series, /system-health/db-activity, POST /system-health/db-activity/cancel, GET /system-health/metrics
• GET /crons/available, GET /crons/monitor, POST /crons/create, /cron-runs, /cron-runs/:id
• GET /plugins/version, POST /plugins/install, POST /plugins/kill, DELETE /plugins/:sysname
• GET /plugin-config, /plugin-config/:sysname/schema, /plugin-config/:sysname/data, POST /plugin-config
• GET /scripts/list/:type, GET /scripts/mapped/:kind/:kindId?
• POST /ms-graph/discover-folders
• DELETE license, vault key-pair generation, AI observability endpoints

Post-upgrade verification checklist

Startup and basics

• The backend started without "ENV variable … is mandatory" in the log
• GET /plugins/version returns the actual version (5.19.x), not 5.18.16
• /status/liveness, /status/readiness, /status/startup OK
• Administration → System Health: all probes green or yellow, none in “unknown” due to missing permissions

Plugins

• Administration → Plugins: all expected plugins are loaded (incompatible ones are silently skipped – compare with the version list)
• helpDocs and docx-generator are installed, if the customer is supposed to have them
• KSeF: resolved

Database and migrations

• npm run db:migrations – nothing is pending, no unexpected orphan other than Migration20260528120000
• PostgreSQL: the type of secret_store.valid_from has been verified
• Certificates: SELECT count(*) FROM secret_store WHERE kind = 'certificate' matches the number of files in the original directory

Crons

• The cron list matches the pre-upgrade state (active / inactive, timing)
• No unresolved warnings from parameter normalization in the migration log
• The cron monitor is being populated; the ES index <index>-cron-runs exists
• EWS crons: configuration extracted from the [tas-3432] log entries and transferred to MS Graph crons
• Test manual run of key crons (Run manually)

Logs and monitoring

• Logs flow into Elasticsearch; no trace of the Arango configuration remains
• Zabbix scrapes GET /system-health/metrics, LLD discovery works
• Log filtering by iproc_id / tproc_id / ttask_id / header_id / cron_id works

Frontend

• Icons are displayed correctly (tas-3710 – a BOM in appStyles.css broke @font-face)
• Login, dashboard, overviews, case detail, workflow diagram
• The case diagram (new payload) renders, including archived cases

Functional verification

• Sending e-mail (notification and report)
• Print / PDF generation (LibreOffice, Puppeteer/Chromium)
• DMS upload + fulltext + previews
• AD/LDAP synchronization (AdSyncCron – fixed in 5.19; previously it silently synchronized no one)
• Templates: saving a task, saving a variable, saving a link in the diagram
• Login of a user with a bcrypt password → verify the rehash to argon2id

Scaling notes

• Plugin restart (POST /plugins/kill) takes down all backends sharing Redis, including cron workers, in a staggered manner. The deployment must start terminated containers automatically.
• The plugin volume (TAS_PLUGINS_LOCATION → featureFlag.plugins.destination) must now be writable at runtime and shared by all instances – plugin installation and uninstallation from the GUI write to it.
• On the Health page, instances are tagged as Web or Cron workers; the instance switcher is in the header.
• Global infrastructure probes (DB / ES / Redis / Tika / mail / Firebase / crons / business) cache their result in Redis for one sampling window, so that they do not run N times with N instances.

What exactly degrades without Redis (TAS_CACHE_MODE=memory):

CI variables (pipeline only, not runtime): PLUGIN_STORE_URL has been replaced by the pair PLUGIN_STORE_DEV_URL / PLUGIN_STORE_PROD_URL (.github/workflows/image-tag-push.yml) – builds from stable/* publish to the PROD store, all others to DEV. The secret CICD_PLUGIN_STORE_PRIVATE_KEY and the variable PLUGIN_STORE_USERNAME (default tas-ci) remain shared. Publishing is opt-in – until the URL is set, it is a no-op.

Rollback

Migrations delete tables and columns; down() restores only the structure, not the data. A practical rollback therefore looks like this:

1. Stop 5.19 (web and cron).
2. Restore the DB from the backup taken in step 2 of the upgrade procedure.
3. Revert to the 5.17 image and the original env (put TAS_CERTIFICATES_STORAGE_PATH back into the config, and the Arango variables if applicable).
4. Revert the plugins to their 5.17 versions.
5. Restore the original Zabbix template.