Upgrade Hubble Application Server

Running commands

Some of the following commands require you to be a root / privileged user. If you are running as a unprivileged user, you can avoid problems by running these steps as root (i.e. using the "sudo -i" command). If you really want to run them as a normal user, please prefix the commands with "sudo" as necessary.

Before starting the upgrade

It is recommended that you update your Linux operating system to get all the latest security patches.

If your deployment uses Platform authentication, configure your firewall to allow outbound HTTPS traffic to the required insightsoftware Platform domains before starting the upgrade. For the list of domains, see Configure Firewall Allowlist for Platform Integration in the Hubble Web System Requirements and Architecture Scalability Guide section.

Alerts

To retain the same alert agents from a release prior to 20.1, copy the IP addresses or DNS names of the web servers into the Alerts Agents field. To disable alerts, leave the Alerts Agents field blank.

Backup Summary

The following table lists all components that should be backed up before an upgrade:

Note: These backup paths assume an install still using /mnt/data (any version prior to 26.2). If you are upgrading from 26.2 or later, your data already lives at /hubble/data — substitute /hubble/data for /mnt/data in all backup commands below.

If upgrading from version 25.3 or earlier, verify the PostgreSQL version before proceeding:

Note: The following checks apply regardless of whether your installation uses /mnt/data or /hubble/data mount paths. These commands verify PostgreSQL configuration settings inside the container and are required before proceeding with the upgrade.

# Check PostgreSQL version

docker exec hubble_hubble_repository_1 psql -U postgres -c "SELECT version();"

# Check default SSL mode

docker exec hubble_hubble_repository_1 psql -U postgres -c "SHOW ssl;"

# Check default password encryption

docker exec hubble_hubble_repository_1 psql -U postgres -c "SHOW password_encryption;"

Back Up the Hubble Application Server

This section describes the pre-upgrade backup procedure for the Hubble Application Server. Perform a full backup before upgrading any machine that has a previous version of Hubble installed. These steps ensure a safe rollback path if issues occur during the upgrade.

Important: Container naming conventions changed between Hubble releases. Older versions use underscores (for example, hubble_hubble_repository_1), while newer versions use hyphens (for example, hubble-hubble_repository-1). Always verify the actual container name on the running system before running backup commands.

Step 1: Identify Running Containers

Container names vary between Hubble releases. Before performing any backup, list all running containers to identify the correct names.

Run the following command to list all running Hubble containers:

docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

Step 2: Back Up the PostgreSQL Database

Create a full database dump from the running Hubble repository container. This backup captures all databases, roles, and schemas, and is the most critical component for rollback.

Create the backup directory if it does not already exist:

mkdir -p /hubble/data/backups/hubble_repository

Perform the database dump:

docker exec <CONTAINER_NAME> pg_dumpall -U postgres \
> /hubble/data/backups/hubble_repository/manual_backup_$(date +%Y%m%d_%H%M%S).sql

Note: Replace container name with Postgres repository container name received in step 1 example → docker exec hubble-hubble_repository-1 pg_dumpall -U postgres > /hubble/data/backups/hubble_repository/manual_backup_$(date +%Y%m%d_%H%M%S).sql

Verify that the backup file was created:

ls -lh /hubble/data/backups/hubble_repository/

Step 3: Back Up the Hubble Configuration

Back up the Hubble configuration directory (/etc/hubble). This directory contains Docker Compose files, environment variables (.env), scripts, and all service configurations required to run the current version.

Create a timestamped copy of the configuration directory:

cp -r /etc/hubble /etc/hubble_backup_$(date +%Y%m%d_%H%M%S)

Verify that the backup directory exists:

ls /etc/ | grep hubble

The output shows the original hubble directory alongside the new timestamped backup directory.

Step 4: Back Up Vault Data

HashiCorp Vault stores all secrets used by Hubble services, including database credentials and S3 keys. Back up the Vault file storage and token files.

Back up the Vault data directory:

cp -r /hubble/data/containers/hubble_vault /hubble/data/containers/hubble_vault_backup_$(date +%Y%m%d_%H%M%S)

Verify the backup:

ls /hubble/data/containers/ | grep hubble_vault

Step 5: Back Up S3 Object Storage (Scality)

Scality S3 stores files such as approval snapshots, user photos, image views, generated PDFs, and temporary exports. Back up both the metadata and data directories.

Back up the Scality storage directories:

cp -r /hubble/data/containers/hubble_storage /hubble/data/containers/hubble_storage_backup_$(date +%Y%m%d_%H%M%S)

Verify the backup:

ls /hubble/data/containers/ | grep hubble_storage

Note: Depending on the volume of stored files (reports, snapshots, photos), this operation may take significant time and disk space. Ensure sufficient disk space is available before proceeding.

Step 6: Back Up SSL Certificates

If custom SSL certificates are configured for the HAProxy load balancer, back them up separately.

Back up the certificates directory:

cp -r /hubble/data/containers/hubble_loadbalancer/certificates /hubble/data/containers/hubble_loadbalancer/certificates_backup_$(date +%Y%m%d_%H%M%S)

Verify the backup:

ls /hubble/data/containers/hubble_loadbalancer/ | grep certificates

Step 7: Back Up Configuration UI State

The configuration wizard stores the last submitted configuration, which is used during re-installation.

Back up the last configuration state:

cp /hubble/data/containers/configuration/lastconfig.json \

/hubble/data/containers/configuration/lastconfig_backup_$(date +%Y%m%d_%H%M%S).json

About Scheduled Jobs in the Backup

The lastconfig.json file contains not only general Configuration UI settings (IP addresses, web protocol, email definitions) but also all scheduled job configurations (Database Backup, Database Maintenance, and Clean View Filters jobs). These job definitions include the job name, job type, cron expression, timezone, and enabled/disabled status.

Scheduled jobs are preserved and restored during upgrade. When you resubmit the Configuration UI form during the upgrade process (as described in the "Update the Hubble Configurations" section of this guide), all scheduled jobs are automatically restored to the new Hubble version with their original settings. No additional steps are required to restore scheduled jobs.

Note: If you are upgrading from Hubble release 25.4 or earlier, the Scheduled Jobs feature was not available and no job configurations are present in the configuration backup. All scheduled jobs must be recreated in the new version.

Step 8: Rename the PostgreSQL Data Directory

Rename the existing PostgreSQL data directory to preserve it for rollback. This allows the upgrade to initialize a fresh data directory while keeping the existing one intact.

Rename the data directory:

# Check PostgreSQL version 
docker exec hubble-hubble_repository-1 psql -U postgres -c "SELECT version();"
mv /hubble/data/containers/hubble_repository/data \
 /hubble/data/containers/hubble_repository/data_<PG_VERSION>_backup_$(date +%Y%m%d)

Note: Replace <PG_VERSION> with the PostgreSQL version, for example, 15.2.

Verify that the renamed directory is present:

ls /hubble/data/containers/hubble_repository/

The output shows the renamed directory (for example, data_9.5_backup_20260325) and no data directory. The upgrade process creates a new data directory automatically.

Note: The manual backup steps above remain available for reference. To automate the backup process, two scripts are now available in 26.2:

Run the following command to take a full backup before upgrading:

Run the following command to check the existing backup status:

Upgrade the Application Server

1. Change to a working directory for the download:

   cd /tmp

2. Stop all running containers:

   /etc/hubble/stop.sh docker ps | grep hubble

   All containers should be stopped.

3. Preserve the existing installation for rollback by moving it aside:

   sudo mv /etc/hubble /etc/hubble_old ls -ld /etc/hubble_old ls -ld /etc/hubble

   The first listing confirms the previous installation is preserved at /etc/hubble_old, and the second confirms the original /etc/hubble is gone.

4. Obtain the deployment package (HubbleApplicationServer-*.tar.gz) and copy it to the Linux machine at the location where you want to deploy the Hubble Application Server.

5. Create the directory and extract the package into it:

   mkdir -p /etc/hubble && tar zxvf HubbleApplicationServer-<version>.tar.gz -C /etc/hubble

6. Migrate the mount paths (required only for installations still using /mnt):

   /etc/hubble/migrate-paths.sh

   This script detects data at /mnt/data and /mnt/docker, remounts it to /hubble/data and /hubble/docker, updates fstab, daemon.json, the .env files, and the deployed scripts, then restarts Docker. The script is a safe no-op if the installation already uses /hubble.

7. Verify that all prerequisites are met:

   /etc/hubble/pre-req-tests.sh

8. The results of the checks are stored in log files at /var/log/hubble. Log files use the format hubble-install-date_and_time.log. Send the most recent log file to the Hubble Support team before the deployment day.

Upgrade Hubble Without Internet Access

Starting with 26.2, Docker images ship as a self-contained bundle (hubble-docker-images-<version>.tar.gz). The bundle contains images covering all services in the stack. The full list of images is recorded in manifest.txt inside the bundle.

Use --offline when the target machine cannot reach Docker Hub, for example when outbound access to Cloudflare or registry IPs is blocked by a firewall. If network access to Docker Hub is restored later, use --online to re-enable image pulls.

Upgrade Steps

1. Transfer hubble-docker-images-<VERSION>.tar.gz to the app server, for example via WinSCP or USB.

2. Extract the archive in any directory:

   tar -xzf hubble-docker-images-<VERSION>.tar.gz

3. Load the Docker images:

   sudo ./hubble-load-images.sh

4. Start app server installation in offline mode:

   sudo ./start.sh --offline

Note: Offline mode persists automatically across reboots. Once --offline is used, systemd service restarts continue operating in offline mode without requiring the flag again.

Scripts Supporting Offline Mode

Note: When running via install.sh --offline or start.sh --offline, the flag propagates automatically. If Configuration/start.sh is run directly, pass --offline explicitly:

Switch Back to Online Mode

Offline mode is persisted via the marker file /hubble/data/.offline. As long as this file exists, every startup, including systemd service restarts, operates in offline mode without requiring the flag.

To return to online mode, choose one of the following options:

Option 1: Remove the Marker and Restart Containers

Use this option when you are ready to bring the system online immediately:

This removes /hubble/data/.offline and performs a full container startup, pulling images and starting all services.

Option 2: Remove the Marker Only

Use this option to remove the marker without restarting containers immediately:

Flag Reference

On Deployment Day

Restore run-list.json:

Update the Hubble Configurations

You will always need to re-submit your configurations using these steps, even if there are no changes to apply.

1. Start the Hubble Configuration UI by running:

   /etc/hubble/Configuration/start.sh

2. Go to the Hubble Configuration form by accessing http:// <application_server_ip_address>:3000/ in a browser.
3. Fill in the form with the configuration details of your server(s).

   Note:

   • Starting from version 26.1, the Application Server IP Address/DNS Name field accepts both an IP address and a DNS name.

   • If you are upgrading from any Hubble release, older than 20.1, you will need to configure the alerts agents for your system in the configuration UI.
   • If you want the same alerts agents as you had prior to the upgrade, then copy the IP addresses/DNS names for your web servers into the alerts agents field (see the image below).
   • If you had alerts and no longer want them, leave the alerts agent field blank. The same goes for if you want no alerts on a new install.
4. Once finished, press the Submit and Download button.
5. You can now the stop the Hubble Configuration UI by running:

   /etc/hubble/Configuration/stop.sh

   Important: If you are upgrading from any Hubble release older than 19.2, you will need to change the docker storage driver to overlay2:

   /etc/hubble/configure-overlay2.sh

   <name_of_device_for_docker_use>

   This command will format the drive, so please ensure you select the correct device. As an example, if device sdb has been provisioned for docker use, run:

       /etc/hubble/configure-overlay2.sh sdb    

   Note: The configure-overlay2.sh script requires a dedicated unused block device. Ensure an unused block device is available before running this script.

   Tip: Use a separate device exclusively for Docker storage.

Update maintenance_repository.sh to Enable REINDEX and Use the Repository Database

After you update the Hubble configurations in the Configuration UI, you must apply a fix to the maintenance_repository.sh script to ensure database reindexing operates correctly in Hubble v26.2.

Background

The maintenance_repository.sh script currently attempts to execute:

This causes PostgreSQL to treat CURRENT_DATABASE as a literal database name instead of reindexing the configured Hubble repository database.

The maintenance script must instead use the configured PostgreSQL repository database (${POSTGRES_DB}) and enable the REINDEX operation by setting MAINTENANCE_REINDEX=true.

Note: This update resolves an issue where reindexing fails with the error: ERROR: can only reindex the currently open database.

Update the Script

1. Access the Job Scheduler Container

   Navigate to the Hubble containers directory and open a shell inside the Job Scheduler container.

   cd /hubble/data/containers/containers

   sudo docker exec -it hubble-hubble_job_scheduler-1 /bin/bash

   cd /scripts

   Verify that the maintenance script exists.

   ls -l maintenance_repository.sh

2. Backup the Existing Script

   Before making any modifications, create a backup.

   cp maintenance_repository.sh maintenance_repository.sh.bak

3. Enable REINDEX

   Open the script for editing.

   vi maintenance_repository.sh

   Locate the following configuration:

   MAINTENANCE_REINDEX=false

   Update it to:

   MAINTENANCE_REINDEX=true

   If the variable is exported, update:

   export MAINTENANCE_REINDEX=false

   to

   export MAINTENANCE_REINDEX=true

4. Update the REINDEX Command

   Locate the REINDEX section of the script.

   Replace:

   psql -h ${POSTGRES_HOST} \
   -p ${POSTGRES_PORT} \
   -U ${POSTGRES_USER} \
   -d ${POSTGRES_DB} \
   -c "REINDEX DATABASE CURRENT_DATABASE;"

   with

   psql -h ${POSTGRES_HOST} \
   -p ${POSTGRES_PORT} \
   -U ${POSTGRES_USER} \
   -d ${POSTGRES_DB} \
   -c "REINDEX DATABASE \"${POSTGRES_DB}\";"

   This ensures the REINDEX operation is executed against the configured Hubble repository database instead of the literal database name CURRENT_DATABASE.

5. Save the Changes

   Save and exit the editor.

   Esc
   :wq

6. Restart the Job Scheduler Container

   Exit the container.

   exit

   Restart the Job Scheduler container.

   sudo docker restart hubble-hubble_job_scheduler-1

   Verify that the container is running.

   sudo docker ps

7. Verify the Scheduled Maintenance Job

   Allow the maintenance job to execute at its scheduled time.

   Review the Job Scheduler logs.

   sudo docker logs hubble-hubble_job_scheduler-1 --tail 100

   A successful execution produces output similar to:

   Running VACUUM ANALYZE on database hubble_repository...
   VACUUM
   VACUUM ANALYZE completed successfully.

   
   Running REINDEX DATABASE hubble_repository...
   REINDEX
   REINDEX DATABASE completed successfully.
   
   Completed /scripts/maintenance_repository.sh.
   Result: Success

Summary of Changes

Expected Result

• The maintenance job performs both VACUUM ANALYZE and REINDEX DATABASE on the configured Hubble repository database.
• The scheduled maintenance completes successfully without attempting to reindex a nonexistent CURRENT_DATABASE.
• The Job Scheduler logs display successful completion of both operations.

Restart the Hubble Application Server Services

1. Submitting your server information in the Hubble Configuration UI will automatically generate all the files needed to run the Hubble Application Server, so you are now ready to start the services.
2. (Optional) If you selected the use of HTTPS (under "Web Protocol" in the Configuration UI), you will need to provide a valid certificate at this point by following the instructions in How to setup HTTPS in the Hubble Supplementary Deployment Topics guide. If you do not have a valid certificate yet and want to use HTTP temporarily, repeat the steps in "Generate the Hubble Configurations" above, and select HTTP as the Web Protocol.
3. To start the Hubble service, run the following command:

   /etc/hubble/start.sh

   Note:

   This script is the only way to start the docker containers.

   The script will always check the prerequisites and will not start the Hubble services until all the prerequisites are met.

   If the start.sh script failed due to the prerequisites not being met, you can carry out corrective measures and re-test them by running the following command:

   /etc/hubble/pre-req-tests.sh

Post-Upgrade Verification

After the upgrade completes, run the following checks to confirm the system is healthy.

PostgreSQL Verification

Check the new PostgreSQL version:

docker exec <CONTAINER_NAME> psql -U postgres -c "SELECT version();"

Verify the on-disk PostgreSQL version:

			cat /hubble/data/containers/hubble_repository/data/PG_VERSION
		

Verify SSL is enabled:

docker exec <CONTAINER_NAME> psql -U postgres -c "SHOW ssl;"

Verify password encryption method:

docker exec <CONTAINER_NAME> psql -U postgres -c "SHOW password_encryption;"

Service Health Check

Verify all containers are running:

docker ps | grep hubble

All Hubble containers should show Up status. If any container is restarting or missing, check its logs with docker logs <CONTAINER_NAME>.

Note: The upgrade guide does not cover web server setup. Once the Application Server upgrade is complete, follow the web server setup steps described in the Hubble Desktop and Web Deployment Guide.

Post-Installation

After the Web Server deployment, remove the file run-list.json from the S3 service:

/etc/hubble/remove_run_list.sh

SSL Certificate and Port 443 Binding (Web Server)

After you upgrade the Hubble Web Server, verify that an SSL certificate is created in IIS and bound to port 443.

Starting with Hubble 26.2, port 443 is mandatory and all Hubble Web traffic routes through port 443 (HTTPS). Both ports 80 and 443 must be available in IIS for Hubble Web to function correctly.

For SSL certificate setup instructions, refer to Create an SSL Certificate and Bind to Port 443 in the Hubble Desktop and Web Deployment Guide.