Upgrade Hubble Application Server

Running commands

Some of the following commands require you to be a root / privileged user. If you are running as a unprivileged user, you can avoid problems by running these steps as root (i.e. using the "sudo -i" command). If you really want to run them as a normal user, please prefix the commands with "sudo" as necessary.

Before starting the upgrade

It is recommended that you update your Linux operating system to get all the latest security patches.

Alerts

To retain the same alert agents from a release prior to 20.1, copy the IP addresses or DNS names of the web servers into the Alerts Agents field. To disable alerts, leave the Alerts Agents field blank.

Backup Summary

The following table lists all components that should be backed up before an upgrade:

If upgrading from version 25.3 or earlier, verify the PostgreSQL version before proceeding:

# Check PostgreSQL version

docker exec hubble_hubble_repository_1 psql -U postgres -c "SELECT version();"

# Check default SSL mode

docker exec hubble_hubble_repository_1 psql -U postgres -c "SHOW ssl;"

# Check default password encryption

docker exec hubble_hubble_repository_1 psql -U postgres -c "SHOW password_encryption;"

Back Up the Hubble Application Server

This section describes the pre-upgrade backup procedure for the Hubble Application Server. Perform a full backup before upgrading any machine that has a previous version of Hubble installed. These steps ensure a safe rollback path if issues occur during the upgrade.

Important: Container naming conventions changed between Hubble releases. Older versions use underscores (for example, hubble_hubble_repository_1), while newer versions use hyphens (for example, hubble-hubble_repository-1). Always verify the actual container name on the running system before running backup commands.

Step 1: Identify Running Containers

Container names vary between Hubble releases. Before performing any backup, list all running containers to identify the correct names.

Run the following command to list all running Hubble containers:

docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

Step 2: Back Up the PostgreSQL Database

Create a full database dump from the running Hubble repository container. This backup captures all databases, roles, and schemas, and is the most critical component for rollback.

Create the backup directory if it does not already exist:

mkdir -p /mnt/data/backups/hubble_repository

Perform the database dump:

docker exec <CONTAINER_NAME> pg_dumpall -U postgres \
> /mnt/data/backups/hubble_repository/manual_backup_$(date +%Y%m%d_%H%M%S).sql

Note: Replace container name with Postgres repository container name received in step 1 example → docker exec hubble-hubble_repository-1 pg_dumpall -U postgres > /mnt/data/backups/hubble_repository/manual_backup_$(date +%Y%m%d_%H%M%S).sql

Verify that the backup file was created:

ls -lh /mnt/data/backups/hubble_repository/

Step 3: Back Up the Hubble Configuration

Back up the Hubble configuration directory (/etc/hubble). This directory contains Docker Compose files, environment variables (.env), scripts, and all service configurations required to run the current version.

Create a timestamped copy of the configuration directory:

cp -r /etc/hubble /etc/hubble_backup_$(date +%Y%m%d_%H%M%S)

Verify that the backup directory exists:

ls /etc/ | grep hubble

The output shows the original hubble directory alongside the new timestamped backup directory.

Step 4: Back Up Vault Data

HashiCorp Vault stores all secrets used by Hubble services, including database credentials and S3 keys. Back up the Vault file storage and token files.

Back up the Vault data directory:

cp -r /mnt/data/containers/hubble_vault /mnt/data/containers/hubble_vault_backup_$(date +%Y%m%d_%H%M%S)

Verify the backup:

ls /mnt/data/containers/ | grep hubble_vault

Step 5: Back Up S3 Object Storage (Scality)

Scality S3 stores files such as approval snapshots, user photos, image views, generated PDFs, and temporary exports. Back up both the metadata and data directories.

Back up the Scality storage directories:

cp -r /mnt/data/containers/hubble_storage /mnt/data/containers/hubble_storage_backup_$(date +%Y%m%d_%H%M%S)

Verify the backup:

ls /mnt/data/containers/ | grep hubble_storage

Note: Depending on the volume of stored files (reports, snapshots, photos), this operation may take significant time and disk space. Ensure sufficient disk space is available before proceeding.

Step 6: Back Up SSL Certificates

If custom SSL certificates are configured for the HAProxy load balancer, back them up separately.

Back up the certificates directory:

cp -r /mnt/data/containers/hubble_loadbalancer/certificates /mnt/data/containers/hubble_loadbalancer/certificates_backup_$(date +%Y%m%d_%H%M%S)

Verify the backup:

ls /mnt/data/containers/hubble_loadbalancer/ | grep certificates

Step 7: Back Up Configuration UI State

The configuration wizard stores the last submitted configuration, which is used during re-installation.

Back up the last configuration state:

cp /mnt/data/containers/configuration/lastconfig.json \

/mnt/data/containers/configuration/lastconfig_backup_$(date +%Y%m%d_%H%M%S).json

Step 8: Rename the PostgreSQL Data Directory

Rename the existing PostgreSQL data directory to preserve it for rollback. This allows the upgrade to initialize a fresh data directory while keeping the existing one intact.

Rename the data directory:

# Check PostgreSQL version 
docker exec hubble-hubble_repository-1 psql -U postgres -c "SELECT version();"
mv /mnt/data/containers/hubble_repository/data \
 /mnt/data/containers/hubble_repository/data_<PG_VERSION>_backup_$(date +%Y%m%d)

Note: Replace <PG_VERSION> with the PostgreSQL version, for example, 15.2.

Verify that the renamed directory is present:

ls /mnt/data/containers/hubble_repository/

The output shows the renamed directory (for example, data_9.5_backup_20260325) and no data directory. The upgrade process creates a new data directory automatically.

Application Server Upgrade Steps

1. Obtain the deployment package (HubbleApplicationServer-*.tar.gz) and copy it to the Linux machine at the location where you want to deploy the Hubble Application Server.

2. Create the directory:

   mkdir -p /etc/hubble
   

3. Extract into it:

   tar zxvf HubbleApplicationServer-<version>.tar.gz -C /etc/hubble
   

4. Stop all running containers:

   cd /etc/hubble && ./stop.sh 
   docker ps | grep hubble 
   All containers should be stopped
   

5. Verify that all prerequisites are met:

   /etc/hubble/pre-req-tests.sh

6. The results of the checks are stored in log files at /var/log/hubble. Log files use the format hubble-install-date_and_time.log. Send the most recent log file to the Hubble Support team before the deployment day.

On Deployment Day

Restore run-list.json:

Update the Hubble Configurations

You will always need to re-submit your configurations using these steps, even if there are no changes to apply.

1. Start the Hubble Configuration UI by running:

   /etc/hubble/Configuration/start.sh

2. Go to the Hubble Configuration form by accessing http:// <application_server_ip_address>:3000/ in a browser.
3. Fill in the form with the configuration details of your server(s).

   Note:

   • Starting from version 26.1, the Application Server IP Address/DNS Name field accepts both an IP address and a DNS name.

   • If you are upgrading from any Hubble release, older than 20.1, you will need to configure the alerts agents for your system in the configuration UI.
   • If you want the same alerts agents as you had prior to the upgrade, then copy the IP addresses/DNS names for your web servers into the alerts agents field (see the image below).
   • If you had alerts and no longer want them, leave the alerts agent field blank. The same goes for if you want no alerts on a new install.
4. Once finished, press the Submit and Download button.
5. You can now the stop the Hubble Configuration UI by running:

   /etc/hubble/Configuration/stop.sh

   Important: If you are upgrading from any Hubble release older than 19.2, you will need to change the docker storage driver to overlay2:

   /etc/hubble/configure-overlay2.sh

   <name_of_device_for_docker_use>

   This command will format the drive, so please ensure you select the correct device. As an example, if device sdb has been provisioned for docker use, run:

       /etc/hubble/configure-overlay2.sh sdb    

Restart the Hubble Application Server Services

1. Submitting your server information in the Hubble Configuration UI will automatically generate all the files needed to run the Hubble Application Server, so you are now ready to start the services.
2. (Optional) If you selected the use of HTTPS (under "Web Protocol" in the Configuration UI), you will need to provide a valid certificate at this point by following the instructions in How to setup HTTPS in the Hubble Supplementary Deployment Topics guide. If you do not have a valid certificate yet and want to use HTTP temporarily, repeat the steps in "Generate the Hubble Configurations" above, and select HTTP as the Web Protocol.
3. To start the Hubble service, run the following command:

   /etc/hubble/start.sh

   Note:

   This script is the only way to start the docker containers.

   The script will always check the prerequisites and will not start the Hubble services until all the prerequisites are met.

   If the start.sh script failed due to the prerequisites not being met, you can carry out corrective measures and re-test them by running the following command:

   /etc/hubble/pre-req-tests.sh

Post-Upgrade Verification

After the upgrade completes, run the following checks to confirm the system is healthy.

PostgreSQL Verification

Check the new PostgreSQL version:

docker exec <CONTAINER_NAME> psql -U postgres -c "SELECT version();"

Verify the on-disk PostgreSQL version:

cat /mnt/data/containers/hubble_repository/data/PG_VERSION

Verify SSL is enabled:

docker exec <CONTAINER_NAME> psql -U postgres -c "SHOW ssl;"

Verify password encryption method:

docker exec <CONTAINER_NAME> psql -U postgres -c "SHOW password_encryption;"

Service Health Check

Verify all containers are running:

docker ps | grep hubble

All Hubble containers should show Up status. If any container is restarting or missing, check its logs with docker logs <CONTAINER_NAME>.

Note: The upgrade guide does not cover web server setup. Once the Application Server upgrade is complete, follow the web server setup steps described in the Hubble Desktop and Web Deployment Guide.

Post-Installation

After the Web Server deployment, remove the file run-list.json from the S3 service:

/etc/hubble/remove_run_list.sh