Back Up Hubble

Hubble automatically backs up its repository and storage service data on the application server at:

The repository contains definitions of ERP profiles, database connections, users, groups, workspaces, reports, metrics, discussions, and approvals/distribution workflows. This is almost entirely meta-data, but reports may contain subsets of ERP data in the form of cached results. The storage service contains PDF and HTML representations of workspaces and reports, generated for the approvals and distribution features, any profile pictures uploaded by users, and images uploaded for use in workspace image views.

Add an NFS Mount

Adding an NFS mount to a /mnt/data/backups directory allows the backups to be accessed from a network file system.

Hubble takes backups of the repository and of the storage services, and saves one backup at a time. The backup directory stores these backups in the following directories:

...where hubble_repository holds repository backups, and hubble_s3_service holds storage services backups.

Important: Do not modify the structure of these directories. Doing so may effect future backups and any restoration attempts.

While it is not mandatory, mounting this directory enables in this way you to retrieve these backups from an outside machine. It is HIGHLY RECOMMENDED that you configure the NFS mount to help prevent the loss of data in an event of a disaster involving the Application Server.

Configure Backups using the Hubble Configuration Form

Backups are configured using the Hubble Configuration form during deployment. During configuration, you are given the option to specify the frequency at which you require backups to take place.

You can choose daily, weekly or monthly backups, and a specific time on these occasions.

Ideally this time would be when Hubble is not expected to be in use, so that the backup process is less likely to be interrupted.

PostgreSQL Backup Script Changes (Version 25.4+)

Important: Starting with Hubble release 25.4, the pg_dump backup.sh file will undergo changes.

Create or update the backup script at /etc/hubble/backup_postgres.sh with the following content.

#!/bin/bash 
set -e

ENV_NAME=$(grep -oP 'S3_STORAGE_ENDPOINT=\K.*' /etc/hubble/.env)
ENV_VERSION=$(grep -oP 'HUBBLE_VERSION=\K.*' /etc/hubble/.version)
DATE=$(date +%Y-%m-%d)

# Find PostgreSQL container
PG_CONTAINER=$(docker ps --format "{{.Names}}" | grep -E "repository" | grep -v "api\|scheduler" | head -1)
echo "Using PostgreSQL container: $PG_CONTAINER"

# Backup PostgreSQL directly from container (has pg_dump built-in
echo "Backing up PostgreSQL repository..."
BACKUP_DIR="/mnt/data/backups/hubble_repository"
mkdir -p $BACKUP_DIR

docker exec -e PGSSLMODE=require $PG_CONTAINER \

pg_dump -U hubble -d hubble_repository -F custom \
> $BACKUP_DIR/hubble_repository-$DATE.bak

if [ $? -eq 0 ]; then
echo "✅ PostgreSQL repository backup successful"
else

echo "❌ PostgreSQL repository backup failed"
exit 1
fi

# Run S3 and Vault backups from job scheduler
echo "Running S3 and Vault backups..."
docker exec hubble_hubble_job_scheduler_1 sh -c "
/scripts/backup_s3.sh
/scripts/backup_vault.sh
"

# Create tar archive
echo "Creating backup archive..."
tar --exclude='./installs' -zcvf "$ENV_NAME-$ENV_VERSION-$DATE.tar.gz" -C /mnt/data/backups/ .
echo "✅ Backup completed: $ENV_NAME-$ENV_VERSION-$DATE.tar.gz"

Schedule Recurring Jobs Using the Configuration UI

The Configuration UI provides a Scheduled Jobs interface where administrators configure recurring jobs for database backups, maintenance operations, and view filter cleanup. These jobs execute automatically on the defined schedule without requiring manual intervention.

About Scheduled Jobs

The Configuration UI supports three job types:

• Database Backup — Creates automated backups of the Hubble repository database on a recurring schedule.
• Database Maintenance — Performs vacuum and analyze operations on the PostgreSQL database to optimize performance and maintain database health.
• Clean View Filters — Removes expired or unused filter definitions from the system.

For new Hubble installations, a default Database Backup job is automatically created and configured to run daily at 12:00 AM (midnight) in the system timezone. Administrators can modify the schedule or disable this job as needed.

Access the Scheduled Jobs Interface

To access the Scheduled Jobs interface, start the Configuration UI Container on the Application Server by running:

Open a web browser and navigate to http://<application_server_ip_address>:3000/, where <application_server_ip_address> is the IP address or hostname of your Hubble Application Server. Click the Scheduled Jobs tab.

Create a Scheduled Job

On the Scheduled Jobs page, click Add New Schedule. Select the job type from the Job Type dropdown (Database Backup, Database Maintenance, or Clean View Filters). Enter a descriptive name in the Name field, such as "Daily Backup at Midnight" or "Weekly Maintenance".

Enter the schedule using cron expression syntax in the Cron field. The cron expression format is minute hour day_of_month month day_of_week. For example, 0 2 * * * runs at 2:00 AM every day, 30 22 * * 0 runs at 10:30 PM every Sunday, and 0 0 1 * * runs at midnight on the first day of each month.

Select the timezone from the Timezone dropdown. The job schedule uses this timezone to determine execution time. Click Save. If the job creation encounters an error, check the Configuration UI Container logs for scheduling-related errors. Errors related to job execution after the job has started can be found in the Job Scheduler container logs.

Manage Scheduled Jobs

The Scheduled Jobs table displays all configured jobs. The table columns include Name (the user-defined name assigned to the job), Job Type (the type of job), Cron (the cron expression that defines the job schedule), Timezone (the timezone used for job execution), Status (whether the job is Enabled or Disabled), and Actions (provides buttons to edit, pause, or delete the job).

To edit a scheduled job, locate the job in the Scheduled Jobs table and click the pencil (edit) icon in the Actions column. Modify the job name, cron expression, or timezone as needed and click Save.

To disable a scheduled job, locate the job in the Scheduled Jobs table and click the pause (orange) icon in the Actions column. The job status changes to Disabled and the job does not execute on its scheduled time. To re-enable the job, click the pause icon again.

To delete a scheduled job, locate the job in the Scheduled Jobs table and click the delete (red trash) icon in the Actions column. Confirm the deletion when prompted.

Job Execution and Monitoring

After configuring a scheduled job, the job executes automatically at the specified time according to the cron expression and timezone. Configuration UI Container logs contain errors related to scheduling the job, such as invalid cron syntax or timezone issues. Job Scheduler container logs contain errors that occur during or after job execution, such as database connectivity issues or insufficient disk space.

Job Migration During Upgrade

When upgrading to a new Hubble version, all scheduled jobs and their configurations are preserved. The configuration is backed up in the lastconfig.json file. When the new version initializes and the Configuration UI is resubmitted with your server configuration, the scheduled jobs are automatically restored with the same settings. If you are upgrading from Hubble release 25.4 or earlier, the Scheduled Jobs feature was not available and no job configurations are present in the configuration backup. All scheduled jobs must be recreated in the new version. See the About Scheduled Jobs in the Backup section in the Upgrade Guide for migration details.

Stop the Configuration UI

After configuring scheduled jobs, stop the Configuration UI Container by running:

Update maintenance_repository.sh to Enable REINDEX and Use the Repository Database

Background

The maintenance_repository.sh script currently attempts to execute:

This causes PostgreSQL to treat CURRENT_DATABASE as a literal database name instead of reindexing the configured Hubble repository database.

The maintenance script must instead use the configured PostgreSQL repository database (${POSTGRES_DB}) and enable the REINDEX operation by setting MAINTENANCE_REINDEX=true.

Note: This update resolves an issue where reindexing fails with the error: ERROR: can only reindex the currently open database.

Update the Script

1. Access the Job Scheduler Container

   Navigate to the Hubble containers directory and open a shell inside the Job Scheduler container.

   cd /hubble/data/containers/containers

   sudo docker exec -it hubble-hubble_job_scheduler-1 /bin/bash

   cd /scripts

   Verify that the maintenance script exists.

   ls -l maintenance_repository.sh

2. Backup the Existing Script

   Before making any modifications, create a backup.

   cp maintenance_repository.sh maintenance_repository.sh.bak

3. Enable REINDEX

   Open the script for editing.

   vi maintenance_repository.sh

   Locate the following configuration:

   MAINTENANCE_REINDEX=false

   Update it to:

   MAINTENANCE_REINDEX=true

   If the variable is exported, update:

   export MAINTENANCE_REINDEX=false

   to

   export MAINTENANCE_REINDEX=true

4. Update the REINDEX Command

   Locate the REINDEX section of the script.

   Replace:

   psql -h ${POSTGRES_HOST} \
   -p ${POSTGRES_PORT} \
   -U ${POSTGRES_USER} \
   -d ${POSTGRES_DB} \
   -c "REINDEX DATABASE CURRENT_DATABASE;"

   with

   psql -h ${POSTGRES_HOST} \
   -p ${POSTGRES_PORT} \
   -U ${POSTGRES_USER} \
   -d ${POSTGRES_DB} \
   -c "REINDEX DATABASE \"${POSTGRES_DB}\";"

   This ensures the REINDEX operation is executed against the configured Hubble repository database instead of the literal database name CURRENT_DATABASE.

5. Save the Changes

   Save and exit the editor.

   Esc
   :wq

6. Restart the Job Scheduler Container

   Exit the container.

   exit

   Restart the Job Scheduler container.

   sudo docker restart hubble-hubble_job_scheduler-1

   Verify that the container is running.

   sudo docker ps

7. Verify the Scheduled Maintenance Job

   Allow the maintenance job to execute at its scheduled time.

   Review the Job Scheduler logs.

   sudo docker logs hubble-hubble_job_scheduler-1 --tail 100

   A successful execution produces output similar to:

   Running VACUUM ANALYZE on database hubble_repository...
   VACUUM
   VACUUM ANALYZE completed successfully.
   
   Running REINDEX DATABASE hubble_repository...
   REINDEX
   REINDEX DATABASE completed successfully.
   
   Completed /scripts/maintenance_repository.sh.
   Result: Success

Summary of Changes

Expected Result

• The maintenance job performs both VACUUM ANALYZE and REINDEX DATABASE on the configured Hubble repository database.
• The scheduled maintenance completes successfully without attempting to reindex a nonexistent CURRENT_DATABASE.
• The Job Scheduler logs display successful completion of both operations.

Back up Planning Data

Once a deployment is complete, planning repository can be backed up using the Hubble Administration tool.

Right-click on the profile and select Backup Budget Database from the menu. This will allow a backup of the planning data to be stored as a .bdb file. Planning data can be restored using this same menu. This backup includes the definitions of planning ledgers, activities, cycles, participants, and scope. It also includes the rows of data entered by users against budgeting and planning tables but it does not include the definitions of custom planning tables.

Back up the Data Drive

By following the instructions above you will get the data backed up in the NFS mount or data drive, but the data drive itself could be compromised.

Important: Backing up the data drive

The data contained in the data drive must not be lost, otherwise the Hubble state will be compromised.

It is therefore very important that this data drive is backed up frequently. That frequency will determine the Recovery Point Objective for Hubble Web.

There should be a backup/restore strategy in place before Hubble is deployed.