Perform Manual Monitoring

Hubble Web Server - Windows

Check Insight Processes

Services that should be running:

• alerts_service

Processes that should be running:

• InsightYbService.exe
• InsightAlertMonitoringServerProcess.exe

Folders and Files that must be present

Note: The RepositorySelection.xml file is in two locations. Both of these files should have the same contents.The parameters for exec have the following functions:

Check IIS

The Hubble web site is hosted by the Internet Information Services (IIS) server. This is a Windows component.

Launching IIS will display the dialog below. There should be a “hubble” application located under the “Default Web Site” node:

Troubleshooting Instructions

• Examine the logs for issues. Locations of the logs:
  • Logs for the deployment process: C:\chef\solo.log
  • Logs for the Hubble application: C:\Windows\Temp\yellowbox-log.txt
  • Logs for the Hubble application: C:\Windows\Temp\yellowbox-ws-log.txt
• Restart the application pools (“.Net v4.5” and “DefaultAppPool”). Instructions can be found here: https://www.iis.net/configreference/system.applicationhost/applicationpools/add/recycling

  

• Restart the IIS web server. Instructions can be found here:
  https://technet.microsoft.com/en-us/library/jj635851(v=ws.11).aspx

Hubble Application Server - Linux

Minimum Required Containers List

Check whether all the containers are running (run docker ps -a):

This will list running containers along with their id, base image, port(s) mapped/used, name and other status information.

You can also check the status of docker-compose, which among other tasks ensures that the containers are restarted on reboot. To do this go to /etc/hubble and run the docker-compose ps:

Testing Containers

• hubble_queues

Check that the web page is active by pointing a browser at:

Test URL

http://<replace_with_hubble_administration_ip>:15672

• hubble_cache

  Run the following command to check whether Redis is responding:

  Command to run

  docker exec hubble_hubble_cache_1 redis-cli client list

• hubble_repository

  Connect directly with a PostgreSQL client (e.g. pgAdmin). For pgAdmin, you can use the following credentials and check whether you get a password prompt (instead of an error message):

  pgAdmin credentials

  Host: Hubble Administration Host IP address

  Port: 5432 Username: postgres

• hubble_storage

  Use a client tool that can connect to a S3 Compatible storage service (e.g. CloudBerry Explorer for Amazon S3), and check whether you can connect. On Cloudberry Explorer, just add a new "S3 Compatible" account with the following:

  CloudBerry credentials

  Service point:

  <containers_ip_address>:591

  Access key:

  run docker exec hubble_hubble_storage_1 cat /var/lib/hubble/admin.key in the Hubble Administration machine and use the output

  Secret key:

  run docker exec hubble_hubble_storage_1 cat /var/lib/hubble/admin.secret in the Hubble Administration machine and use the output

Drives that should be Mounted and the Folders and Files that should be present

Docker allows the mapping of folders from the host to inside a container. The following table contains a list of mappings used:

Use this to inspect what the container contents is without having to use a console to access the container.

Note: Modification of these files will directly affect a container and should only be attempted with caution, or not at all.

Location of Logs

The service(s) hosted within a container will output information to log files.

The location of these logs files (within the container) are shown in the following table:

There are plans to centralise logs at a later date.

Troubleshooting

• Restarting a container

  To restart a container, use the Docker restart command:

  Restarting a container

  docker restart <container_name>

  The response should be the name of the container.

  

• Running a console in the container

To run a console within a container, use the Docker exec command:

Running a console within a container

docker exec -it <container_name> sh

The parameters for exec have the following functions:

-i : makes the execution 'interactive'.

-t : does not allocate a pseudo-tty, which is not required as we use our own.

The final part of the command specifies the command to run, which in the example above is 'sh'. This could be replaced by another linux shell, such as “bash”, which has more features such as tab-completion.

Once the console is running, commands can be executed directly in the container, such as “ls” to list the directory contents.

To close the console, type the 'exit' command:

• Check what is stopping the containers from running

  If you find that there are no containers after a reboot (i.e. docker ps and/or docker-compose ps show no running containers), you can troubleshoot this by checking the output of the hubble- docker-compose.service. You can do this with the following command:

  Check Hubble service status

  systemctl status hubble-docker-compose.service

  This last few lines of output from this command should give some information about the error. An example output can be seen below (although it is from a successful run of the service):