We sometimes have to take the cluster down for scheduled maintenance and upgrades. We usually do this at 11:05pm or 12:05am on the Saturday/Sunday changeover. We did this this weekend.
We did a number of upgrades and improvements, including:
Replaced two older SAN units with new ones. This will increase disk I/O performance for a number of customers.
Updated firmware in all SAN units.
Re-sync'd a database that had gotten out of sync with its replica on Friday evening.
Increased temp and heap table space, which should reduce disk I/O, and increased innodb_buffer_cache as well, potentially easing disk I/O. The combination should result in better database performance.
Upgraded early cluster servers to same kernel and slightly newer Xen to be the same as the newest servers in the cluster.
Getting ready at the data center 2 hours before the maintenance window.
Working on one of the servers.
Swapping SAN units.
Upgrading software and firmware.
Ready to restart the cluster.