BK Brant Kuykendall
DN Don Nash
GM Glenn Martin
JA James Alexander
JB Jim Bulluck
JR James Russell
JW Jason Wang
KW Kimberly Wilkins
MD Michael Dougherty
RG Robert George
SE Shaun Evans
TN Tom Nail
Time Who Status What
7:30 Operator on Duty, JB Complete Drain Mainframe Initiators
7:30 MD Did not get turned off Stop SCOM
7:50 ALL Complete Dial into conference call.
7:50 RG Complete Take Blackboard out of rotat
7:55 ALL Complete Conference call roll call.
8:00 Operator on Duty, JB depends on initiators being drained Stop Mainframe printing
8:00 TN Complete Turn on maintenance window
8:00 SE Complete Shutdown Sharepoint
8:00 CW Complete Shutdown TIM
8:05 KW excluding webspace DBs, wait for TIM Shutdown Oracle servers in U
8:05 GM Complete Shutdown AEMS
8:05 BK did not happen automatically; manual Shutdown Virtual Center for
8:05 MG Complete Turn off mainframe email (ja
8:05 VS Complete Unmounted /backup NFS mo
8:30:00 AM (8:37) NET Complete Shutdown and reload UDCA*
9:00 DN Complete Shutdown webspace
9:05 KW depends on webspace down Shutdown webspace Oracle D
9:15 KW depends on UDCA up Startup Oracle (TIM, OVPD) d
9:15:00 AM (9:25) GM depends on UDCA up Restart AEMS
FYI UMBS, ironports, webmail su
9:15:00 AM (9:30) MG depends on UDCA up Activate Mainframe email, et
9:15 JA Complete Shutdown SQL02 cluster
9:15 JR Complete Shutdown Remedy
9:30 Operator on Duty, JB Complete Restart Mainframe Initiators
9:30 MK, UT NIC Complete Switch to Web Central altern
9:40:00 AM (10:10) KW Complete Confirm Oracle DBs in UDCB
9:45:00 AM (10:10) NET Complete Shutdown and reload UDCB*
10:15:00 AM (10:56) JA Complete Restart SQL02 cluster
10:15:00 AM (12:XX) KW Complete (problems with scripted startup) Restart Oracle
10:15:00 AM (10:58) MK, UT NIC Complete Switch Web Central to prima
10:25:00 AM (10:56) BK Restart
already setup (automatic if SQL02 is up) not automaticVirtual Center for ESX
10:25:00 AM (11:15) SM depends on SQL02, Vcenter Bounce Adobe Connect Serve
10:25:00 AM (11:02) JR depends on SQL02, Vcenter Restart Remedy servers
10:25:00 AM (11:10) SE depends on SQL02, Vcenter Restart Sharepoint
10:35:00 AM (11:03) SM depends on Virtual Center Bounce Blackberry (BES) serv
10:45:00 AM (11:23) DN depends on Oracle Restart Webspace
10:45:00 AM (11:33) RG depends on Oracle restart Blackboard
11:00:00 AM (11:26) Operator on Duty, JB depends on UDCA, webspace up Restart Mainframe printing
11:00:00 AM (11:15) MG depends on Oracle Restart Mainframe ID card, e
11:00 TN depends on UDCB MySQL restart blogs and wikis
Drain Mainframe Initiators
Stop SCOM
Dial into conference call.
Take Blackboard out of rotation
Conference call roll call.
Stop Mainframe printing
Turn on maintenance window in Zenoss
Shutdown Sharepoint
Shutdown TIM
Shutdown Oracle servers in UDCA (priority) and UDCB (afterwards)
Shutdown AEMS
Shutdown Virtual Center for ESX and vDesk
Turn off mainframe email (javamail)
Unmounted /backup NFS mount on iqdware1/2/5/6 which comes from verges (COM)
Shutdown and reload UDCA* routers (65xx series)
Shutdown webspace
Shutdown webspace Oracle DBs
Startup Oracle (TIM, OVPD) databases
Restart AEMS
UMBS, ironports, webmail survived, no problems
Activate Mainframe email, etc.
Shutdown SQL02 cluster
Shutdown Remedy
Restart Mainframe Initiators
Switch to Web Central alternate site (mcfeely)
Confirm Oracle DBs in UDCB are down
Shutdown and reload UDCB* routers/switches (65xx series)
Restart SQL02 cluster
Restart Oracle
Switch Web Central to primary site
Restart Virtual Center for ESX and vDesk
Bounce Adobe Connect Servers
Restart Remedy servers
Restart Sharepoint
Bounce Blackberry (BES) servers
Restart Webspace
restart Blackboard
Restart Mainframe printing
Restart Mainframe ID card, etc.
restart blogs and wikis
did not hear back from Rich Kreig about shutting down blogs/wikis
Command decision (GB, TN, VS) to shutdown gracefully so any active sessions do not lose/corrupt backend DB
Oracle shutdown (UDCA/B) took slightly longer than planned
have cut-n-paste available?
have shells open on all DBs prior to work?
tasks occurred that were not on planning spreadsheet
force physical sit-down to review tasks?
Mainframe printing depends on Webspace
well, most mainframe printing goes to webspace, no sense in bringing up earlier
Need to know what router devices are on
in case shutdown process does not go as planned
Take a snapshot of Zenoss Event console prior to maintenance (not just production event console)
good idea
What Oracle DB is started in what priority/order would be useful
problems with IQ (iqtfs 7/33, iq-tweb2 8/11) came back as 100/half instead of 100/full
nobody from project IQ on call (not expected to be necessary, VS left note with RH and MM)
MM restarted servers; negotiated to 100/full
problems with pts (8/1 came back as 100/half instead of 100/full)
great quality control JW, checked all link status prior/post on all 5 routers (good info to have)
shibboleth2 went down (unplanned) depends on AD (which remained up). Shibboleth production is OK.
EA to look into this
why did my workstation lose connectivity when udcb06 went away?
question on total amount of time w/ udc routers down
uprint01 is the device that front-ends print05,06 that needed to be restarted
LL fixed this
problems with MySoft DB
UMBS did not come back up
wb10 uses alias for ruger (umbsdb) to connect to its DB.
DN changed config to point to ruger directly. Will need follow up with JA
not lose/corrupt backend DB
ion event console)
H and MM)
info to have)
boleth production is OK.