MtgSumm20091203 by gegeshandong


									                                                           CMS Technical Users Group
                                                                   Meeting Summary
                                                                  California State University

 Meeting Date             December 3, 2009   Time         9:00am – 11:00am
 Meeting Location         Teleconference     Dial-In #    866-213-2185       Participant Code   5600403
 Chairperson              Charles Holmberg
 Attachments -
 print and review
 Topic / Discussion                                                               Led by
1. Welcome, Roll Call, Review Agenda                                              Charles Holmberg

2. WAN Service Updates                                                            Ed Smith
   CENIC Service update, TIS update, other:

WAN Update:

Update on campus connectivity issues:

We are working with Sonoma and AT&T to identify and agree on a diverse on-campus fiber path for their
second gigabit connection.

Summary of recent NOC tickets:

Nov 3, regular maintenance to upgrade IOS in backbone and ISP routers out of Sunnyvale which briefly
impacted most connections to our Sunnyvale pop including East Bay, San Jose, Humboldt, Sonoma and
the CMS circuit to SLC. Most were not service-affecting due to built in redundancy at the campuses and

RT 186811: Ticket opened at Unisys' request for DR test in December. Date pending.

RT 186060: Ticket for maintenance window so AT&T could test CSU Humboldt's gig circuit due to errors.
Errors were resolved by switching to a different fiber pair.

RT 186949: Ticket for brief bounce on CSU Fresno's gig circuit which was due to brief loss of signal
within Telco infrastructure for undetermined reasons.

RT 167226: Old ticket to replace a fan in MLML's gatekeeper which was completed in early November.

RT 188676: Ticket for NOC alarm at CSUMB. Was due to campus having to reboot one of their routers.

RT 189082: Maintenance ticket for campus power upgrades at San Jose State on Nov 13.

RT 189107: Ticket to work with Sac State on some IPv6 issues.

RT 189179: Ticket for brief bounce on San Marcos' Gigaman circuit on Nov 13. LAN interface never
bounced. No RFO was found.

RT 188362: Abuse ticket for a complaint someone at Channel Islands downloaded a copyrighted TV

RT 188980: Another abuse ticket for suspicious activity coming from CMA.

RT 189142: Abuse ticket for suspicious activity coming from San Jose State.

RT 189082: Ticket for maintenance window for San Jose State to work on their terminal server and
Last Revised: 7/29/2012                                                                          Page 1 of 8
CMS Technical Users Group

RT 189542: Ticket for an alarm on 11/15 on MLML's connection. Brief bounce on the circuit. No RFO.

RT 190172: Ticket for contractors at Pomona to move some fiber connections to another conduit on

RT 190569: Ticket for alarm on CMA's connection which turned out to have been caused by an on-
campus power issue.

RT 192072: Ticket for a brief loss of connectivity at East Bay on 11/28 which was due to an internal DOS
attack and needing to reboot their firewall.

Per our discussion and per your request, here are the primary site contacts our NOC currently uses. Keep
in mind that these are technical WAN contacts. We maintain alternate and tertiary contacts for most sites.
If you know of any recent changes, please send them to Ed Smith,

CSUB Lancaster Center
Ron Watts (Onsite tech)

CSU Bakersfield
Mike Fleming

CSU Chancellor's Office
Jason Solis

CSU Channel Islands
Herb Aquino (Manager)

CSU Chico
Dennis Partington

CSU Dominguez Hills
Mahabub Alam

CSU East Bay
Chris Da Silva

CSU Fresno
Gabriel Lewis

CSU Fullerton
Saeid Mirdamadi

CSU Fullerton Irvine Campus
DeLana? Bush-Hamblin

CSU Humboldt
Rick Garcia
Last Revised: 7/29/2012                                                                         Page 2 of 8
CMS Technical Users Group

CSU Long Beach
Craig Kleen

CSU Los Angeles
Bob Hoffmann

CSU Monterey Bay
Rick Skibinski

CSU Moss Landing Marine Labs
Jeff Arlt

CSU Northridge
Will Moran

CSU Office Govt Affairs
Dave Avery (Only works M/W/F)

CSU Sacramento
Carl Oakes

CSU San Bernardino
David Hatch

CSU San Bernardino Palm Desert
Earl Wilson

CSU San Marcos
John Humes

CSU Sonoma
Jean Sugiyama

CSU Stanislaus
Noel Whitehead

CSU Stanislaus Stockton Center
Michael Tablett

CSU WestEd
Jack Hunker

Cal Poly Pomona
Gabe Kuri
Last Revised: 7/29/2012          Page 3 of 8
CMS Technical Users Group

Cal Poly SLO
Rick Uc

California Maritime Academy
Walter Abarca

Calpoly Netlab
John Bellardo

Gary Simpson

Pete Braccio

San Diego State Univ
Kent McKelvey

San Francisco State
Jack Tse

San Jose State
Dennis Fox

For CMS related circuit issues, here is who we have listed as CMS contacts:

CMS Support Center – available 7x24x365: (562) 346-2456
CMS Support Center email:

Secondary contact/escalation point if Support Center doesn’t respond: Sterling Wong
Secondary phone: 562-951-4322, after hours phone: 562-544-3346
Secondary email:

3. Technical Sub-Committees and Forums
   .1 Web Cast/Presentation Status                                                 George Mansoor
        There was a request for App Designer training. This will be developed with a web cast to be
   .2 Enhancements to the Technical Process                                        George Mansoor
        No change in status due to resource constraints caused by furlough days.
   .3 Application Performance Management Project                                   Yves Hepperle
        Pre-load caching: working well, may need to put in more than 10 components but limit to 20
           minutes; improves HR more than SA. San Luis Obispo, Stanislaus and Sonoma have moved
           into production; email Charles if interested in a list of the components they use.

4. Security Migration of VPN – Status                                          Alex Harwood
    No update.

Last Revised: 7/29/2012                                                                    Page 4 of 8
CMS Technical Users Group

5. Data Center SLA – Status                                                  George
    September and October SLAs will be sent to the TUG.
    All SLAs were met in September. In October, all met except for the SEV 3 responses.

6. Campus Firewall Review                                                    George
    The plan to do a complete firewall access review with all campuses has been postponed due to
      furlough caused resource contention. However, campuses can request a review if there is a
    This can be removed as an agenda item.

7. Segregation of Duties – KPMG Audit                                             Ron Heeb
    No update.

8. Human Resources Update                                                 George
    This agenda item will be changed to an item for updates on CMS and CFS activity.

9. Common Financial System - Status                                         George
    There are three (3) First Wave campuses: Channel Islands, Fullerton and Humboldt. They will
      receive code in March 2010 and will implement in July 2010.
    Remaining campuses will install CFS common code by July 2010 and will all be in production by
      July 2011.
    A presentation on security strategies is scheduled for January 2010.
    Working with the First Wave campuses in Integration Broker issues.
    MP 2 will contain the release of the common code.

10. HCM 10G Upgrade – Status                                                    George
     Performance issues have been identified and fixes will be posted for those campuses which have
      already migrated. Fixes will be included in an MP for those campuses who have yet to migrate.
     Mentor application upload: need to change the stats method; it will be a standard part of 10G.

11. Meeting Calendar – 2010                                                       Charles
     Proposed calendar was discussed and approved.

12. Environment Updates
    .1 Data Center Update                                                         Michael Kelley
        Unisys Facility update since last mtg & status
        Outages, projects, issues, concerns

Major Successes:
    November Maintenance Window completed successfully without any incidents or impact to the
          o Successfully replaced SAN Storage Processor restoring redundancy to the shared
             storage device DC-408.
          o Successfully replaced a blade in a network switch, preventing potential down time due to
             blade failure.
    Unisys and CMS successfully completed all Oracle 10G upgrades for the month of November.

Major Issues:
Unisys Incident #     Client Incident #(s)       Incident Start Date        Incident End Date            Duration (hrs)
EA-002790113          90194                      11/18/2009 09:34 CT        11/18/2009 17:44 CT          2.18
EA-002789999          90193                      11/18/2009 08:02 CT        11/19/2009 14:38 CT
EA-002790966          90194                      11/19/2009 06:55 CT        11/19/2009 12:24 CT          7.77
Problem Description:          We had a Storage Processor panic on DC-408. This caused multiple file systems to go
                              offline. Due to VCS errors on CSSLU363, the file system could not be remounted without
                              a reboot of the entire cluster. A workaround was put in place to hold tape backup media
                              onsite to service Cloning. This issue ultimately led to the emergency reboot of the
Last Revised: 7/29/2012                                                                           Page 5 of 8
CMS Technical Users Group

                            CSSLU362/363/364 cluster.
Business Impact:            Connection to cluster storage was lost. /bktodisk file system offline. Production
                            databases on CSSLU362/363/364 offline.
Root Cause:                 Hardware / Other
Root Cause Desc:            On Wednesday, November 18 a storage processor on an EMC Clarion storage device
                            panicked and failed. This caused a loss of connectivity to two file systems resulting in an
                            outage to development databases on cluster 382/383. In addition we lost our
                            connectivity to the disk backup file system on production cluster 362/363/364. Unisys
                            immediately implemented the manual cloning process in production as a workaround to
                            preserve cloning service. In order to restore service in development, the database
                            cluster 382/383 was rebooted. The service outage lasted for 2 hours and 11 minutes
                            and impacted 4 campuses.
                            On Thursday, November 19 Unisys experienced an outage to production finance
                            databases on database cluster 362/363/364 initially impacting 7 campuses. Per
                            Symantec, the root cause of this issue was the panic of the storage processor causing
                            cascading VCS clustered file system errors. There were escalation failures as the
                            automation that creates alerts did not function properly and no individual from our
                            organizations recognized the true impact of the errors encountered. To resolve the
                            technical problem Unisys rebooted the entire production database cluster causing an
                            outage to both Financial and Human Resource databases impacting 16 campuses. The
                            outage duration was 7 hours 46 minutes.

                            The storage processor was successfully replaced during the November maintenance
                            window, 11/29/09.

    .2 CMS Update                                                                George
         Changes to architecture, environment, processes, other
         Performance (overall, search tuning, other)
          Campuses should periodically run @messages purge (APPMSGARCH). This has been a
             PeopleTools issue since release 8.48.
          There was confusion on the recent email regarding submitting dates for the application of MP
             7. This only applies to campuses that need to apply it in December. Other campuses can
             submit their dates in January.
          The 1099 fix for Finance will be posted on 12/17/2009.
    .3 Security Update                                                           Alex
          Reviewing FTP access accounts with campuses.
    .4 ISO Update                                                                Charles
          No update.
    .5 IAM Update                                                                Charles
          The group is implementing Federation and Shibboleth for CFS and the data warehouse.

13. Campus Updates / Concerns / Future Meeting Topics / Other                     Member initiated
        Next meeting: January 7, 2010, teleconference.

14. Agenda Items for Future Meetings                                              Member initiated

       Others?

Last Revised: 7/29/2012                                                                           Page 6 of 8
CMS Technical Users Group

Technical Users Group Meetings for 2009

     Month                         Day           Time           Location        Dial in: 866-213-2185

     January                            1   9:00 – 11:00    HAPPY NEW YEAR           Cancelled

     February                           5     9:00 – 3:00    Embassy Suites,

     March                              5   9:00 – 11:00      Teleconference          5600403
               HEUG                             3/22 – 25, Anaheim

     April                              2   9:00 – 11:00     Teleconference           5600403

     May                                7     9:00 – 3:00   Crowne Plaza, LAX

     June                               4   9:00 – 11:00     Teleconference           5600403

     July                               2   9:00 – 11:00     Teleconference           5600403

     August                             6   9:00 – 11:00     Teleconference          Cancelled

     September                          3    9:00 – 11:00    Teleconference           5600403
     Oracle OpenWorld                       9/20 -24

     October                            1     9:00 – 3:00    Teleconference           5600403

     November                           5   9:00 – 11:00     Teleconference          Cancelled

     December                           3   9:00 – 11:00     Teleconference           5600403

CMS Technical User Group Participants:
   Campus / CMS            TUG Rep and Alternate              Present
  Implementation                  Attendee
     Bakersfield          Vic Arcabos                   X

 Chancellor’s Office      Lillian Guo
                          Rick Wong
  Channel Islands         Angela Stockmon               X
                          Lacey Lovejoy
       Chico              Dan Burns                     X
                          Aron Graham
  Dominguez Hills         Michael Kelley

       Fresno             Brad Barker                   X
                          John Wagenleitner
      Fullerton           Joseph Luzzi                  X
                          Susan Kachner
      East Bay            Matt Collins                  X
                          Roger Chen                    X
     Humboldt             Mike Bradley

Last Revised: 7/29/2012                                                                           Page 7 of 8
CMS Technical Users Group

   Campus / CMS            TUG Rep and Alternate       Present
  Implementation                   Attendee
    Long Beach            Michael Markoski         X
                          Cathy Gottlieb
    Los Angeles           Mike Lee                 X
                          Eric Her
 Maritime Academy         Jannette Corpus
                          Raikumar Duraisamy       X
   Monterey Bay           Eric Simoni
                          Sunil Pasala             X
     Northridge           Ron Heeb                 X
                          Colin Chin
      Pomona              Bev Geuting
                          Mauricio Calderon
    Sacramento            Marc Fox                 X
                          Clint Lee
  San Bernardino          Lenora Venturina
                          Mojdeh Vaziri            X
     San Diego            Cyndie Winrow
                          Ralph Suelzle            X
   San Francisco          Andy Cho                 X
                          Reji Titus
     San Jose             Mike Dunefsky            X
                          Jerry Crawford
  San Luis Obispo         George Westlund          X
                          Sharon Anderson
    San Marcos            Diane Petersen           X
                          Margo Lopez
      Sonoma              Lou Ann Seaman           X
                          Geoff Cirullo            X
     Stanislaus           Charles Holmberg         X
                          Carrie Retemeyer

    CMS Central           George Mansoor           X
                          Jeff Whitney
                          Gary Geidel              X
                          Yves Hepperle            X
                          Kai Markowitz
                          Chad Woods               X
                          Andy Sydnor

       CENIC              Ed Smith                 X

       Unisys             Jeff Simper              X
                          Connie Vaughan           X
                          Michael Kelley           X

Last Revised: 7/29/2012                                          Page 8 of 8

To top