The Quilt NOC Common Practice Panel April 4, 2007

The Quilt NOC Common Practice Panel April 4, 2007 Common NOC Practices 4/05/2007 NOC Practices CENIC Staffing and Trouble Ticketing • Staffing: 24x7 • Process to Report Trouble – Listed POC calls (preferred) or emails NOC – Ticket logged – Updates provided via email Trouble Ticket Tools and Management – Tool: Request Tracker (RT) – Ticket Management: • Classified by severity • Urgent and high tickets communicated to CENIC staff within 30 minutes • ~5% Escalated to core eng team – Ticket Status Notification • Urgent or High Status – 3 phases of notification Trouble Ticket Closure and Follow-Up – Follow-up Email • • • Trouble Ticket Review – All high and urgent tickets reviewed by Director of Network Operations and NOC manager Common NOC Practices 4/05/2007 NOC Practices CENIC Communications • Communication with Members – Regular stats available online for open/closed tickets • NOC to NOC Communication • Planned Outage Notification – Posted to website and emailed Network Reporting – Regular stats available online for open/closed tickets • • NOC Information Webpage: http://noc.cenic.org/ Common NOC Practices 4/05/2007 NOC Practices LEARN Staffing and Trouble Ticketing • Outsourced: Texas A&M Computing and Information Services (CIS) • Staffing: 24x7 (15 full-time and 30 part-time staff) • Process to Report Trouble – Customers call or send email – Ticket is logged and acknowledge – Updates • Major outage via conf bridge followed by summary email to the community • Otherwise, updates sent to affected customer via email Common NOC Practices 4/05/2007 NOC Practices LEARN Staffing and Trouble Ticketing • Trouble Ticket Tools and Management – Tool: Home grown (Keystone) – Ticket Management: • Log troubles and queries – Process to handle non-member customers • Classify (H,M,L) • Escalation process – Ticket Status Notification • Telephone and Email • Trouble Ticket Closure and Follow-Up – Telephone or Email • Trouble Ticket Review – Frequent review of tickets with NOC service provider Common NOC Practices 4/05/2007 NOC Practices LEARN Communications • Communication with Members – Mostly live communication – Reports furnished to members upon request • NOC to NOC Communication – Ad Hoc – NOC maintains a list of partners • Planned Outage Notification – Announced during the weekly status call – Email outage notification to the member community • Network Reporting – Today… Network stats behind closed network – Project in progress to make it available to members (InterMapper maps and Switch/Rtr proxy) Common NOC Practices 4/05/2007 NOC Practices OSHEAN Staffing and Trouble Ticketing • Staffing: 24x7 Outsourced to Atrion Networking Corp. (Local RI Systems Integrator) $50k/year • Process to Report Trouble – Listed POC calls (preferred) or emails NOC – Ticket logged – Updates provided via email – call to OSHEAN Director of Technical Services during business hours Trouble Ticket Tools and Management – Tool: What’s UP Gold – Ticket Management: • Classified by severity • Urgent and high tickets communicated to OSHEAN staff within 30 minutes • Escalated to core eng team (Atrion and OSHEAN members make up core eng team) – Ticket Status Notification • Urgent or High Status – 3 phases of notification • In cases of an outage, ticket updated every 30 minutes Trouble Ticket Closure and Follow-Up – Follow-up Email – Ticket closed on weekly Operations call with Core Engineering Team Trouble Ticket Review – All high and urgent tickets reviewed by Director of Technical Services and NOC manager • • • Common NOC Practices 4/05/2007 NOC Practices OSHEAN Communications • Communication with Members – Regular stats available online for open/closed tickets – Members are contacted by phone during trouble and upon closure • NOC to NOC Communication – Atrion NOC responsible for communication with other NOCS • Level3, Cox, Verizon, Boston University, NoX, National Grid, Fibertech, Cisco • • Planned Outage Notification – Posted to website and emailed Network Reporting – Regular stats available online for open/closed tickets • NOC Information Webpage: noc.oshean.org (public mrtg page) – Also use Solar Winds with password protection for member access Common NOC Practices 4/05/2007 NOC Practices 3ROX Staffing and Trouble Ticketing • Staffing: 24x7 on-call • Process to Report Trouble – Engineers or NOC from direct customer contacts us – Ticket logged – Updates provided via email Trouble Ticket Tools and Management – Tool: RT, www.bestpractical.com, O’Reilly book – Ticket Management: • Classified by severity • Typically person who opens ticked works ticket Volume – < 30 / qtr Trouble Ticket Closure and Follow-Up – Customer notified of completion Trouble Ticket Review – Rare but easily done • • • • Common NOC Practices 4/05/2007 NOC Practices 3ROX Communications • Communication with Members – Individual updates on each ticket • Planned Outage Notification – Emailed – 2 week lead time desired – Pro-active notice to customers of noticed customer outage • 24x7 automatic paging/monitoring – Mon • Network Reporting – Reviewed at regular Member meetings Common NOC Practices 4/05/2007 NOC Practices OSCnet (Formally OARnet) • Staffing: 24x7 (Third Shift via pager) • Process to Report Trouble – Phone via 800 number or email ( Currently updating to allow web access for self-serve) – Ticket logged – Updates provided via email or call back on request Trouble Ticket Tools and Management – Tool: Remedy (Currently Version 6; migrating to Version 7 over summer; ITIL) – Compute Platform: Apple Mac Book - allows flexibility; can plug in at any POP on reserved port; distributed monitoring systems – Ticket Management: • NOC has 3 Levels – Level 1 - Support Desk & Triage – Level 2 First level Engineering - Non-critical request; Basic issues – Level 3 Second level Engineering - Urgent & Critical issues; Within second level Engineering are two departments- Optical & IP/Switching services • • Triaged by first level NOC and assigned to proper department - all communications result in ticket Urgent and high tickets escalated to core eng team within 15 minutes • Common NOC Practices 4/05/2007 NOC Practices OSCnet (Formally OARnet) • Ticket Closure – – – – Client must acknowledge resolution to their satisfaction Client can use either phone or email to authorize closing ticket All High Priority are Tickets are reviewed by senior staff Post mortem for all High Priority Tickets is automatic; are offered on request for lower level tickets Common NOC Practices 4/05/2007 NOC Practices OSCnet (Formally OARnet) • Trouble Ticket Closure and Follow-Up – Client must acknowledge resolution to their satisfaction – Client can use either phone or email to authorize closing ticket – Post mortem for all High Priority Tickets is automatic; are offered on request for lower level tickets • Trouble Ticket Review – All High Priority are Tickets are reviewed by senior staff; NOC Manager, Operations/Optical Manger, IP Services Manager, Director of Operations and ocassionally the Director. Common NOC Practices 4/05/2007 NOC Practices OSCnet (Formally OARnet) • Communication with Members – Email & Phone – Will be adding web interface for customer enabled ticket review • NOC to NOC Communication – Email & Phone • • • Planned Outage Notification – Posted to website, emailed and called Network Reporting – Regular stats available online via Statseeker NOC Information Webpage: http://www.osc.edu/oarnet/support/index.shtml Common NOC Practices 4/05/2007