Anatomy of an Implementation
Guerin C. Goldsmith and Hsiaosu Hsiung
s VoIP standards mature and ven- with the transition’s complexities.
A dors promise more stable tech-
nologies and products, organiza-
tions have rushed to overhaul their
telecommunications service. Many have
found a disappointing gap, however. The
The project was enterprise-wide, and
operational standards were high: No ser-
vice interruption would be tolerated. Oth-
er than presenting each user with a new
phone, the change to VoIP service was
What does it take to new technology is far from plug-and-play, to be completely transparent: Employees
and its implementation can be fraught with would not have to alter their dialing pat-
successfully execute problems if you make that assumption. terns or telephone use. Following these
a VoIP project? Noblis recently discovered this when it mandates, Noblis successfully imple-
decided to upgrade its private branch ex- mented a VoIP system in fall 2005 with no
Meticulous planning change (PBX) equipment to VoIP technol- disruption to normal business operations.
and consideration of ogy. The project provided a realistic envi-
ronment in which to identify and address
the user perspective VoIP’s technical challenges and gain both System planning
can overcome many hands-on experience with an emerging Figure 1 shows the resulting system,
technology and application insights from which is layered over the data network.
problems, as this a client perspective. Among the many les- SBC is the local exchange carrier, and
case study illustrates. sons learned was that judiciously apply- AboveNet in Reston, Virginia, provides
ing tested project management methods— the alternative facility for the call man-
chieﬂy, careful planning and risk assess- ager cluster. VoIP calls between station
ment—was the best antidote to dealing sets at Noblis’ Fairview Park (FVP) facil-
• VoIP implementation planners must deeply understand implementation details and installation
nuances. Companies without that knowledge should hire a system integration contractor.
• A table of failure scenarios can help the project team speculate about risks and instill conﬁ-
dence in management that mitigation approaches are in place.
• A system design that minimizes transition headaches and meshes well with business op-
erations must consider call plans, interconnection services, security, and support for analog
• A phased approach to VoIP transition allows for a period in which users are on both the PBX
and VoIP systems. Noblis found that two weeks of dual use was enough time to identify and ﬁx
Vo i c e o v e r I n t e r n e t P r o t o c o l 9
Cisco 4507R Cisco 4507R Site Access Switches/Routers
Noblis User Cisco 3750 or Cisco 2650XM, Cisco 2821
Closet Switch Closet Switch
Ethernet Analog Line Local and
Cisco 6500 Cisco 6500
Core Switch Core Switch
Cisco 3845 Ethernet Cisco 3845
SBC ATM Fractional
Local Router Software
Gateway MCS 7835
Figure 1. VoIP network architecture in the VoIP project. The Noblis VoIP network consists of two layers implemented by closet switches and core switches,
all with redundancy. Redundant closet switches are on each ﬂoor at the Falls Church, Virginia, (FVP) facility; a second call manager server is in Reston,
Virginia. The VoIP system interconnects to the local exchange carrier (SBC) and to AT&T through VoIP media gateways. At FVP, VoIP calls made to locations
not on the VoIP network are completed through SBC for local calls and through AT&T for long distance calls.
ity in Falls Church, Virginia, and between data network also made it easier to con- In house or contract?
FVP and remote sites are completed over verge voice and data features, such as Planning for a VoIP implementation
SBC’s Multi-Protocol Label Switching voicemail. Noblis then baselined the ex- demands careful coordination with all
(MPLS) network. The VoIP network at isting data network to determine what ad- parties involved in the VoIP system in-
each remote site is composed of a site ac- ditional upgrades or modiﬁcations would stallation and deployment, which in turn
cess router, which provides connectivity be needed to support VoIP. requires a deep understanding of imple-
to the public switched telephone network As part of network baselining, Noblis mentation details and installation nu-
(PSTN) and MPLS network through me- reviewed its business operations and list- ances. Not having internal resources with
dia gateways. A site switch provides local ed features and functions that the PBX that level of understanding, Noblis de-
area network (LAN) interconnectivity to provided, as well as desired new features cided to hire a systems integration con-
all user VoIP station sets and other VoIP and functions that only the new VoIP sys- tractor, SBC, which would be responsible
equipment at the site. tem would offer. The project team looked for installing and testing the VoIP equip-
As Figure 2 shows, the project team de- at the existing infrastructure’s ability to ment and for transitioning voice service
voted the ﬁrst three months to planning, meet these new functions, including any from the PBX. SBC essentially became
during which they made several key de- impact on wiring and cabling, heating; the prime contractor, subcontracting work
cisions about desired features, equipment ventilation, and air conditioning. The to the VoIP equipment vendor. SBC also
vendors, migration strategy, and risk miti- team also examined the power availabili- helped identify features and functions that
gation. ty and power supplies for VoIP equipment the initial inventory overlooked and used
and station sets. the ﬁnal inventory in its planning.
Functions and features Another task was to take inventory of Some organizations might have enough
As one of the ﬁrst steps in planning, the infrastructure, which involved identi- internal expertise to test, install, and acti-
an enterprise must decide if the VoIP net- fying all the planned locations of station vate the VoIP equipment themselves and
work should support both data and voice sets, such as desktops, hallways, lobbies, will not have to hire an integration con-
or voice alone. Noblis decided to imple- cafeterias, phone banks, laboratories, tractor. However, all organizations will
ment its VoIP system using the company’s waiting areas, common areas, restrooms, need a VoIP equipment vendor and will
data network—in part, because it had conference rooms, equipment rooms, and generally have to investigate vendors,
made earlier upgrades to the data network elevators. It was also important to de- create a candidate list of four or ﬁve, and
to ensure that network equipment would termine the locations of analog lines for conduct a cost-beneﬁt analysis to select
support network quality of service and devices, such as modems and facsimile a winner. Any such vendor investigation
different data trafﬁc classes. Using the machines. should include
10 Vo i c e o v e r I n t e r n e t P r o t o c o l
Jan 2005 Feb 2005 Mar 2005 Apr 2005 May 2005 Jun 2005 Jul 2005 Aug 2005
Figure 2. VoIP project timeline. Planning is key to a successful effort, including selecting equipment vendors and setting up communication channels so
that everyone understands the new services.
• a survey of each vendor’s product features vice impairments, the project team created call plans, interconnection services, secu-
and comparison against the organization’s a table of possible failure scenarios. For rity, and support for analog devices.
features list; each scenario, the team listed the failure’s
impact and the means of mitigating its ad- Call plans
• enough technical discussion with vendors to verse consequences. The team used this ta- Because number, dialing, and routing
ensure that engineering and support staff un- ble both to speculate about failure scenari- plans can signiﬁcantly affect both the VoIP
derstand how the products would operate in os and to instill conﬁdence in management system implementation and the organiza-
their organization; and that mitigation approaches were in place. tion’s business operations, the path of least
resistance is to maintain as much similarity
• an assessment of the vendor’s past installa- between the old plans and the VoIP plans
tions that might be similar to the one planned, System planning action items: as is feasible. Users can then retain the
including a list of references. extension numbers and dial calls as they
• Decide if the VoIP system will support did with the PBX. This arrangement will
data as well as voice. make the transition to VoIP service much
For its VoIP equipment vendor, Noblis smoother. The dialing plan must also ac-
chose Cisco, which had supplied the com- • Choose an integration contractor or
commodate 911 calls, which need enough
pany’s data network equipment. Choosing decide to test, install, and deploy the
location information for emergency re-
the same vendor eliminated cross-vendor VoIP system yourself.
sponders to locate the caller.
interoperability problems between data • After determining the desired functions Noblis was able to preserve all existing
and voice network equipment and gave the and features, choose a VoIP equipment plans except that it had to change the preﬁx
company a contractor it knew had exten- vendor. of the 10-digit number used to place calls
sive experience with VoIP systems. Cisco to remote sites. The VoIP system also pro-
also provided input into the features and • Create a table of failure scenarios and vides enough information for 911 respond-
functions list and used the ﬁnal inventory use them to identify risk mitigation ap- ers to ﬁnd the exact room in which the call
to plan equipment deployment and create a proaches. was made.
bill of materials. Sometimes it is not possible or efﬁcient
to retain the existing call plans. If an orga-
Risk mitigation nization must revise the plan, it should do
Typically, VoIP systems have at least one
Design and engineering
so as early as possible because any changes
or two nonnegotiable requirements. For A system design that minimizes transi- will affect the network equipment conﬁgu-
Noblis, these were voice quality and sys- tion headaches and meshes well with busi- ration and business operations. Revisions
tem reliability and availability. To prepare ness operations is fundamental to a VoIP also add a component to user training: fa-
for VoIP equipment-related failures or ser- project’s success. Considerations include miliarizing users with the new plan.
Vo i c e o v e r I n t e r n e t P r o t o c o l 11
Directories available for the desired speed of connec- Table 1. Support methods for analog devices.
Any VoIP implementation will involve tion between the off-site user and Noblis’
the use of directories. The decision is LAN. For these reasons, Noblis decided
whether to maintain an enterprise-wide not to implement the softphone until it Method
directory or to implement multiple stand- could resolve security and performance is- Media gateway Terminal adaptor
alone directories. If the choice is multiple sues, and to date the softphone has yet to be and RJ-11 wiring and RJ-45 wiring
directories, the enterprise must ﬁnd a means implemented.
to synchronize their contents. Lower initial cost Must maintain only
Noblis elected to operate with separate Analog devices RJ-45 wiring
directories. One reason was the poor per- In most organizations, the VoIP system
Easy to move or add
formance of the Lightweight Directory must also support existing analog devices, analog devices
Access Protocol interface between the such as facsimile machines and modems.
VoIP system’s directory and the multiple Table 1 shows two support methods. One
enterprise versions of MS Active Directo- method is to connect the analog device to Must maintain both Higher initial cost
ry, which caused users to experience long the VoIP system’s media gateway using the RJ-11 and RJ-45
delays while browsing the VoIP directory. existing RJ-11 wiring. The other method is
Instead, the VoIP system maintains its own to deploy an IP terminal adaptor adjacent Difﬁcult to relocate the
directory, which is synchronized ofﬂine to the analog device that would convert the device
with the company’s separate version of analog signal to a digital signal. The digital
Active Directory. The resulting directory signal, in turn, would connect to the LAN
system works somewhat differently from through an RJ-45 wall jack, on which the or cutover, will unfold, troubleshooting
Noblis’ original plan to use an enterprise- VoIP system operates. With the second equipment and settings, establishing a
wide directory, but it effectively avoids the method, users can more easily move the backup and recovery mechanism, training
problematic interface. analog device throughout the facility or add users, and instituting proven project man-
analog devices, but the initial cost is higher agement practices.
Interconnection because each device requires a terminal
Telecommunication services for inter- adaptor. Schedule
connecting on- and off-network locations Noblis elected to use the existing RJ-11 Perhaps the best way to develop a
directly affect the VoIP system’s perfor- wiring because it did not foresee the need VoIP implementation schedule is to cre-
mance, cost, and security. As Figure 1 to move existing analog devices or add new ate a work breakdown structure (WBS)
shows, Noblis decided to use the PSTN to ones. that covers all the activities associated
connect to off-network users and SBC’s with implementing the VoIP equipment
MPLS network to connect the FVP and on- and migrating service. A WBS details the
network remote sites. The MPLS network Design and engineering action items: tasks involved, assigns responsibilities
makes it possible to implement quality of to accomplish the tasks, and establishes
service, which means that the network can • Consider keeping the existing number, anticipated timelines to complete them.
support real-time application issues, such dialing, and routing plans to make the SBC used Microsoft Project to automate
as voice trafﬁc. transition to VoIP service easier. the WBS, making it easier to manage. The
team frequently reviewed the WBS to en-
Security • If you can’t keep the existing dialing sure that the project stayed on schedule
The scope of security measures and the and routing plans, make any revisions and used it to plan workarounds if certain
means to implement them depend heavily as early in the project as possible to tasks were delayed.
on the VoIP network infrastructure. If the ensure that they are well designed and SBC also held weekly meetings with all
Internet is the site-to-site transport mecha- engineered. interested parties, and Noblis’ intranet car-
nism, a ﬁrewall to handle network address • Select a plan for handling directories. ried the status of all activities on a shared
translation will be needed, as well as a team site. Any implementation or deploy-
means to handle intrusion detection and • Choose the site interconnection service. ment problems were discussed during
prevention applications. the weekly meetings, and decisions were
• Review security measures and accom-
Although the mobility of computer-based reached on the spot.
modations for 911 calls.
telephones, or softphones, offers great ﬂex-
ibility, it also presents additional security • Decide how to support analog devices. Cutover
and performance concerns. 911 calls are an The cutover can be done all at once or in
issue because the needed number identiﬁ- phases. In the phased approach, some us-
cation would come from where the call is ers are put on the VoIP system, while oth-
handled (the FVP location on the VoIP net-
Implementation issues ers remain on the existing system. The two
work), not where the call is placed. Also, Implementation issues include making user groups must be able to communicate
the required voice bandwidth may not be a schedule, deciding how the transition, during this dual-use period.
12 Vo i c e o v e r I n t e r n e t P r o t o c o l
Noblis felt that the phased approach was echo. Network administrators were able User training
a better choice because it helped gel users’ to trace the cause to the clock references Regardless of the implementation de-
conﬁdence that the VoIP system would for the media gateway and primary rate in- tails, any transition to VoIP services is
operate reliably. Moreover, any problems terface that connects the media gateways likely to require user training. The enter-
would affect only a few users, and network with the PSTN. Because the clocks were prise must consider the type of training and
administrators would have time to ﬁx them not synchronized, the media gateway’s training materials needed, whether training
before rolling the system out completely. echo cancellation logic could not func- should be formal or informal, and whether
Noblis decided to have a dual-use period tion properly. VoIP administrators solved it should be done in-house or by the VoIP
of two weeks, which it felt was ample time the problem by replacing the primary rate contractor.
to identify and ﬁx problems and acquaint interface card with one that could properly Noblis employed a train-the-trainer strat-
users with new features. attain synchronization. egy, in which SBC, in concert with Cisco,
After user training, the company al- The exact troubleshooting context will provided training to select CIM staff, in-
lowed users to retain their old station sets differ across organizations, but regardless cluding those responsible for maintaining
but also provided new ones, which users of the speciﬁc problems, a good practice is the VoIP system. The CIM trainers then
were expected to try out. Inbound calls conducted formal employee training in a
could be made to either system. Users’ pilot demonstration at the FVP site using
extensions remained the same, but the actual station sets, with which they dem-
exchange codes for the PBX and VoIP onstrated common operational features of
systems were different. At the end of the Regardless of the the VoIP system. Individuals then used the
two weeks, the transition to the new VoIP sets at their seats as trainers showed them
system was considered complete, and implementation how to activate features. CIM trainers also
network administrators removed the old details, any traveled to remote sites to provide similar
station sets. By this time Noblis’ Corpo- training and answer questions.
rate Information Management (CIM) staff transition to VoIP In addition to the formal training, train-
had gained experience operating the new services is likely ers handed out wallet-sized user guides to
system, and users had had enough experi- all employees along with a pamphlet ex-
ence in using the new phones to grasp the to require user plaining feature operation in greater detail.
phone’s operation and features. training. Employees could also ﬁnd these training
CIM staff then sent out a notice to PBX materials on the Noblis intranet.
voicemail users that their messages would
be retained only until the company de- Project management
commissioned the PBX voicemail system From a management perspective, a VoIP
in several weeks. At that time, the VoIP to have the vendor meet with the organiza- system project is no different from any
system would be operable, and voicemail tion’s technical staff to review all equip- other project that requires enterprise-wide
would go to the new system only. ment settings and connections. coordination. Noblis followed best practic-
This cutover strategy worked well, and es, such as ensuring that all parties accept
users accepted the change without undue System recovery and backup the implementation plan and will adhere
complaint. However, phased migration For rapid network recovery, a distrib- to it, thoroughly examining activity work-
might not be suitable for all organiza- uted VoIP system architecture makes sense ﬂow, and establishing workaround plans
tions, since dual-use might not be feasible. because all call processing and gateway to accommodate changes in planned tasks.
Noblis had a separate area code that it could equipment then has alternative connec- Of utmost importance is the need to estab-
map to the same numbering plan as the 610 tion paths to other telecommunications lish communication channels at all levels
area code. This situation made it possible networks, such as the PSTN. Noblis’ VoIP so that individuals know from whom they
to operate the old and new systems simul- system architecture uses redundant call should take direction and from whom they
taneously. Not all companies will have the manager clusters and alternate paths to should seek help. Everyone should also un-
luxury of a second area code and will need both the interexchange carrier and SBC. derstand when problems or issues should
to resort to an all-at-once, or ﬂash, cutover It is vital to back up VoIP databases, be escalated, to whom, and the proper pro-
to the new VoIP system. Organizations that such as directories, as well as numbering cedure for doing so.
are unsure about which approach is bet- and dialing plan databases. Noblis opted to Other best project management practices
ter—phased or ﬂash—should ask the VoIP use its existing backup procedures—part address the inevitable planning deviations.
equipment vendor how it would conduct of its Continuity of Operations Plan—for The team must set up a mechanism for re-
the transition from the current voice envi- VoIP system corporate and user ﬁles. The viewing what motivated the deviations and
ronment to the new VoIP system. plan describes when to implement backup their impact on schedule and cost, as well
procedures and when to resume normal as how plans might be revised. Again, con-
Troubleshooting operation. Network administrators make sensus and follow-up will ensure that the
During implementation, some Noblis daily backup ﬁles and take them to an off- implementation proceeds according to the
users experienced a few seconds of voice site location for safekeeping. agreed-on revisions.
Vo i c e o v e r I n t e r n e t P r o t o c o l 13
tion testing duplicates initial testing, but in
a real-world setting. It attempts to ensure
Implementation action items: Testing action items:
that all the equipment interconnections are
sound and that the network infrastructure
• Develop an implementation schedule • Conduct initial testing of VoIP system
using a work breakdown structure. components using all potential calling
Similarly, Noblis’ engineering staff and
scenarios. Be sure that all appropriate
• Decide if cutover will be phased or occur CIM members—those ultimately respon-
parties review and agree on the test pro-
all at once and establish suitable training sible for the VoIP installation and deploy-
to ease the transition on users. ment—prepared and reviewed the veriﬁca-
tion test procedures. CIM and Noblis’ en- • Conduct veriﬁcation testing to baseline
• Have a vendor meet with the project gineering staff conducted the veriﬁcation system performance after installing the
team to review all equipment settings tests at the FVP facility. Testers simulated system but before making it operational.
and connections. the environment using equipment and net-
work connections like those in the opera- • Conduct acceptance testing to identify
• Incorporate network and system redun- tional network. and resolve operational problems. Make
dancy and have a plan for backing up the sure that users have a backup service
ﬁles associated with the VoIP system. Acceptance at this time.
Acceptance testing, which occurs after
• Use project best practices to manage • Document all test plans and results,
the system becomes operational, ferrets out
workﬂow and handle any deviations. particularly for veriﬁcation testing, which
any unanticipated problems. If the existing
voice system is still in operation, it can be establishes the system’s baseline per-
a fallback if the VoIP system experiences a formance.
major failure. Acceptance testing is the last
Testing step to ensure the VoIP system’s integrity
Testing includes developing a compre- and viability. he technology to enable a VoIP system
hensive test plan, test procedures, and a
test schedule. The test plan should cover
how to test the VoIP system equipment.
For Noblis, acceptance testing did not
involve formal procedures. Rather, engi-
neering staff and CIM members exercised
T has undergone signiﬁcant advances in
reliability and ease of implementa-
tion, but activating a VoIP system within
Procedures should detail the actual equip- the system to ensure that the functions and an enterprise still requires careful planning,
ment settings and conﬁgurations to be test- features operated according to their speci- execution, and testing. It also requires ap-
ed, test sequences, and expected results. A ﬁcations and reported any operational ab- preciating the technical culture shock to
good strategy is to make the test schedule normalities to Noblis’ help desk where net- management, staff, and technical support
part of the project WBS. work administrators analyzed and resolved personnel as they shift to a VoIP system.
the call problems after the level one trou- An organization must strive to view the re-
Initial ble desk operators took the calls. The help quirements and service transition through
Initial testing should be done in a labora- desk continues to maintain a list of all VoIP the users’ eyes so that individuals will em-
tory setting, in which the individual VoIP operational troubles that stem from VoIP brace the new VoIP service.
system components are tested as they equipment malfunction. All solution strate-
would be conﬁgured when installed. Tests gies are kept in a lessons learned database,
evaluate the components’ ability to handle which CIM members access to maintain
all potential call types and use switch and and troubleshoot the VoIP system. Guerin C. Goldsmith is a principal engineer
at Noblis, where his experience includes
function settings that would occur in the telecommunications, VoIP, digital transmission,
real environment. Test documentation IP-based video teleconferencing, analog/digital
video technology and production, and program
Noblis’ initial test procedures were Test documentation is important in all management. He received an MSEE from the
based on inputs from Cisco (its equipment testing phases, but it is especially impor- University of Illinois. Contact him at ggoldsmi@
vendor), SBC (its systems integrator), and tant for veriﬁcation testing, since testers
its CIM and engineering staff. All inter- are recording the system’s performance at
ested parties reviewed these procedures at the beginning of its life cycle. Such knowl- Hsiaosu Hsiung is a senior fellow at Noblis, where
his experience includes large-network simulation
SBC’s facility before any testing began, edge is useful in activating equipment war- and design, next-generation Internet, VoIP,
and Noblis’ engineering staff participated ranties or in reviewing any performance electronic commerce, secured access control
technologies, Internet Protocol cyber attack,
in the actual testing. service level agreements in contractor or public key infrastructure technologies, and data
vendor agreements. communications. He received an MS in computer
science from the University of Connecticut.
Veriﬁcation Noblis documented test results and Contact him at firstname.lastname@example.org.
Veriﬁcation testing occurs after the equipment settings for the tests, which it
equipment is installed but before it be- retains with other as-built drawings and
comes operational. In essence, veriﬁca- system documentation.
14 Vo i c e o v e r I n t e r n e t P r o t o c o l