LLU Disconnections Update - PlusNet by wulinqing

VIEWS: 8 PAGES: 2

									LLU Disconnections

On Saturday Plusnet (with the help of contributors on our portal forums and Usenet) identified a
common pattern for the frequent disconnects being reported by some of our customers who's
service is provided by Tiscali. Problem number 34944 was raised at 2pm on Saturday
26/Aug/2006 and assigned priority one, Plusnet's highest classification. After eight days of
investigation no definite cause has been found, but I'm happy to update you about what is known,
the steps that have been taken and how its being progressed.


The problem
The symptoms of this problem are that customer are reguarly disconnected and that during the
disconnection the 'sync' light on their broadband modem/router remains solidly lit. The effected
customers find they can reconnect instantaneously or within minutes. If these customer's connect
via the Tiscali test login (testing@dslconnect.co.uk) they receive a stable connection although this
login doesn't allow them to browse the Internet.

Situations where the sync light flickers, is not lit, where authentication fails consistently or where
connecting to the test login fails to provide a stable connection are unrelated and should be raised
by the normal fault procedures.

The fact the connections are was stable whilst connected to the Tiscali's test line proves the fault to
lies not on the customers line but in Plusnet or its supplier's networks. Our observations suggest that
up to one thousand customers are effected, just over five and a half percent of the approximately
Eighteen thousand customers who's services we provide by Tiscali. The disconnects occur between
4 and 40 times per day. Customers are disconnected simultaneously but not all affected customers
are disconnected each time.

Timeline

Saturday (26th)
On the day the problem was reported the on duty network engineer (Adeyinka Makinde) was called
out. He inspected the network monitoring in place and quickly ruled an equipment failure in our
equipment or in Tiscali's equipment at the point they hand customer connections over to ourselves.
Reviewing our internal 'users connected via Tiscali' graphs Adeyinka identified a common 'blip' the
last of which corresponded to the last drop reported by customers. Tiscali's first line support were
alerted at 15:03.

At 18:00 the problem was escalated to Mark Lovely, the main engineer responsible for our
Broadband interconnects and (fortuitously) second line network support that weekend. Mark began
investigating the problem using the diagnostic facilities of our ERX devices, which are the first item
of Plusnet's equipment that receives data from Tiscali's network.

He quickly identified the users affected, gained confidence that the cause was within the Tiscali
network. Mark passed his finding and extensive debugging information to Tiscali's engineers. We
were asked to review the affected users and identify any geographic, product or customer
equipment pattern. No such pattern was identified. At the end of the Saturday our supplier was
unable to confirm ownership of the fault.
Sunday (27th) and Monday (28th)
Tiscali escalated the report to their second line support. Mark continued to monitor the
disconnections and work with Tiscali's engineers. Together they were able to watch several failures
as they occurred. By analysing the network path of customer disconnections Mark was also able to
rule out a intermittent failure within one of our ERX's line cards (input modules) the Tiscali
equipment they connect to or the fibre optical cables with interconnect them.

Tuesday (29th)
The problem was internally escalated to Philip Webb, Plusnet's highest ranking network engineer
and our final technical authority on network infrastructure. Mark and Phil jointly reviewed the
configuration of the ERX's and ruled out and misconfiguration of this equipment. Following
further discussions and reviewing Tiscali's second line engineers were able to accept ownership of
the fault and escalate the problem to third their line support.

Wednesday (30th)
A major failure of another part of the Tiscali network took priority and occupied their third line
support. We continued contact their network operation staff to indicate further disconnection events.

Thursday (31st)
The problem was formally escalated to the Plusnet CEO (Lee Strafford) on Thursday. Mr Strafford
and his team of directors made representations to their contacts within Tiscali. They will continue
to pursue the matter at this level until a resolution is achieved.

Friday (1st Sept)
Tiscali's technical account manager for Plusnet working with their third line support identified the
source of the authentication system errors and these were corrected. Tiscali identified the
disconnections as originating from a section of their network infrastructure which is leased from
NTL. Tiscali have raised the issue with NTL and are progressing the investigation with them.

Further Work
We continue to run our ERX devices with diagnostic logging enabled. The information is reviewed
and summarised daily. The results are shared with Tiscali.

Plusnet's network team contact their opposite numbers at least Tiscali twice a day. The services of
our network engineers have been placed at their disposal and on Monday we will reiterate our offer
to source external expertise or diagnostic equipment to help identify the cause.

								
To top