CABC_6 Location: fiber cut affecting CN2 connectivity

Incident Report for Bandwagon Host Status

Resolved

We observed full restoration of our link. We will be closing this incident at this time.
Posted Apr 30, 2025 - 22:57 PDT

Update

2025-05-01 01:02:36 GMT - Splicing Operations have commenced on the north side of the 88 count and 216 count fiber cables. Splicing on the south end of the fiber cables is expected to commence at approximately 01:20 GMT.

2025-05-01 00:26:34 GMT - Field Operations have advised that placement of the fiber cables has been completed. Splicers have begun prepping the fiber ends for splicing operations. Splicing Operations are expected to commence by approximately 02:15 GMT and be completed at approximately 06:30 GMT.

2025-04-30 23:44:40 GMT - The train has cleared the area and crews are resuming efforts to pull the replacement cables into place. Crews have advised that cable placement is expected to be completed by approximately 00:45 GMT on May 01, 2025. Once the cables are placed the fiber ends will be prepared for splicing.

2025-04-30 23:33:06 GMT - The restoration effort was temporarily halted as a result of a train needing to stop over the restoration zone. This is a rule in place by the railroad administrator, and is a legally enforceable mandate. The train is again moving, but at a slow pace. Repair efforts are expected to resume momentarily.

2025-04-30 22:52:26 GMT - The Event Management team is actively working to gather the latest information from the Network Operations Center (NOC) regarding the service interruption. Any new developments will be communicated as soon as possible.

2025-04-30 21:56:00 GMT - Crews have new cables at the South Splice location and are waiting for train to go through at the moment. The North splice is expected to have the fiber cables approximately. 45 minutes after the train clears the location.

2025-04-30 21:07:46 GMT - Crews have completed placing the replacement fiber cables over the railroad. Crews are continuing efforts to pull the cable into the north and south hand holes. Once the fiber is placed the ends will be prepped for splicing operations.

2025-04-30 20:22:06 GMT - The Lumen NOC has engaged repair crews for an update on cable placement.

2025-04-30 18:51:43 GMT - First cable continues to be pulled and placed. The pull over the railroad track is being directed by the local rail authority flaggers and must be done in a safe and manageable manner. Once the first cable is pulled crews will commence pulling the second cable on over the track.

2025-04-30 17:56:49 GMT - Deployment is underway to place the temporary aerial cable. As the aerial cable crosses over a railroad track, crews are only being allowed to pull one cable at a time. Construction crews are onsite and commenced with excavation to dig a trench for the section of the temporary cable that is overground.

2025-04-30 16:55:10 GMT - Splice crews are prepping the existing tail end of the fiber for splicing. Aerial crews are placing the replacement cable on the temporary route. Construction crews continue to enroute to the fault stie.

2025-04-30 15:42:37 GMT - Locates have arrived onsite and marking buried cables and lines. Aerial crew have arrived onsite.

2025-04-30 15:27:38 GMT - Cable reel is onsite, and aerial crews will arrive onsite shortly. Once onsite crews will begin prepping and deploy the aerial cable on the temporary route. Construction crews are enroute ETA 17:00 GMT.

2025-04-30 14:58:13 GMT - Emergency locates are enroute. The ETA is 30 minutes.

2025-04-30 14:43:44 GMT - Construction and aerial crews are obtaining the necessary equipment and material for repairs at this time. Emergency locates have been engaged; however, no ETA is currently available.

2025-04-30 13:23:09 GMT - Emergency locates have been engaged to mark buried utilities in the area. The aerial foreman has arrived onsite and has advised the cable placement should begin at approximately 15:00 GMT.
Posted Apr 30, 2025 - 18:36 PDT

Update

More updates from last night, repair efforts still underway:

Notes History:

2025-04-30 12:32:34 GMT - A temporary aerial fiber will be placed in order to expedite service restoration. Construction crews will begin to excavate on the east and west areas of the cable to create a path for the cable to run. Once the temporary cable is in place, a splice team will begin to prepare the ends for splicing.

2025-04-30 11:29:16 GMT - The Lumen NOC, OSPE, and construction crew are investigating the route to determine the most viable option for a temporary repair. As the damage is significant along the span, a temporary repair will expedite service restoration.

2025-04-30 10:36:37 GMT - The Lumen NOC is in communication with the construction crew onsite to obtain an update regarding restoration efforts.

2025-04-30 09:51:05 GMT - The construction crew continues to work to reach and assess the affected conduits.

2025-04-30 07:45:51 GMT - The OSPE has reported that teams remain committed to their diligent assessment of the duct system. However, progress has been gradual, primarily due to the significant conduit span which is approximately 1,500 feet between manholes. As such, the next notification will be provided at 09:45 GMT or as additional information is obtained.


2025-04-30 06:52:41 GMT -
Field Operations reports that teams are diligently working on duct system assessments, though progress remains slow. Crews anticipate that the affected cables have sustained partial damage and will require a full cut to facilitate repairs once a viable restoration plan is determined.
To keep impacted customers informed, those utilizing intact fiber strands have been proactively notified of the ongoing repair efforts. Meanwhile, crews are actively evaluating the most efficient approach to expedite service recovery and minimize downtime.

2025-04-30 06:01:17 GMT - Due to the nature of the fiber damage, and the conditions with which teams are working to finish duct system assessments, efforts remain ongoing and are progressing slowly. Crews have also advised that the affected cables are anticipated to be partially damaged and will require a full cut to complete repairs when assessments determine the viable repair plan. As such, the services on intact fiber strands have been notified proactively of the ongoing repair efforts, and crews are working to determine the most viable way to expedite the recovery of services at this time.

2025-04-30 05:01:55 GMT - Construction crews onsite are working to use pressurized air to verify the duct systems at this time. Once completed for assessments, further details pertaining to the permanent or temporary repair plan timeframes will be provided.

2025-04-30 04:04:25 GMT - Crews onsite are continuing efforts to verify the duct system conditions. The OSPE has advised that should viable, undamaged duct be available, crews will work to place the new replacement fiber sections through the viable ducts for repairs. If crews are unable to locate undamaged duct work, more complicated repair efforts to place temporary cable sections to expedite repairs while pending permanent repair efforts will be carried out.

2025-04-30 03:09:56 GMT - The OSPE has advised that the section of damaged fiber is located within a duct system approximately 800 feet in length and approximately 15 feet deep. Crews are working to determine whether undamaged ducts are present onsite as additional efforts are underway to locate the nearby splice enclosures for repair planning efforts onsite. Additionally, construction crews have arrived onsite with the exception of a crew enroute to procure additional 216-count cable. Onsite efforts to complete necessary assessments for repair planning are underway at this time, and further information will be provided as details become available.

2025-04-30 02:18:48 GMT - Construction crews remain enroute to the fault location at this time as teams were rerouted to obtain replacement fiber cable for repair efforts onsite. The OSPE has advised that an 88-count and 216-count buried fiber cables were damaged in a duct bank approximately 15 feet deep by unrelated construction work taking place in the area. Once onsite, construction teams will commence with efforts to determine whether undamaged duct system is present for the use of repair efforts onsite. Crews are anticipated to arrive shortly, and further details will be relayed as onsite assessments continue.

2025-04-30 01:22:01 GMT - Additional fiber testing was conducted across the fiber span, and measurement data continued to indicate a failure near the Portland end of the affected fiber span. As such, Field Operations dispatched to the indicated fault vicinity to traverse the span in search of the suspected damage and has advised that evidence of a cable damage has been identified within a railroad right-of-way (ROW). Construction and repair teams are enroute to assist with ongoing isolation and repair efforts, and an ETA of 02:00 GMT has been provided for construction crews. The construction foreman dispatched ahead of enroute teams and is working with the OSPE at the indicated fault location to verify the damage and complete preliminary assessments onsite at this time. Additional information pertaining to the failure will be provided as information becomes available.
Posted Apr 30, 2025 - 15:40 PDT

Identified

We have detected degraded latency between CABC_6 location to/from China. This incident is related to a fiber cut. Please refer to the information below:

2025-04-29 23:41:38 GMT - 2025-04-29 23:38:40 GMT - On April 29, 2025, at 23:27 GMT, Lumen discovered a service impact in Portland, OR. Your customer trouble ticket has been related to a network event; all network events are automatically escalated to Lumen leadership.

2025-04-30 00:33:30 GMT - The Lumen NOC was engaged, and preliminary research indicates that a suspected fiber fault is impacting client services. As such, Field Operations and the Outside Plant Engineer (OSPE) were engaged to assist with ongoing isolation efforts, and Field Operations is dispatching to a nearby facility to conduct fiber testing along the span in an effort to pinpoint the damage location. Additionally, remote testing was conducted on the span from an enabled device in Portland, and result indicate a break in the span near the Portland end of the fiber path. Splice and repair teams have been mobilized for dispatch to the indicated fault vicinity to assist with isolation efforts as additional fiber testing is actively underway, and an ETA for mobilized teams will be provided once established with the OSPE.
Posted Apr 29, 2025 - 19:29 PDT