Emergency response to network outages is something that all Colorado OSP operators should be concerned with. Building the right SOP and selecting the right emergency repair vendors are both critical to minimizing network downtime in the event of a physical damage.
Intro to Trace Fiber Services
Trace Fiber Services is a Colorado-based telecom contractor that specializes in underground horizontal directional drilling, fiber splicing, and light civil construction. The Trace team supports a multitude of customers across the Mountain West with new construction, maintenance tasks, and emergency repair.
Outside Plant (OSP) Fiber Outagesin Colorado
Nobody likes when the internet work, but that same frustration is magnified by thousands when backbone or backhaul OSP networks go down. These networks power the internet and our cellular phone networks, so they impact much wider area or groups of people.
What is an OSP outage?
An OSP outage is defined as a situation when an OutSide Plant (OSP) network becomes non-operable. To fully understand what this means, it is important to conceptualize what an OSP network consists of. OSP networks connect data centers, distribution points (e.g., 5G Cell Towers), and other buildings. This network consists of fiber optic cables, conduits, enclosures, and transmission/receiving equipment. This network generally runs through Rights of Way (ROWs) across public and private land.What causes OSP outages in Colorado?
As transmission equipment and the physical fiber optic cable has become more reliable, the major cause of an OSP outage is a physical disruption of the fiber cable. Any disruption of the fiber, conduit, or communications gear in a handhold could cause a fiber outage.
Some of the most common underground utility disruption causes include:
-
Excavation mishaps (often caused by failure to locate, improper locates, or ignoring the locate data)
-
Animal intervention (generally in ducting, not cables)
-
Vehicle accidents
-
Natural disasters (think earthquakes, mudslides, flooding)
In some cases, the organizations performing underground construction may accidentally damage existing fiber or other underground utilities. This has unfortunately been the case in Colorado Springs where sub-contractors for local internet providers hit other utilities over 700 times.
How expensive is a fiber strike or disruption in Colorado?
The cost of a fiber OSP outage can be measured in two different ways. First the cost of downtime to the fiber operator and second, the cost to remediate the situation and restore connectivity. Additionally, some municipal governments may issue fines to organizations that strike other utilities.
Cost of downtime to network operator: Every year, the mobile phone industry in the US loses $15B from network outages. Obviously this includes the variety of situations that can cause outages, but it’s very clear that outages are expensive for network operators. Depending on the fiber line’s use, length, and owner, downtime could cost the operator anywhere between $25k / hour and $300k / hour.
Cost to remediate OSP disruption in Colorado: The second cost of an outage is for a contractor or set of contractors to remediate the damage in order to restore connectivity to the line. This cost also varies on the situation based on how much work is required to locate, expose, replace conduit, re-pull fiber, and then splice the fiber. In complicated situations or extremely remote locations, this cost could break into the six digits.
Cost of fines from municipal governments: Some local governments in Colorado (such as Colorado Springs or Colorado 811) have instituted steep fines (up to $75,000 per instance) against contractors who hit underground utilities.
Who pays to fix a damaged fiber OSP network in Colorado?
Initially, the fiber OSP operator will pay to have a contractor remediate the situation. This is simply a matter of needing to get the network operational as fast as possible. So the OSP operator will issue a work order or other authorization to an emergency repair firm, like Trace Fiber Services, to make the repair.
In some instances, an OSP operator may issue multiple work orders for different services. This is generally the case when one single organization doesn’t have all of the capabilities necessary to make the repair.
Once the repair has been made, the network operator will seek damages from the party at fault for the outage. This could be done by issuing an invoice for the repair to an irresponsible construction company or taking someone to civil court.
Preparation for OSP outages
How does the Colorado environment impact fiber network decisions?
Colorado’s varied physical environment along with it’s seasonal climate cause network operators to generally put cabling underground. In some cases, it’s cheaper to utilize existing telephone pole infrastructure to place fiber networks in an aerial configuration. In Colorado, aerial fiber networks are generally only used for last mile (FTTH) or middle mile portions of larger networks due to their exposure to the elements.
Elsewhere in Colorado, fiber conduit is generally installed with a horizontal directional drill (HDD) or through trenching. Trenching is usually performed in extremely rural circumstances where it’s not cost effective to drill (and there a minimal obstructions).
If you are an OSP network operator in Colorado, you should make sure that your preferred vendor for emergency callout work is capable of working in all of the environments and installation types that your network uses. If you have questions, contact Trace Fiber Services to learn about our expansive service offering.
How can Colorado network operators prepare for damages?
In today’s construction world, damages are an unfortunate reality. So OSP fiber network operators must establish plans to respond quickly and effectively to network outages.
These preparations should include:
-
Customer Preparation
-
Rapid notification to impacted customers
-
Plan to handle customer questions and issues
-
-
Network Operations Preparation
-
Monitoring and interruption detection
-
Fail over plans to utilize other fiber routes (where possible)
-
Agreements with service providers for additional capacity
-
-
External Operations Preparation
-
SOPs for activating repair contractors
-
Pre-negotiated service agreements with repair contractors
-
Stockpile of unique or long lead network components
-
Plan for activating internal personnel to support network repair
-
Damage claim information collection requirements and forms
-
Why is a do-it-all contractor critical for restoring network connections?
When a fiber strike occurs, minimizing the outage time is critical to maintaining network availability. In this case, OSP operators should work with ROW contractors that internally have all of the necessary capabilities to repair the line. If a contractor must subcontract critical functions (e.g., fiber splicing) the emergency call out may be unnecessarily extended as they work the phones to find someone to help.
Working with an all-in-one contractor like Trace Fiber Services can save time and money by centralizing the damage alert to one organization. Additionally, crews who work together are generally more productive and know each other’s SOPs than different companies coming together during an emergency.
Collaborating with Colorado-Based Emergency Service Providers
Working with a telecommunications contractor for the first time during a critical outage isn’t the best time to test out capabilities. Ideally, you have ‘test driven’ the contractor through less time sensitive work like a maintenance line relocation. This practice highlights the value of collaborating with a Colorado-based OSP construction firm like Trace Fiber Services.
During an outage, clear communication and rapid response are paramount to restoring critical network capability. Once a damage is identified, there should be a clear communication path for the network operator to alert the contractor. The underground construction contractor will need the 4W’s (where is the damage, what happened, who else will be involved, what site considerations are there) to be able to establish an effective response.
Once the emergency response contractor has the information, they’ll need to activate their crews and assets to respond. This could be some combination of light civil equipment (e.g., mini excavators), drilling assets, lighting, manual digging equipment, traffic control, and fiber splicing gear. Additionally, they will likely need emergency utility locate support to ensure that damage to other utilities does not occur during the response. If the company doesn’t have these assets and crews locally, the response will take more time as they need to work through other companies.
Partnering with Reliable OSP Experts in Colorado
As highlighted above, having reliable emergency partners is critical to rapid network restoration during a physical damage. There are many options, including large national contractors, but prioritizing local capabilities can yield more effective emergency responses.
Why Local Capabilities Matters for Emergency Fiber Optic Contractors
Local capabilities are critical to a fiber damage because there just isn’t time to muster crews from out of state. Additionally, local contractors should have existing construction licenses, relationships with municipal authorities, and are likely to best understand the environment.
OSP operators should look for three critical aspects when selecting potential contractor for emergency fiber repair service:
-
Technical experience – the contractor should have deep technical experience in civil construction and fiber optic repair. Ideally, this contractor has demonstrated these capabilities on other, less time sensitive projects in the past.
-
Local assets / crews – the ideal emergency contractor has the full spectrum of repair assets and crews in the service area. They don’t rely on sub-contractors to perform critical portions of a damage response.
-
Emergency experience – the emergency response contractor should be well versed in responding to emergent situations. This will be obvious in how they muster, what capabilities they bring to the damage site, and how they operate at the damage site.
Questions to Ask When Choosing an OSP Partner for Emergency Response
When interviewing a potential emergency response partner, you should ask them:
-
What construction and fiber assets do you own?
-
How many of those assets and crews are based in Colorado?
-
How many internal crews do you operate?
-
How often do you perform emergency fiber damage repairs?
-
What experience do you have responding in bad weather?
-
Are your crews willing to work at night to fix an outage?
If you want to learn more about how Trace Fiber Services can support you as an emergency fiber repair contractor, get in touch with our team today!