Reduce Technical Debt and Prevent System Outages
As a business owner, you’re well aware that tech plays an important part in keeping your company up and running. However, what happens when technical debt starts to build-up? Recent FAA IT outages caused massive flight delays nationwide, which came directly on the heels of a Southwest Airlines mass flight cancellation which was also due to IT issues. For IT folks, this is proof of the significant consequences businesses face if their tech infrastructure isn’t managed properly.
It’s a scary reminder that businesses, no matter their size, are not immune to technical difficulties. There are steps you can take as a business owner to prepare and prevent outages in your own company. Read on to learn what happened, why it resulted in so many flight cancellations, and what you should do as a business to prepare and prevent outages like what happened in the recent IT outages for Southwest and the FAA.
Overview of the Recent System Outages Experienced by Southwest Airlines and the FAA
The Federal Aviation Administration (FAA) and Southwest Airlines have been in the news recently for separate system outages. The FAA outage caused widespread delays across the country. As a business owner, you may be wondering what these outages mean for you and your company’s plan to reduce technical debt. Here’s a quick rundown of what happened and what you need to know.
- On January 11, 2023, the Federal Aviation Administration’s (FAA) computer systems went down for eight hours.
- The outage caused over 1,300 flights to be delayed or cancelled across the country.
- Southwest Airlines was one of the airlines most affected by the outage, with over 400 delays and cancellations.
- The FAA attributed the outage to a “corrupted file.”
Cause of FAA Outage
The FAA’s systems were down for 8 hours. A corrupted file in the FAA’s computer system caused an outage that disrupted more than 1,300 flights across the United States. This corrupted file caused an issue with data processing and communication systems connected to air traffic control centers throughout the country. As a result, all flights destined for airports within a 200-mile radius of any of these centers were either delayed or canceled altogether.
If the FAA had been keeping up with their technology, reducing technical debt actively, and creating redundancy across their network, there is a good chance this type of issue would have been prevented.
Even the President and CEO of the U.S. Travel Association agrees, saying “Today’s FAA catastrophic system failure is a clear sign that America’s transportation network desperately needs significant upgrades.”
Southwest Airlines IT Meltdown
In another recent fiasco, Southwest had a separate outage that caused mass cancellations of over 2,500 flights. There were multiple causes including an unprecedented winter storm, staff shortages and of course, IT systems. No other airline had a number of cancellations as high as Southwest, and this can be directly attributed to their outdated IT systems and a lack of managing technical debt.
There were multiple issues with their IT systems, between the phone system not working, grossly outdated scheduling software and antiquated IT infrastructure. According to one report, the process of maching up the crew with the aircraft simply could not be handled with their technology.
There were even reports of fully manned crews and planes ready, but unable to take off due to these outdated systems.
Southwest’s Existing Technical Debt
The problems with Southwest have been around for a very long time. However a complete and total crash and not being able to keep up with the flight changes and weather exasperated the problem leading to this fiasco. Unfortunately their systems were simply not prepared to deal with anything outside of the normal day to day operations.
The IT systems with Southwest haven’t changed much since the 1990s due to unwillingness to invest in technology. Between their phone systems, computers, software and infrastructure, they simply should have upgraded their systems years ago.
Manage Technical Debt to Avoid Business Interruptions
The FAA’s systems are not only used by airlines; they are also used by airports, air traffic controllers, and other aviation-related entities. This means that the impact of the outage was felt far beyond just the major airlines. Businesses that rely on these services were forced to shut down operations while the systems were being restored. In some cases, this meant lost time and money as well as frustrated customers who had been expecting timely service or deliveries.
For example, businesses that rely on shipping goods via air freight were greatly impacted by this outage as their goods could not be shipped until the issue was resolved. Additionally, companies whose employees frequently fly for business meetings were also affected as they could not send workers out of town until after the problem was fixed.
What Does This Mean For Your Business?
This major incident is a reminder of how dependent businesses are on technology and third-party vendors. As we increasingly rely on technology and automation in our day-to-day operations, it’s crucial that we have contingency plans in place in case something like this happens again. From ensuring our data is backed up regularly, systems are kept up with to reduce technical debt, and having redundant systems for critical processes, businesses should be prepared for any potential disruptions or outages that may occur in the future.
Prepare and Prevent Your Business from Outages
Prevention is the best medicine in this case. Businesses should strive to reduce technical debt to prevent disasters from occurring in their own business. Tech debt can creep up on you, foregoing that switch or server this year and kicking the can down the road will only pile things up and can be disastrous. There are a few steps to help you manage technical debt. Let’s take a look at some of those steps below.
Know Your Systems
It is essential to know exactly what systems your business uses and how they work together. Understanding which systems are interconnected and how they interact with one another is key to avoiding outages like the ones experienced by the FAA and Southwest Airlines. Even having a basic list kept in Microsoft Excel for example is better than having no inventory list in place.
Backup Solutions
Backup solutions are a necessity these days for your data. Making sure an entire image capture of your server is backed up to the cloud, with an easy path to restore is critical. Don’t just backup, make sure someone is monitoring your backups to make sure they are successful, and tested on a regular basis.
Monitoring Services
In addition to having multiple backups of your data, monitoring performance metrics is also key when trying to detect potential problems with your system before they lead to an outage. Monitoring performance metrics helps identify trends over time so that potential problems can be identified before they lead to an outage or performance issues. Additionally, this type of monitoring helps identify areas where improvements may need to be made so that optimal performance is maintained throughout all of your systems.
Update Server and Network Equipment
Keeping your IT Infrastructure on a regular replacement schedule is a necessary component to solve technical debt. Maintaining a warranty on all systems throughout their lifecycle, and replacing after that warranty is expired is a best practice when it comes to maintaining your IT systems. This way, you never have to worry about an old system bogging down your network, or a failed hardware component being defective and not being covered by warranty services.
Update Software Regularly
Software updates help keep your systems running efficiently and securely by addressing any known bugs or security vulnerabilities in the software itself. Keeping up with updates is important because it helps ensure that your system remains secure from malicious attacks or hackers who may be trying to access sensitive information in your system without authorization. Aside from regular software updates, new versions should be upgraded to when available to ensure new features are added regularly and your software stays under support through the vendor.
Security Protocols
Finally, creating strong security protocols is essential for protecting against outages—or worse, malicious attacks or data breaches. Strict password policies, Multi-factor authentication, Managed Detection and Response services and an enterprise grade firewall are a few of the mandatory IT security services needed to protect your business these days.
Conclusion
Outages like those experienced by the FAA and Southwest Airlines are nothing short of disastrous for businesses. However, with proper preparation and prevention measures in place, such outages can largely be avoided or at least minimized significantly. By investing in backup solutions, monitoring services, reducing technical debt and implementing strong security protocols, you’ll be better prepared to face any potential IT related issues. No matter what type of business you run, these measures are essential for safeguarding against costly downtime in the future.
Give yourself peace of mind that you are taking steps towards safeguarding your business from outages related to technical debt. Contact Sirius Office Solutions and take one step closer to upgrading your business’ tech stack!
