When organizations are seeking out the right OT network security for their OT environments and OT devices, the clear objective is to decrease and eliminate risks. Too often organizations only adopt the minimal level of security. While each organization defines its security risk levels, it is often based on their production environments, industrial devices and the critical risk factor of their facility production.
Many organizations will use different techniques to manage their risks, but one of the most common methods is patching. At the heart of every security strategy, patching is one of the key elements to securing any potential vulnerabilities within an organization. Despite patching being commonly used in risk management strategies, advancing patching for OT devices is still a work in progress.
Patch management in OT and Industrial Control Systems (ICS) comes with many security challenges. From lack of OT experts, proprietary hardware and software, compliance regulation reporting, minimal testing equipment and device and system maintenance, many industrial organizations struggle to clearly understand how they need to patch their vulnerable devices. This results in unmanaged patches.
Improve OT Network Security with industrial Device Vulnerability Management
When deciding what needs to be patched, security teams need to decide and evaluate the practicability of OT patching for their organization. With OT environments, applying patches is a balance that is based on the security benefits of what the patch provides versus the disruption of operational activities due to patching. These both are crucial factors to consider when patching OT environments.
With every standard OT network security patching program, it starts with 4 steps to success. The first step is to detect and discover which assets you have within your OT environments. The next step is to assess the industrial devices and OT equipment for vulnerabilities. There can be different types of vulnerabilities but most vulnerabilities will fall under the categories of security risks or software and device misconfiguration.
The third step is to analyze and prioritize the vulnerabilities. Here is where organizations learn which devices are vulnerable and which are not and what priority should be assigned to patch the vulnerable devices. In some cases in this step, organizations will question should we even patch the vulnerability or why should we care about it? While it’s an organization’s job to decide what to patch and not to patch, we recommend patching all vulnerabilities to ensure the security of an organization will be secure.
The fourth and final step is remediating the vulnerability. This is where security teams will patch the vulnerabilities within their industrial devices. For example, patching a PLC, fixing device configurations and more.
IT Patching Does Not Work in OT
Today’s organizations need to run different security testing to clearly understand which vulnerabilities they have in their OT environments. In IT security, most organizations will adopt vulnerability scanning tools. With asset vulnerability scans, these are typically assessed based on port enumeration and authenticating to the devices to get comprehensive configuration/policy and registry information. While this might be useful for IT security it doesn’t work with OT network security.
For example, an automotive manufacturer in Germany had a couple of critical servers that were connected to their production line. Their servers crashed after scanning for vulnerabilities. They only scanned to see if they had one vulnerability in their environment. While they knew exactly what they were scanning for, it resulted in their OT environments being affected. The servers were a key part of their manufacturing process and the failure caused downtime and a loss of revenue of over a million dollars.
When they investigated what the problem was, they identified that the scanner opened 13 sockets while the servers only supported up to four sockets in parallel. They flooded the servers with a capacity of three times higher than what was normal. The servers were unable to handle their operational processes and crashed.
The lesson learned in this example is if you come with an IT security approach of scanning for vulnerabilities with OT, an organization might cause more damage than a cyber attack.
Don’t Forget about the Costs
Now that organizations know the four-step process of device patching management, the cost of patching is a crucial aspect they must be aware of. Once organizations have all the information (asset inventory, network mapping, disclosure sources and maps of vulnerabilities) and they are ready to patch the vulnerabilities they need to understand the price to patch.
With each patching process, there is a different cost associated with it and it shouldn’t be taken lightly. Every industrial organization’s biggest nightmare is production downtime. With every patching process, the organization will experience some kind of downtime but when managed correctly, it will only be for a short period of time. However, when an organization doesn’t manage the industrial device management process correctly it can financially impact the organization not only in the production line but also in the headlines.
At SCADAfence, we have helped many industrial organizations to patch their OT devices. One common theme we have seen is when we show the organization the different vulnerabilities, they go ahead and they only fix that vulnerability and don’t fix their entire vulnerable device. This is a huge problem because if an organization doesn’t fix the core issue of a vulnerability, it will be easy for attackers to find another vulnerability. Organizations need to patch the entire device to ensure no vulnerabilities are left behind.
Industrial Device and System Patching Comes with Benefits
Now that we explained the risks of the cost impact of improper patching methods, organizations should consider what are the benefits. While patching OT devices can be risky at times for devices and servers to crash which results in downtime, there is a real benefit to patching.
One of the biggest benefits that organizations experience is having an asset inventory, this is a great place to start. Adopting an automatic asset inventory provides the most efficient and the most accurate method to visually manage an organization's industrial devices and understand if there are vulnerabilities in those devices. By mapping vulnerabilities to assets, it will allow organizations to prioritize the patching of vulnerable devices and increase the visibility into the connection points of each device on the OT network.
In addition, we recommend isolating vulnerable devices from the OT network. In some cases, some OT devices will have a vulnerability that does not have a patch available. This could result from the protocols of a specific industrial device having too lenient restrictions which would make the device more vulnerable. By isolating vulnerable devices it will help organizations from allowing attackers to move laterally within their OT environment.
Simplifying Industrial Device Management
Moving forward, organizations need to assume that there are always unpatched devices in their OT networks due to not being able to be patched or because they haven’t been patched yet. If organizations adopt a concrete industrial device patching management strategy it will allow their security teams to efficiently detect vulnerabilities and attacks early on before attackers exploit the devices.
To answer the question, “to patch or not to patch”, is not a simple yes or no answer.
We recommend adopting an industrial device patching approach based on actual trial testing with different scenarios. By understanding real-time device data and vulnerability information it will allow organizations to prioritize their patching of industrial devices.
To learn more about industrial device patching, on November 10th at 11 am EST, Rapid7 and SCADAfence will host a joint webinar: The Comprehensive Guide to Industrial Device Patching. Register here.
During the webinar, we will provide three excellent tools that will help you with the decision-making process if “to patch, workaround or do nothing.”