OT and ICS Industry veteran Paul Smith, author of “Pentesting Industrial Control Systems” has recently joined the SCADAfence team in the role of Field CTO. We interviewed Paul to get his thoughts on the current state of OT security, challenges that need to be addressed and his predictions for the future.
He was interviewed by content marketing manager, Joan Weiner Levin.
Joan Weiner Levin: Hi Paul. Welcome to SCADAfence! We’re so excited to have you on board. Can you start by sharing a little bit about your background and why you are particularly interested in OT security.
Paul Smith: I grew up in Calgary, Alberta, Canada. They call us ‘little Texas’ because the economy is so heavily influenced by oil and gas. After a number of years working in the oil and gas sector, it felt almost natural for my father and I to start our own consulting company. Leveraging his years of experience and my computer science background. We performed forensic audits inside of the measurement space in oil and gas, which is a very niche vertical where we had to solve many interesting technical problems. I had spent my entire career until then looking through data and how systems are interrelated inside oil and gas trying to find answers and solutions to “Red Herring'' problems.
During a project that my father and I were working on, I met Austin Scott who presently works at Dragos, Austin at that time was working on a compressor upgrade project and he invited me out to a “CalSec” Calgary Security meetup. I was hooked, I started investing time in understanding how people formulated careers in this space. I then was invited to attend a “Red vs Blue” event that the Department of Homeland security was hosting out of Idaho National labs. While attending this event I met some of the industry's finest people, I still stay in touch with a number of individuals. It was from this event that I was eventually offered a job to join Lockheed Martin.
Shortly after this event I decided to attend a SANs conference in Orlando, it was really the only ICS related security course being offered. Justin Searle was the instructor and this is where I met Michael Assante and Rob Lee. Michael dropped in to give us a pep talk and welcome us to the industry as it was either the first class or second that had ever been presented. Rob Lee had just started Dragos at this time. When working at Lockheed Martin I had numerous discussions about buying two specific new startups in industry one being Dragos and the other was Indegy. Both companies were at a very early stage, Dragos hadn’t even commercially released CyberLens yet. Friends of mine were visiting Israel and got very excited by technology they saw created by a Team 8 foundry company, the product was called ICS Ranger, and that company would go on to come out of stealth mode and brand themselves as Claroty, shortly after this I met with one of the Nozomi founders and became enamored by the possibilities of the product and in the end started working for them for a period of time as well.
JL: What are some of your immediate goals in your new role as field CTO for SCADAfence? Like what do you hope to accomplish first?
Paul Smith: The first thing is making sure the SCADAfence Platform is the best performing product in the market.
We are now industry leaders, and I want to make sure that we always stay ahead of our competition.
JL: Why did you choose to join SCADAfence? You're a celebrity in our field. You're a well published author. You're also very well known in the industry. Why did you decide to be a part of the leadership in SCADAfence?
Paul Smith: I don’t know if I would say celebrity, maybe been around the block once or twice as for SCADAfence, it is a lean team, it's got the right funding. I like working with a company when it’s small, hungry, scrappy, and people are wearing multiple hats. It’s on the cusp of blowing up to be big, and that's something really alluring to me. I like it because now I can come in and put an idea on the table and we bat it around as a team and then we shape it, hone it, and finally we implement and run with it. We are in a constant state of innovation while exceeding customers expectations.
JL: How do you want to work with SCADAfence’s customers? What is your ideal customer relationship?
Paul Smith: I want to be a trusted advisor. I want our customers to know that they are first and foremost, we are addressing their concerns and features prior to chasing PR. I want SCADAfence to be the first thought in their heads. When they have a problem in their field or network, they can call us up. Queue up the shameful plug, but in all honesty I want the customers to know that they can call either our managed services team or professional service team and will get the answers they seek. Whether it is writing OT protocol rules, testing packet rules, writing yara rules, adding/removing firewall rules, performing firewall swap outs or whatever it happens to be, I want people to start thinking of us as unbiased experts in this field, the trusted advisors of OT Cyber Security.
JL: What are currently the biggest challenges in the world of OT Cyber security.
Paul Smith: Number one is staff. It's always been staff. Companies can't find enough of the best, well-qualified people that they need to hire.
Next, I’d say it’s human error. A lot of the OT security issues we see out there are operator error. Someone who is not properly educated on how to execute changes in an environment can accidentally take down an entire facility. We see this all the time.
For the real cyber threats, if we look beyond human error and its operational impact, I would say it's nation state threats. The threats and attacks that are happening inside of Ukraine as a result of the Russian attacks right now are pretty insane and indicative of what can happen.
JL: Let’s talk for a minute about the current situation in Ukraine. There have been a number of reported attempted cyber attacks against electrical stations and attempts to damage Ukraine’s fragile critical infrastructure. For those of us observing this from the west, from an OT perspective, what about this situation should alarm or concern us?
Paul Smith: I've had this conversation multiple times with people and they think Russia has all this old military hardware, these bombs and tanks and infantry and it's falling apart.
But what you don't see is the cyber warfare going on in the back-end. The next world war isn't going to be fought with guns and traditional weaponry, it's going to be fought in cyberspace. You can cause a country to essentially implode just by knocking out their critical infrastructure.
People have asked me, why isn’t Russia just sending more people in on the ground. And I tell them, it’s because you don't see what's happening on the back-end. That’s a major part of the war. If you take down a city like New York, and they can't get power back up in under two weeks, you don't even have to shoot a single bullet. People will turn on each other, they'll figure out ways to survive at all costs. Remember no power means no pumps, no pumps no fresh water, and even worse… no Twitter! I’ll say this, you take down critical infrastructure, you can take down a country.
JL: Is this nightmare scenario preventable?
Paul Smith: To a certain degree, yes. But the problem with technology, the beauty and the problem, is that it's always evolving. And we're always innovating. But the cost of innovation is security. To be new and leading is great, but it doesn't always mean it's new, leading, and secure. Security is usually an afterthought.
A lot of engineering companies are trying to change that and put security in the design, but you can’t always do that. You don't know what you're securing, because if you're trying to engineer to be secure, then it is near impossible to innovate at the same time.
JL: You mean security is an afterthought of design?
Paul Smith: Yes. But from a technology perspective, I don't see this as a problem. Because if you try to put security into your engineering design, it will actually stifle innovation. For example, if an organization tried to create certain things to be completely secure, they would never be able to build them. Because they could have never innovated past the security boundaries that would have to be put into place. If you always put boundaries there, and say you can't go past these boundaries then you'll never innovate past the boundaries.
We haven't invented the next thing that you have to secure yet. If you don't innovate past that, then there's no chance of ever seeing what the next wave of security is going to have to be, and that's why I say it's a mixed bag. Can we secure things? Absolutely. But as we innovate we have this lag until we find the security gaps. So we invent a new thing, and then, there's the gaps. Now we have to invent something to secure that, because we've never had to secure this before.
A good example is self-driving cars with AI. There is this vision of what those self-driving cars need to be. But if someone puts some obstacle there, like a little orange dot, extended symbols on signage or something no one ever considered, it throws the whole self-driving car off course or can change a stop sign into a 45 m/hr speed sign, this is called adversarial ML attacks. No one could have predicted this because the fundamental technology for ML vision models had never been invented before.
JL: Let’s talk about legacy equipment, the older technology that is still running in manufacturing plants and critical infrastructure facilities. Is there technology still in place that is just too old to be secure, or is the older technology more secure because we’ve had more time to make it secure?
Paul Smith: I talk a lot about this topic, because I say the people who could actually fix the older technology are no longer with us and that is a major risk. So it's so archaic, that it's secure by nature. But just don't look at it, don’t touch it, because if it falls down, we'll never be able to fix it again. The old legacy stuff is hyper vulnerable. But more from an obsolescence perspective. Now if we talk moderate to old equipment, this is where you will find the highest most vulnerable assets. This technology was first/second generation adoption of ethernet cards, moving away from serial communications. It has become a major issue in industry where companies feel that if it is producing, don’t mess with it. The cost benefit analysis isn’t there for them to justify implementing new technologies yet. This is why we haven't seen solutions such as GE predicts and Siemens Mindsphere eclipse the market, new technologies just come with price tags that executive teams feel aren’t warranted.
JL: Why aren't more people choosing an OT cyber security career?
Paul Smith: The reason people don’t go into OT is because really OT security is two, maybe two-and-a-half different roles. Often, companies put up a job posting with a certain salary rate. My reaction is, “well, that's an interesting salary. The rate is lower than either an automation specialist or an IT specialist.” So they’re trying to pay someone who has to know both job roles less than either singular job.
If you combine the salary for both, then you could have more interesting opportunities for people to grow into. Someone would say to themselves, I’ve had to learn all the OT background, and now I have to learn all the IT cyber elements, like all the networking gear, all the endpoint technology, all the frameworks and security standards, and you only want to pay me same or even less than this other person, I'm just gonna do that other job, because I’ll get paid the exact same.
The market still hasn't adjusted salary rates for what it really means to do the job of OT cyber security.
JL: Let’s talk about the relationship between IT and OT. How should those two sides be working together, and what are they currently missing in that relationship?
Paul Smith: We've been talking about IT-OT convergence for a long time. And I think the gaps are slowly fading. I always said that it's easier to take an automation person, and maybe it's biased because I come from that side, and teach them the security side. As opposed to taking an IT security individual and teaching the automation side, because the automation side is more finicky, it’s not straightforward programming and implementation. Every decision being made inside the controllers can cause millions of dollars of impact.
There has to be more open conversations. For more mature companies, I would say, take one of your automation guys and put him right in your SOC and have him talk directly with all the IT staff there. A lot of these products feeding up data into a SOC use language that the IT analysts don't fully understand. Whereas if you put an automation guy there, he will be able to translate it. One of the value points for all this technology is we need to change the language to make sure we can communicate both to an automation specialist and an IT security specialist. Because if we put both languages in a security alert, it's easier for them to communicate and talk to each other.
JL: What is the role of governments in securing the OT? What is the ideal collaboration between the government and the private sector in securing public critical infrastructure?
Paul Smith: When it comes to private companies securing public critical infrastructure, there should be a lot of vetting and a lot of oversight, especially as it relates to major city centers. So if we’re discussing water treatment plants, or electrical facilities, if you're a third party vendor, you need to be subject to governance. Governments should have a big stick to use for enforcement because one bad incident can impact millions of lives.
There needs to be a heavier influence of government mandates and sanctions on third parties. And I know for a company like SCADAfence as an Israel-based company, selling into critical infrastructure in North America, that would put a little bit of a hamper on some sales, but it would also force us to comply with standards. Then everyone would feel safe, and there would be full transparency. And then once you have that stamp of approval facilities would be more comfortable working with approved third party vendors.
JL: What about governments encouraging private companies to do more for their OT security. Should the government be telling private manufacturers that they should do more to protect their OT?
Paul Smith: Yes. I do feel that the government needs to have more say in the manufacturing of products that impact people on a whole. Pharmaceuticals are a great example. If you have a disruption in drug supply, how many people is that impacting? If a company manufactures insulin pens for diabetics and their production goes down because of an OT security incident, and people miss their shots, you're killing people because of that cyber incident. So anything that can critically impact people's lives needs to have a little bit more government oversight. I don't like a lot of government controls. But I do feel in the case where people's lives can be impacted, government enforcement for companies to maintain a dedicated level of security practice is necessary.
JL: What is the future of OT security? What do the next three to five years look like?
Paul Smith: Oh, yes, that crystal ball stuff. Where we are now is still pretty immature in terms of OT security. From an industrial OT security perspective there were companies that were ahead of their time, and they owned the market share and then they just stopped innovating, and they fell apart. But I think we're coming full circle.
If you look at the way our technologies evolved, passive detection became super hot, super silver bullet, we're all in that market. Venture capital money was just being dumped into it. And now executives are concerned that they don't get full visibility that way. So we needed to add an active component, but everyone was staying away from active at that time. Now people are more open to active. Ten years ago, that’s how companies were doing this, and they had a massive install base. And they lost market share to passive companies. Now passive companies are supplying an active component/device as part of their product offering, which is where these other guys were 10 years ago. So it comes full circle.
I think you're gonna see a lot of IT implementations like XDR, and SOAR. Customers are going to start utilizing and coordinating their various security tools. There is a shortage in experienced individuals and the only way to offset that is more intelligence and more automation. Also companies are going to be a lot more open to agents installed out there in their OT environment, telling them what they see so they can be more secure. Agents in OT doesn't sound very sexy to me, because it's been done forever ago, but it's how the industry is maturing and evolving. So that is what I see in the next 3ish years, I predict that in the next 5 years there will be an adoption of AI at the edge providing interesting ML model solutions. I don’t want to give away too much of our secret sauce!
JL: Finally, because we always need to know. Do you have any pets?
Paul Smith: I do. I have a very sweet German Shepherd. Her name is Bailey, like the Irish cream, we named her because she is the same color as Baileys.