There was a spate of cyber attacks on grocery store freezers last month, and today the French government publicly claimed that the Iranian Revolutionary Guards were the culprits. (The attack involved taking over the control systems and running aggressive defrost cycles that cooked the food that was supposed to be preserved. There were also attacks on other networked hardware, but I will stick to freezers to reduce the word count.)
The initial reaction of a layperson is that it is absolutely insane that freezers across the free world are vulnerable to Iranian attacks. And as a fossil electrical engineer from the early 1990s, this layperson argument is fundamentally correct. It would be straightforward to have freezers that have almost all the current control capabilities without this vulnerability (at a cost) — but you can’t get there from here (easily).
The current dire state of industrial technology was predictable back when I was an electrical engineer — you only needed to pay attention to the archival record of the state of technology in the early 1990s (that is, the Dilbert comic strip).
Where We Are Now, And Why It Stinks
The article “Let’s Talk About Why all This OT Stuff is Connected” by Lesley “Hacks4Pancakes” explains the current situation, and how we got here. I recommend reading it if you are interested in this topic. I will offer a limited summary of points that are of interest to my argument.
The starting point to having an industrial control system that is safe from hackers is not connecting it to the public internet. (The jargon term is “air gapping”: there is air in between your control system circuits and the public internet, and air does not conduct electricity. You also cannot have wireless connections. The term originates from elsewhere.) However, air gapping is viewed as being effectively impossible.
You need central controls for many industrial systems. In the case of freezers, the defrost cycles are linked to logistical flows, which are centrally controlled. You cannot have individual stores deciding which freezers to defrost as well as the timing, as they might not have enough cold freezer space for a delivery. Phoning stores to give orders is error-prone and requires staff to be around at all times to pick up the phone. (The logistical challenges with freezer defrosting is a point that I saw emphasised elsewhere, the linked article probably understates the requirement for central control.)
Vendors (hardware suppliers) demand internet connectivity if you want timely service. They want their service staff to be able to sit in offices around the world (particularly in regions with low labour costs) to be able to troubleshoot problems remotely.
There are convenience factors. Governments mandate that stores have freezer temperature logs that can be consulted, and it is nice to collect the data in one place to meet those requirements.
This is a key pair of paragraphs from the article.
Before we can talk about the current state of OT, we need to go back 15 to 20 years to discuss IT/OT convergence. Before then, industrial devices and network infrastructure were often highly bespoke, heterogeneous technologies engineered by vendors. Most communication was serial-based, protocols were custom, and even device firmware was developed in-house. A Rockwell system looked nothing like a Siemens system. When you purchased a system, it was designed, deployed, and supported by that specific OEM. They supplied the replacement parts. They trained the technicians and installers.
This got expensive, and it slowed down logistics, not just for parts but also for labour. So clever engineers and businesspeople looked at the IT landscape and had an idea. Using Windows computers as human-machine interfaces, and commercial Ethernet to encapsulate serial traffic, would be much cheaper to buy, replace, and train technicians to support.
This is why everyone converged on systems that were uniform and accessible to the public internet. But as soon as that is done, you are stuck in the forever war of patching security vulnerabilities. And since these are large, interconnected systems, you end up with lots of vulnerable points of attack. (For example, someone sets up a wifi network in the store that devices connect to, even though they are not supposed to, and so you have an “attack surface” that was never taken into account in system design.)
In summary, if you are an IT person involved in this mess in 2026, you are stuck with patching this vulnerability and hoping the next one will take awhile to develop.
A Fossil’s View
I left electrical engineering in 1998, after a spell as a post-doc in control systems engineering. However, I had done contract work and projects that would be extremely relevant to the task of building a central control system for freezers distributed around a region (control systems, implementing a software interface to a controller board on a PC, working on a system that polled data from sensors on a proprietary network, worked in a design tool for private networks).
Using technology that was available in 1992, it would have been straightforward to have a few personal computers running MS-DOS or the new-fangled Linux operating system acting as a central control system for freezers, communicating over a private network (hosted by the local friendly telecoms company) to other personal computers that interface directly to the freezer control systems. The whole system would be air gapped from the pre-World Wide Web (http protocol) internet (hackers already existed). If you wanted to get data off the network? Well, that is what sneakernet is for (you got somebody wearing sneakers to carry a disk with data from one computer to the other). This would have been a relatively small project, assuming that you had freezers that could be controlled by an external controller (which may or may not have existed in 1992, but that would be an easy capability for manufacturers to add).
If anyone suggested connecting the control system to the public internet, they would probably have been on the receiving end of potty language from controls engineers like myself. Why are you exposing systems that are a core part of your business to hackers around the world? A grocery store without freezers is a dead grocery, similarly a water utility that loses control of its water purification systems.
What Happened?
In 1992, it was easy to see why technical competence was collapsing around the developed world. It was chronicled on a daily basis in the Dilbert comic strip, which some people mistakenly assumed was just for entertainment. This was not the case. In the immortal words of the great Twentieth Century philosopher Homer Simpson: “It’s funny because it’s true.”
Scott Adams, the late creator of Dilbert, wigged out near the end of his life, which casts a shadow on his work. But his early work (at least) was widely cut out and hanging in engineer’s cubicles in the early 1990s because his anecdotes of technical incompetence were visible to anyone in engineering at the time. Although some of his early material was based on his personal observations from when he was in industry, he also ended up being fed anecdotes from throughout the engineering industry. As such, it is a record of management trends, even though the details are lightly fictionalised.
The key message of Dilbert was that the (possibly) competent engineering team was being continuously stymied by their pointy-haired boss, who knew literally nothing about the technical details of the work his engineering team did. At the top of the hierarchy, the C-suite was selected based on height and their quality of the hair. As such, decision-making was dysfunctional from an engineering standpoint, instead being driven by short-term financial incentives.
If we go the wider Scott Adams universe, he wrote some books on management. I no longer possess the volume, but he made an interesting argument. Firms were technically incompetent, but they survived by having a protective moat — their competitors were similarly technically incompetent. New firms only need a brief burst of competence to grow to sufficient size to then be able to survive off of inertia.
Software Engineers Are Not Engineers
We can then get to another point that modern readers might pick up when reading early Dilbert. Dilbert and his co-workers were always working on different random projects in episodes, but they often involved manufacturing some kind of technical widget. That is, they were electrical engineers. Yes, there would be programming projects — everyone in the early 1990s in electrical engineering was doing design work and bureaucratic work on computers. (In fact, I did a portion of my thesis writing remotely from home.)
This is very different than modern Silicon Valley, where “technology” is synonymous with “software engineering.” Many electrical engineers in my day would have snorted at “software engineering” as not being real engineering. (I had a long contract with software engineers before my doctorate started, so I was not dismissive, but I certainly did not see the work as “engineering.”) Engineering training is supposed to create people who are necessarily somewhat pessimistic, spending their time looking at trade-offs and risk factors. This is not the vibes one sees in the modern “technology” industry, were the top people are hucksters of various stripes.
Since I left electrical engineering in the late 1990s, I obviously cannot comment on more modern attitudes. My feeling is that I am dinosaur in this respect.
Freezer Incidents Entirely Predictable
As the linked article outlines, the path that led towards the software vulnerability was entirely predictable based on trends that have been in place since the 1980s (if not 1970s), even if the actual vulnerabilities appeared later. My only disagreement with the quoted text is the use of the word “clever,” although the people involved might have thought of themselves as such.
By pushing aside the electrical engineers and letting pointy-haired bosses and software engineers call the shots, the extremely obvious requirement of putting an air gap between mission-critical control systems and the public internet was dropped. The software engineers insisted that they have a “clever” way of ensuring security. The vendors are “clever” and put backdoors in their hardware so that they can do cheap remote maintenance. (Rumours are that some backdoors were mandated by national security apparatuses so that they could hack hardware deployed in foreign countries — e.g., Stuxnet.) Once that total containment is dropped, it is just a question of time for more bridges to appear.
End result is that almost all mission-critical hardware is protected by software with holes, and security experts continuously sticking fingers into new holes as they appear.
And since literally everybody is doing it, there is no easy fix for any individual firm. You are stuck with the expensive task of burning down your entire technical infrastructure, and rebuilding everything behind private networks — and trying to figure out how to hire enough competent people to build and maintain the proprietary systems. You then have to deal with vendors who do not want to give up remote access, and thus you cannot get reasonable service contracts.
Concluding Remarks
The path of least resistance is everyone keeping doing what they are doing, and hope that security incidents are not too expensive. Since I am semi-retired, this is not my problem.
Appendix: Power Systems
Power electronics was still in its infancy in my days, and the material sciences advances that allowed solar and wind power to be competitive did not exist. As a result, my limited contact with power systems is entirely useless for the modern task of connecting renewables to the grid.
For the grid itself, the key thing to understand that it lives and breathes at 60 hertz (in most of North America), at extremely high voltages. If you get nearby conductors in the grid that are far enough out of phase1, you might develop a voltage gradient of about 3 kV per mm. At this gradient, electrons flow freely between molecules in the air, turning air from a “insulator” to an extremely efficient conductor. You then get a pretty blue arc in the air, which then very efficiently discharges the voltage differential (and destroying any engineering hardware nearby).
This means that the high voltage backbone grid itself is actually fairly simple in principle: everything is oscillating in tune with the grid signal (or it dies). The system is somewhat stable: small deviations in voltage create a feedback loop that forces electrical machines (motors, generators) to synchronise with the grid. (An electrical generator is a motor run backwards, a small out-of-phase voltage temporarily turns the generator into a motor that pushes the rotor [the part of the machine that rotates, for example by running water] into phase with the grid.) The job of the grid operator is to ensure that the grid voltage stays in line with where it is supposed to be (voltage level, frequency, waveform is sinusoidal). It was possible to achieve this with mechanical means — we had electrical grids before the rise of the transistor, never mind semiconductors that can survive power voltages and currents.
The rise of renewables let utilities allow customers to sell power back to the grid. My old school reaction would be to say “over my dead body,” but I am not in touch with the modern power electronics wizardry that makes this possible. It is obviously technically possible to allow this, and it is economically efficient.
The problem with this new regime is that since you are dealing with households, you are pretty much forced to connect to the public internet to synchronise with their mini-renewable power source. As such, utilities have no choice but to have exposure to the public internet, and thus some vulnerabilities.
Alternating current is a sinusoidal wave. If two electrical signals are “in phase,” the two signals rise and fall at the same time, and so voltage differentials are at the minimum. But if they are out of phase, they rise and fall at different times. The worst case is being 180 degrees out of phase — the signals are opposite signs. So when one is maximum positive, the other maximum negative.
No comments:
Post a Comment
Note: Posts are manually moderated, with a varying delay. Some disappear.
The comment section here is largely dead. My Substack or Twitter are better places to have a conversation.
Given that this is largely a backup way to reach me, I am going to reject posts that annoy me. Please post lengthy essays elsewhere.