Cyberattacks and the culture of cost-cutting

The recent major cyberattack on hospitals, even one here in Ontario, is continuing evidence that short-sighted practices of the past are coming home. And I would suggest that the root problems are more extensive than just software, but extend into every corner of the built environment.

Back in the 1970’s, there was an idea floating around the software developers I worked with called ‘Truman’s Triangle’. The concept was that there are tradeoffs between how quickly or cheaply something can get done and how good it was. Pick any two, they said — if you want it good then it will likely not be cheap or quick. Problem is that management always went for cheap and quick — issues could always be fixed in the next release (which frequently never happened).

As a developer, one common mistake in writing code is neglecting to check boundaries when writing into memory. So if one read 512 bytes from disk, there had better be a 512 byte buffer to accept it. Otherwise it overwrites whatever was above it in memory. A typical neophyte error and the root cause of buffer overflow, one of the common pathways for malware to gain a toehold. And if in a rush because the vendor had a committed release cycle — the 2001 version of windows (XP) there were probably a few such errors. Testing usually checks the expected behavior of a program — not what breaks otherwise. The software house where I worked had one guy who torture tested programs — if your stuff could get past Charlie Brown (his name, really) it was pretty bulletproof. Never saw this anywhere else.

In the real world this issue is compounded because the vendor (Microsoft) wants to sustain sales by forcing users to by the latest version when it is released. Not only are there improvements that may be desirable (and hopefully a few bug fixes) but changes to make the operating environment incompatible with the prior release. Hardware vendors facilitate this by only writing their drivers for the latest versions. And application vendors will put out a new version for the latest OS so that they get an upgrade sale as well. This is not always bad — sometimes the new version is better. And support has to be paid for somehow.

As bugs, like the buffer overflow, get discovered and complained about the vendor pushes out patches to fix things. These have to be installed and tested — sometimes the patch breaks things elsewhere, so some care and thought is required. A ‘fools rush in’ area to be sure. Why systems folk appreciate the value of patching but are sometimes reluctant to do it. And the thinner they are spread the greater the chance that this is one item than languishes.

So Microsoft adds pressure by dropping support, including providing patches, for those versions of the OS that they want off the landscape. For isolated machines like single desktops or computers embedded in machinery this is probably not an issue, IMHO. But for anything that is network connected, especially to the Internet, it is a serious threat, almost extortion. It is worth noting that after formally dropping support for XP, Microsoft has just released a patch for XP for the current, very conspicuous, bug.

But in the real world of enterprise the decisions around how to implement and maintain a computer application, a hospital medical records system for example, is made by people more interested in their balance sheet than the esoterica that they may be hacked. After all, that only happens to others — we are so much better, smarter, whatever… will never happen to us. So remote access is simplified with more a view to convenience than security, firewall rules not maintained, logs not checked, backups not monitored and tested to ensure they are really recoverable. And remote staff are used for support because its cheaper to get someone out of bed than have local staff — and they may need to get at everything remotely.

And finally we have a whole landscape of malefactors who want to steal, destroy or harass for a wide variety of reasons. And unlike conventional insurgency or warfare, this can be done from the comfort of a remote office or home with some chance that the source will remain hidden. And if greed is the driver, ransomware [locking a users system until ransom is paid] is seen as a whole lot better than just stealing the data and trying to sell it elsewhere.

So when one asks why are we not doing more to prevent these kinds of attacks, it is worth following the trail of breadcrumbs to realize that the issue exists because of deliberate decisions on many levels, most but not all, I believe, in ignorance, possibly willful, of the long term consequences of their choices. And until robustness and security are taken seriously at every point in the chain, the situation may change but only appear to get better.

But there is a larger issue seen in this problem. While there may be funding for the original creation, adequate funding for maintenance is a different problem. So corners get cut or ignored completely in keeping things running smoothly. And when funds get tight, maintenance and support is the first thing to go. And besides, every politician or business person likes the attention garnered for doing something new. But keeping things running smoothly and economically… not so much.

But it is not just software and computer systems where this is a problem. Look around… bridges are rusting, roads are crumbling, pipes leaking, food safety questionable. The list goes on and on — neglect, greed, short-sighted decision-making. Problems with building on flood plains that no one really knew about because the information was not collected or maintained. But because of that very human tendency to think that the potential consequences of taking the cheap/quick approach will not happen to them because they are…

The true miracle is that anything works at all.

Advertisements

Perversity of Renewable Power

For centuries, western civilization has worked to lessen the dependency on the weather for conducting our daily manufactured lives. That given, it does seem interesting that the current populist push for ‘renewables’ is a move to reverse that long avoided position. Although I might add that all forms of energy are actually renewable — just that the timescales for coal or uranium might be a tad long for the typical investor mindset.

Solar panels both photovoltaic and thermal require sunshine to harvest energy. A long spate of cloudy weather (this past winter comes to mind) significantly compromises their output. Not to worry though — here in Ontario there are immense solar panel arrays scattered about the rural landscape — after all, what do we need farmland for? Everything we need can be flown in from China, Chile or Mexico. Been noticing that last winter no one could be bothered to clear the snow from these things — suspect the Province was paying them for the power they might have generated, so why waste money on maintenance?

Wind farms are even more interesting — the huge monsters now in fashion need just enough — not too much and not too little. So areas that have any wind at all, away from the urban areas, are being carpeted with wind farms — mostly over the objections of the folks who live there. Freezing rain shuts them down. No way to defrost things — need to wait for the sun. Too much wind — you hope the automagic controls stop them. Watching video of overspeed turbines tearing themselves apart is pretty entertaining. But a broken blade can get lobbed a kilometer or more — and the Ontario setback is 550 meters. Don’t want one through my living room window… And we won’t mention that when the original research was done it was observed that little of the local winds blew during the same timeframes as the power was needed. Terawatt batteries? Dream on.

What we have seen over the last few years is that here in Eastern Ontario we are getting more days of cloud [I am an amateur astronomer] and wind levels overall are dropping in the Great Lakes area. And when it does blow it can be more extreme. And we will ignore for the moment the side effects of the technology itself — solar farms absorb heat differently than fields. And would not evaporate moisture in the same way as plants. Similarly, wind turbines work by taking energy from the low level air movement — and affect atmospheric mixing and increase low level turbulence. And as they turn the vibration shakes the ground.. big critters seem to ignore it but one wonders about worms?

So here we have it… re-introducing technology driven by weather as a means to ‘fight’ changes in the weather. One might think it was a big bet that conditions won’t change. And the apparent direction of the change we see moves away from the current conditions that support these things. Glad everybody is eager to make a quick buck from this stuff. But are we not being a bit over eager? Makes my head hurt…

Climate Change — the New Religion

I have been watching with some dismay the comments on Facebook and other places about the New York Times hiring Bret Stephens as a columnist. Ascribed to being an extreme climate change denier, parallels have been drawn with Holocaust deniers and suggestions made that the Times should be boycotted until they release him.

Interesting… he is accused of heresy for doubting the predictions of hard green religion and the demands for making specific changes in power generation and other things to save the planet. The science is fine but maybe we should not be so confident about our models and think about this a bit more. Guess this is what passes for non-belief in these hyper-partisan days. Infant damnation or atheism with no middle ground.

I will confess to similar leanings that have grown with the shrillness of the critics. I justify my thoughts with a comparison to political and economic forecasting — and the economic behavior of human society with its billions of interacting parts is quite simple compared with the climate.

Politicians of all stripes routinely tout specific programs to restore prosperity and end various flavors of unfairness. And their allies in business argue for policies to favor their industries with a heavy hand for similar reasons. Mostly these fail, often spectacularly. Trickle down, a favorite of the current White House occupant, has been proffered a number of times — cut taxes on the rich, the job creators and prosperity will flow down over everyone. So far not so much it seems — the latest was Kansas, where they had to dip into emergency funds to keep the state afloat. But the results are ignored… the real benefits achieved, if any, go to a more restricted group. So why do we believe them when the evidence is right in our faces?

So here in Ontario we have a Green Energy Act that removes planning control from local governments and substitutes the will of a highly politicized government power system. Rural areas are being covered with enormous wind farms against the will of most of the residents and in violation of various treaties and so forth. And harm… well, if the research was not done in Ontario it just doesn’t apply. And when locals fight it is against the government and the ‘renewables’ industry. And with the project across the street, the developer admitted that even though the costs are more than double any other project it is so profitable they have to do it. Curious… Ontario has a huge surplus of wind power being sold at a loss to surrounding areas, and soaring power costs — where is the money going?

Similarly, on a planetary scale, there is urgency in saving the planet by deploying more and more solar panels, wind farms and so forth. And making other changes to decarbonize the economy by industrial taxes and mandating expensive technology. But transportation, which accounts for almost 40% of greenhouse gasses in Ontario, continues to be the realm of cars, trucks and airplanes. Trains, which produce a tiny amount of GHG per passenger mile in comparison, continue to be under attack. And regional bus service has been shut down in a number of places. So if you don’t drive and cannot afford to fly you are going no place. And logging continues everywhere — although in North America there is replanting, unlike places where the forests are cut for agriculture, beef or palm oil.

The science is clear — hard to argue with years of rising temperatures, rising sea levels and melting ice. But some do… Where it gets trickier is in the efficacy of the models — and this is where things diverge. Problem as I see it is that compared with something simple like the planetary economy, the climate is a non-trivial system. We have human activity to be sure — coal, oil, gas and forests all cheerfully burned to power our civilization. And in some quarters nowhere near fast enough. But there are other factors — and more are discovered every week. We have the heat flow from the sun, the impact of large scale magnetic fields on all sorts of interactions, thermal properties of the earth and seas. And this is to say nothing of the solid gas hydrates on the ocean shelves, the gasses coming from permafrost organics that have been frozen for millennia. And the venting from millions of beef cattle who are very gassy on the diets we feed them to speed the trip to the table.

The problem with models is that at best we have simplifying assumptions about the factors we know about, the actual interactions may be a bit more complicated — and likely non-linear in ways we cannot even imagine. Then there are the factors we suspect, the known unknowns. And then there are the unknown unknowns… So while the models may be descriptive, I suspect they have a long way to go before becoming prescriptive. So airconditioning the arctic to refreeze the ice cap (and where does the rejected heat go one might ask?) or putting a giant parasol in space — if we could do it might have other effects than the one predicted by the proponents. But it is increasingly obvious that it is heretical to suggest otherwise.

Back to the columnist… the stuff I have read of his seems to be nothing more extreme than saying trust the science but the predictions not so much. A sense of modesty is called for about what we know and, probably more important, what we don’t. And what we can do to effect long term change. Not sure this is climate change denial in my book — but some seem to think so. And are suggesting that perhaps burning at the stake for heresy should be revisited.

There is one other factor that suggests caution. There are places right now that are being harmed by climate change — island chains vanishing under the sea, coastal erosion and flooding here in North America. And arctic communities under threat because the ice is melting and the permafrost is thawing — so their homes are vanishing. Curiously, we have no money to help any of these folks. But I guess if we don’t like them probably plenty of money to bomb them…

And changing the entire basis of our collective societies from burning stuff to something less destructive is not an overnight task, nor a free one. (Assuming there was the political will to do that, either.) Might be easier if we were not so eager to make more people and worsen the problem — but that is another rant. And if the climate modelers are right, even if we stopped everything right now it will take centuries before things change.

So I suspect that in reality the targets are where there is an easy buck to be made, like here in Ontario, and the sincere believers are being encouraged to think that these projects are the solutions to a planetary catastrophe and no one must stand in the way. Any one who disagrees is a heretic and must be burned. Of course that adds to greenhouse gasses but who cares, anyhow… not when there is money to be made. And in the end, the climate will do what it wants and we will adapt to it or perish.

Peephole Optimization — Making Things Worse One Improvement at a Time

Been a while since I had much to say on this forum. My indignant sputters over the decline and fall of practically everyone has found ample outlet in the various news sources I read. And besides, why kick? No one is listening…

But today, Cold Air Online Facebook – Cold Air Onlineposted a link to Facebook for an article in another blog Blowing It On The Wind that brought a whole bunch of things together. My response was:

It actually does make perfect sense. What we have is an example of what in software development is called peephole optimization. One looks at a very small part of the overall program and makes changes to optimize it against some criteria — calculated speed of execution, storage use, etc. But because of the narrowness of the view, the overall destructiveness of the changes are not seen — in that the overall program runs much slower or perhaps doesn’t produce the intended results. Human systems are rife with these kinds of errors — in healthcare, to reduce costs, services are consolidated into larger and larger catchment areas – regional and then no doubt provincial hospitals. So costs go down, but overall results are worse because sick people must be transported further and by more expensive means. And more die in transit. So costs for running the hospital are lower but overall costs and public results are much worse. Horrah! This green energy stuff is very much the same — wonderful solutions to very narrowly defined criteria but overall almost everyone is net much worse off. Now what problem were we trying to solve? If it was poverty among the well connected, likely addresses. For everyone else maybe not so much.

This is the problem — we are surrounded by governments and businesses trying to optimize their processes to (hopefully) improve services and (even more hopefully) reduce costs. So they enlist the help of specialists to target items for improvement — but, as far as I have seen, show little interest in looking at the broader picture or for that matter even checking up afterwards to see if the results were achieved, and if so, at what cost?

Getting a liberal arts education, in the traditional sense, has very much fallen out of fashion. Instead, people are urged to study what at one time would have been considered very industry-specific skill sets (programs that were once funded by companies) instead of broader based programs. One no longer studies the philosophy of science but how xyz enterprises maximizes shareholder value by ignoring customer complaints. And so on. Looking at the big picture is very much a subversive view that must be stamped out.

So we have a case in Ontario where a previous conservative government, in the interest of achieving cost savings through economies of scale, embarked on a wholesale program of forced municipal consolidations. Where I live was one so affected — our little municipality was amalgamated with a shore-based area that has little time or money for our concerns. But because we are so different from them we have fewer services and higher costs. And our transportation issues all wind up in the lap of the Province, who are obsessed with the not inconsiderable issues of Toronto and have no attention for us — so the ferry service that we all depend on has been crippled this year because a needed upgrade for one of the other services was allowed to fall through the cracks. And the upgrade to end loading docks that would alleviate the slow strangulation of island farms has languished for over two decades and has no believable target for implementation. But we can no longer do it ourselves…

The example referenced at the start is good in that to optimize the favored power sources the suppliers of baseline power are being hurt. The net result is its all more expensive than it should be. But even this is an example — power generation is only a small fraction of overall greenhouse gas production. I tend to think of the long lines of cars and trucks slowly crawling across Toronto because road construction to address growing business volumes has been a political football. And the ‘affordable’ suburbs have no practical public transit that would allow people to reach their jobs. So TTC continues to mean ‘Take The Car’ as opposed to anything that addresses the problem. Oh, did I mention that they shut the system down on weekends to do maintenance, because the hours of system shutdown (it only runs 18 hours a day) are too inconvenient for the service organizations? But that is another rant…

I guess that at the core is a very human tendency to focus on what they want to see and ignore the messy details around it. This is magnified by an even more human trait to look at the short term and leave the long term to someone else. A recent news item — ‘petrolium company researchers in the 1970s noticed that their businesses were accelerating greenhouse gas accumulation that would lead to global warming — but their work was suppressed’. Why should anyone be surprised? While a philosopher might have suggested that the vast resources of the company could have gone to helping development of non-combustion driven transportation so the vast chemical productions from petroleum could continue, burning it all makes more sense in a short term, take the money and run environment.

Part of the scientific method is to formulate a hypothesis based on observation, then test that hypothesis to see if the predicted results are achieved. The political and business processes appear to be different in that there is no testing. Or if there is, the scope is very tightly defined so no inconvenient facts get swept into the results.

Here in rural Ontario, we are the dubious recipients of more and more wind plants, producing power that we cannot use, being harmed by soaring power prices we cannot afford. And when this stress makes us keel over, we have a long trip to the hospital, because service to the people who need it was not one of the optimization criteria used by the Province. I hope we live through it…

Reality?

This morning I watched ‘Valentino’s Ghost: Framing the Arab Image’, a documentary on Al Jazeera. It covered the history of interactions between Europe and the Middle East — the invasions and conquests by Spain, France and England of different parts of the African North Coast and Middle East. The shameful history of the US and Iran. And of course Israel and Palestine. It is not pretty and it did make me wonder — what do we really know? How much of what is in the press is true and how much is just spin and illusion? And more to the point, if other countries were invading you, driving you from your homes, killing your families and taking your resources — what would you do? And how would you feel towards them?

There was a quote I heard a while back, from the first head of the CIA who remarked that if the average person in the street had any idea of what was going on then he had not done his job.

Not much to say, really. Just the thought that when situations are looked at from other perspectives sometimes it is difficult to believe our self-righteous rhetoric. And wonder how we can get beyond this state?

Public Services

Once upon a time, governments in the US and Canada embarked on programs to provide services to their citizens that were felt to be essential for all citizens but otherwise unlikely to be profitable for private enterprise. Rural electrification comes to mind. Municipal water supplies, mail, highway construction, even public health programs. As a former Canadian Prime Minister opined — ‘governments do what (only) governments can do’. I chose to interpret this as to mean that nation building sometimes means that government services may not be uniformly profitable when provided for every citizen but the effect of those services will benefit all.

That was then. Now it seems that the rule is profit must be derived from everyone. If servicing them is not profitable then they are simply cut off or made to pay steep costs unrelated to those of their fellow citizens in more fortunate areas. So we have the mail services being consolidated to group public dumping sites (regional mail boxes), rural electric power rates a multiple of what urban denizens pay, transportation being reduced to private car or nothing as rail and bus services are rolled back outside urban areas. And while governments have lots of money to spend bombing some poor slob on the other side of the planet, they have little to repair roads, build transit or operate hospitals in rural areas. The mantra is ‘economies of scale’ (which they do not understand), so it is seen as better to consolidate services into a mega-hospital in Toronto and shutdown the regional ERs. If the patient dies on the four hour drive then think of the money we saved…

I sometimes think that the MBA disease — how much money did we make this afternoon, and who cares what that did for long term prospects, has infected society from top to bottom. Long term planning is sacrificed for short term gains, however minor. Look at the last election — the ruling party in Ontario blew 1 billion dollars in cancellation fees by shutting down two regional power plant constructions to save a few parliamentary seats. That they are building a replacement plant within eyesight of this author next to an idled power plant is beyond belief — transmission losses pushing the output 200km suggests the original siting near point of need made sense. Besides, Ontario has a growing power surplus that is costing us all dearly and is projected to continue doubling costs every five or six years. Other than government employees, who would want to live in such a place?

So the roads and bridges are crumbling, ferry services where they have been too cheap to build bridges is struggling while strangling the economic development of the serviced areas. The towns that once had thriving local industries are dying all around us. Government policy it seems is that instead of 25 local dairy farms there are 2 or 3 because the one place they can sell their milk doesn’t like diverse suppliers — too much paperwork. And the variety of local cheeses is replaced by mega-corp bland products — made in multi-ton batches and predictable though flavorless.

A long time ago I ran into a comment that said, in effect, that all politics are local. The reason the leaders of large aggregates of people seem so unresponsive is that the world looks different from there. At that distance the issues of large pools of people merge and vanish. And besides, how can they keep track? Probably means that our ideas of government are wrong. Democracy works when the population is small enough that the leaders have a chance of knowing the electorate — like the Scandinavian countries. In Canada even the provincial aggregates are too large — the Premier, surrounded as she is with Toronto, just cannot imagine anything other that the big city. Rural areas get the hindmost, so to speak.

And looking at the costs of fixing what the big cities have now — the numbers are almost science fiction. No place in the budget for that. And besides, if a few potholes did go away, how many votes would that garner? The reality is that the infrastructure, like our society, is just falling apart from neglect. I am now wondering what it will take for people to start just walking away from it?

Schumacher was right. Small is beautiful — it is affordable, maintainable and likely governable. But as long as our leaders are infected with the disease that clamors bigger, bigger, bigger, ever more, our ability to live in a functional environment becomes ever more distant.

System scalability — a classic fail (again)

A very long time ago a firm I worked at decided to deploy a new, distributed application to the traders. And in keeping with all the latest ideas the application code was loaded from a common server connected to the trade floor. The idea was simple — code changes would be done to the one copy on the server so anytime anyone signed in they would get the latest and of course greatest…

There was, of course, a small catch in this idea. While loading an application like this from the server is fairly quick, 100 more or less simultaneous logons are a different matter. It took hours and there was serious talk of having some poor clerk come in at 4am to sign on all the workstations so by the time everyone got in their machine was ready to go. It was, I confess, fun to watch the very smart development team confront the fact (with some help) that their clever idea was actually pretty costly.

From talking to my brother-in-law over the weekend it sounds like the medical records system he is required to use for charting was designed by some of the same folk. Logon is 15-20 minutes (this is a doctor sitting there waiting for the system to respond, remember). Record changes takes minutes. The day starts early and ends late due to the ponderousness of the whole application and the multiple layers of signon security built in. Didn’t ask if a chart could be shared — I am almost afraid of the answer.

Problem is that computers are really poor at sharing anything but very good at faking it. Easy to forget, especially in these days of applications with little bits spread all over the landscape. And on local networks, no matter how fast, transfers happen one bit at a time.

The old and much maligned mainframe systems did one thing right — their design sacrificed pretty much everything that slowed down processing and allowed applications to shovel data through at great rates. But these systems required a real understanding of what was going on to design and operate. Not quite as light and fluffy as todays GUI (gooey?) development tools that emphasize simplicity and cute effects for down to the bone functionality.

So we save money on building the code, pat ourselves on the back for what a contemporary, glossy system we have built. And force the highly skilled and not inexpensive people who have to use it to fiddle while waiting for someone’s cleverness to work. Guess this is another case of cost cutting at the wrong end — somehow it seems to me that lifecycle costs for all those high priced users would have been a better optimization target than development. But then they did have a few massive fails getting it off the ground to begin with. Glad the sales folk and the well-connected consulting house management made their commissions. Looks like the rest of us will be paying for this for years.