Peephole Optimization — Making Things Worse One Improvement at a Time

Been a while since I had much to say on this forum. My indignant sputters over the decline and fall of practically everyone has found ample outlet in the various news sources I read. And besides, why kick? No one is listening…

But today, Cold Air Online Facebook – Cold Air Onlineposted a link to Facebook for an article in another blog Blowing It On The Wind that brought a whole bunch of things together. My response was:

It actually does make perfect sense. What we have is an example of what in software development is called peephole optimization. One looks at a very small part of the overall program and makes changes to optimize it against some criteria — calculated speed of execution, storage use, etc. But because of the narrowness of the view, the overall destructiveness of the changes are not seen — in that the overall program runs much slower or perhaps doesn’t produce the intended results. Human systems are rife with these kinds of errors — in healthcare, to reduce costs, services are consolidated into larger and larger catchment areas – regional and then no doubt provincial hospitals. So costs go down, but overall results are worse because sick people must be transported further and by more expensive means. And more die in transit. So costs for running the hospital are lower but overall costs and public results are much worse. Horrah! This green energy stuff is very much the same — wonderful solutions to very narrowly defined criteria but overall almost everyone is net much worse off. Now what problem were we trying to solve? If it was poverty among the well connected, likely addresses. For everyone else maybe not so much.

This is the problem — we are surrounded by governments and businesses trying to optimize their processes to (hopefully) improve services and (even more hopefully) reduce costs. So they enlist the help of specialists to target items for improvement — but, as far as I have seen, show little interest in looking at the broader picture or for that matter even checking up afterwards to see if the results were achieved, and if so, at what cost?

Getting a liberal arts education, in the traditional sense, has very much fallen out of fashion. Instead, people are urged to study what at one time would have been considered very industry-specific skill sets (programs that were once funded by companies) instead of broader based programs. One no longer studies the philosophy of science but how xyz enterprises maximizes shareholder value by ignoring customer complaints. And so on. Looking at the big picture is very much a subversive view that must be stamped out.

So we have a case in Ontario where a previous conservative government, in the interest of achieving cost savings through economies of scale, embarked on a wholesale program of forced municipal consolidations. Where I live was one so affected — our little municipality was amalgamated with a shore-based area that has little time or money for our concerns. But because we are so different from them we have fewer services and higher costs. And our transportation issues all wind up in the lap of the Province, who are obsessed with the not inconsiderable issues of Toronto and have no attention for us — so the ferry service that we all depend on has been crippled this year because a needed upgrade for one of the other services was allowed to fall through the cracks. And the upgrade to end loading docks that would alleviate the slow strangulation of island farms has languished for over two decades and has no believable target for implementation. But we can no longer do it ourselves…

The example referenced at the start is good in that to optimize the favored power sources the suppliers of baseline power are being hurt. The net result is its all more expensive than it should be. But even this is an example — power generation is only a small fraction of overall greenhouse gas production. I tend to think of the long lines of cars and trucks slowly crawling across Toronto because road construction to address growing business volumes has been a political football. And the ‘affordable’ suburbs have no practical public transit that would allow people to reach their jobs. So TTC continues to mean ‘Take The Car’ as opposed to anything that addresses the problem. Oh, did I mention that they shut the system down on weekends to do maintenance, because the hours of system shutdown (it only runs 18 hours a day) are too inconvenient for the service organizations? But that is another rant…

I guess that at the core is a very human tendency to focus on what they want to see and ignore the messy details around it. This is magnified by an even more human trait to look at the short term and leave the long term to someone else. A recent news item — ‘petrolium company researchers in the 1970s noticed that their businesses were accelerating greenhouse gas accumulation that would lead to global warming — but their work was suppressed’. Why should anyone be surprised? While a philosopher might have suggested that the vast resources of the company could have gone to helping development of non-combustion driven transportation so the vast chemical productions from petroleum could continue, burning it all makes more sense in a short term, take the money and run environment.

Part of the scientific method is to formulate a hypothesis based on observation, then test that hypothesis to see if the predicted results are achieved. The political and business processes appear to be different in that there is no testing. Or if there is, the scope is very tightly defined so no inconvenient facts get swept into the results.

Here in rural Ontario, we are the dubious recipients of more and more wind plants, producing power that we cannot use, being harmed by soaring power prices we cannot afford. And when this stress makes us keel over, we have a long trip to the hospital, because service to the people who need it was not one of the optimization criteria used by the Province. I hope we live through it…


This morning I watched ‘Valentino’s Ghost: Framing the Arab Image’, a documentary on Al Jazeera. It covered the history of interactions between Europe and the Middle East — the invasions and conquests by Spain, France and England of different parts of the African North Coast and Middle East. The shameful history of the US and Iran. And of course Israel and Palestine. It is not pretty and it did make me wonder — what do we really know? How much of what is in the press is true and how much is just spin and illusion? And more to the point, if other countries were invading you, driving you from your homes, killing your families and taking your resources — what would you do? And how would you feel towards them?

There was a quote I heard a while back, from the first head of the CIA who remarked that if the average person in the street had any idea of what was going on then he had not done his job.

Not much to say, really. Just the thought that when situations are looked at from other perspectives sometimes it is difficult to believe our self-righteous rhetoric. And wonder how we can get beyond this state?

Public Services

Once upon a time, governments in the US and Canada embarked on programs to provide services to their citizens that were felt to be essential for all citizens but otherwise unlikely to be profitable for private enterprise. Rural electrification comes to mind. Municipal water supplies, mail, highway construction, even public health programs. As a former Canadian Prime Minister opined — ‘governments do what (only) governments can do’. I chose to interpret this as to mean that nation building sometimes means that government services may not be uniformly profitable when provided for every citizen but the effect of those services will benefit all.

That was then. Now it seems that the rule is profit must be derived from everyone. If servicing them is not profitable then they are simply cut off or made to pay steep costs unrelated to those of their fellow citizens in more fortunate areas. So we have the mail services being consolidated to group public dumping sites (regional mail boxes), rural electric power rates a multiple of what urban denizens pay, transportation being reduced to private car or nothing as rail and bus services are rolled back outside urban areas. And while governments have lots of money to spend bombing some poor slob on the other side of the planet, they have little to repair roads, build transit or operate hospitals in rural areas. The mantra is ‘economies of scale’ (which they do not understand), so it is seen as better to consolidate services into a mega-hospital in Toronto and shutdown the regional ERs. If the patient dies on the four hour drive then think of the money we saved…

I sometimes think that the MBA disease — how much money did we make this afternoon, and who cares what that did for long term prospects, has infected society from top to bottom. Long term planning is sacrificed for short term gains, however minor. Look at the last election — the ruling party in Ontario blew 1 billion dollars in cancellation fees by shutting down two regional power plant constructions to save a few parliamentary seats. That they are building a replacement plant within eyesight of this author next to an idled power plant is beyond belief — transmission losses pushing the output 200km suggests the original siting near point of need made sense. Besides, Ontario has a growing power surplus that is costing us all dearly and is projected to continue doubling costs every five or six years. Other than government employees, who would want to live in such a place?

So the roads and bridges are crumbling, ferry services where they have been too cheap to build bridges is struggling while strangling the economic development of the serviced areas. The towns that once had thriving local industries are dying all around us. Government policy it seems is that instead of 25 local dairy farms there are 2 or 3 because the one place they can sell their milk doesn’t like diverse suppliers — too much paperwork. And the variety of local cheeses is replaced by mega-corp bland products — made in multi-ton batches and predictable though flavorless.

A long time ago I ran into a comment that said, in effect, that all politics are local. The reason the leaders of large aggregates of people seem so unresponsive is that the world looks different from there. At that distance the issues of large pools of people merge and vanish. And besides, how can they keep track? Probably means that our ideas of government are wrong. Democracy works when the population is small enough that the leaders have a chance of knowing the electorate — like the Scandinavian countries. In Canada even the provincial aggregates are too large — the Premier, surrounded as she is with Toronto, just cannot imagine anything other that the big city. Rural areas get the hindmost, so to speak.

And looking at the costs of fixing what the big cities have now — the numbers are almost science fiction. No place in the budget for that. And besides, if a few potholes did go away, how many votes would that garner? The reality is that the infrastructure, like our society, is just falling apart from neglect. I am now wondering what it will take for people to start just walking away from it?

Schumacher was right. Small is beautiful — it is affordable, maintainable and likely governable. But as long as our leaders are infected with the disease that clamors bigger, bigger, bigger, ever more, our ability to live in a functional environment becomes ever more distant.

System scalability — a classic fail (again)

A very long time ago a firm I worked at decided to deploy a new, distributed application to the traders. And in keeping with all the latest ideas the application code was loaded from a common server connected to the trade floor. The idea was simple — code changes would be done to the one copy on the server so anytime anyone signed in they would get the latest and of course greatest…

There was, of course, a small catch in this idea. While loading an application like this from the server is fairly quick, 100 more or less simultaneous logons are a different matter. It took hours and there was serious talk of having some poor clerk come in at 4am to sign on all the workstations so by the time everyone got in their machine was ready to go. It was, I confess, fun to watch the very smart development team confront the fact (with some help) that their clever idea was actually pretty costly.

From talking to my brother-in-law over the weekend it sounds like the medical records system he is required to use for charting was designed by some of the same folk. Logon is 15-20 minutes (this is a doctor sitting there waiting for the system to respond, remember). Record changes takes minutes. The day starts early and ends late due to the ponderousness of the whole application and the multiple layers of signon security built in. Didn’t ask if a chart could be shared — I am almost afraid of the answer.

Problem is that computers are really poor at sharing anything but very good at faking it. Easy to forget, especially in these days of applications with little bits spread all over the landscape. And on local networks, no matter how fast, transfers happen one bit at a time.

The old and much maligned mainframe systems did one thing right — their design sacrificed pretty much everything that slowed down processing and allowed applications to shovel data through at great rates. But these systems required a real understanding of what was going on to design and operate. Not quite as light and fluffy as todays GUI (gooey?) development tools that emphasize simplicity and cute effects for down to the bone functionality.

So we save money on building the code, pat ourselves on the back for what a contemporary, glossy system we have built. And force the highly skilled and not inexpensive people who have to use it to fiddle while waiting for someone’s cleverness to work. Guess this is another case of cost cutting at the wrong end — somehow it seems to me that lifecycle costs for all those high priced users would have been a better optimization target than development. But then they did have a few massive fails getting it off the ground to begin with. Glad the sales folk and the well-connected consulting house management made their commissions. Looks like the rest of us will be paying for this for years.

Forward Into the Past

Thought crossed my mind today, somewhere between the news flier touting propane appliances as the latest ‘green’ technology and the ‘science’ article questioning why people persist in the controversial strategy of even considering nuclear power — that perhaps we should simply declare that the 19th century was really the best of all times and that we are rolling everything back to then. Finish the job of dismantling the grid, stop commercial airplanes and cars, continue to do nothing to hold back the return of highways to the dirt, no more vaccinations — disease is good and culls the population, and so on. Just think… Steam Punk as prescience.

After all, the 19th century robber barons appear to be the model of the ideal contemporary citizen. Social services? A wasteful excess — besides, no one worth taking care of does not have the means to hire their own doctors. Who cares about the others. Not worth mentioning anyhow. Climate change? A myth — besides, there are those ski resorts in Antarctica to consider now that Whistler is getting rain. And if we cut wages enough then maybe our suits and gowns can be tailored here rather than in Hong Kong. And it will be easier to get domestics.

And look at how much money can be made by keeping the world in a perpetual state of regional wars. And provides a sink for the surplus lower classes. The security services will keep the rabble away from the estates anyhow.

And I am sure there will always be some aviation — got to have some way of getting the produce in from South America now that California is going back to desert. Too bad we were so successful in killing off local agriculture. But we can sell all that abandoned land around the wind plants to folks from away and they will bring in their own coolies to till it. As for the huge pile of people in the cities? Heck, they will find ways to cope. After all, ‘Soylent Green’ provided a model.

The kernel of truth in all this is that the strength of the 19th century, as practised in the industrial US anyhow, was that it was as locally self-sufficient as possible. Bringing stuff from away was difficult and expensive so people made due with what was available. Houses used local materials and were designed to suit local conditions, not fashion statements from somewhere else. The local machine shop and pottery made what you needed as did the local woodwright. And most food was local too. Self-reliance, a traditional virtue I think we have largely forgotten.

Nice fantasy. But tough to do when there are sooooo many people and even more to come. Problem is that coping with the world we are making will just not work by rolling back the clock. 21st century problems require 21st century solutions. After all, at one time gasoline was considered too dangerous to use, but we learned. Supporting huge piles of people simply requires huge amounts of power — and there are very few choices available to produce it. We need to stop whining and do some engineering — or the road back to the 19th century will not be pretty.

Shift Overload

This afternoon yet another exhausted train crew were unwilling participants in a derailment and subsequent fire. The poor people in that part of Saskatchewan are concerned., I have read no news of the crew or whether this was yet another track fault or someone snoozed through a signal.

A recent article in the news indicated that train crews, due to the random scheduling of their work, are often exhausted and can fall asleep at the controls. A study had been under way by Transport Canada but was killed — the rail union carried it on to conclusion. Some decades ago train crews worked a scheduled shift but under the new and improved management they are scheduled without warning on an almost random, rotating basis. SO few have the luxury of a decent nights sleep.

Almost three decades ago, when I took over management of a 24×7 datacenter operation, I made a point of reading everything I could find about the issues of staffing this type of operation. What I found was a population of research that indicated it took almost two weeks before a person stabilized on a particular time schedule. So I tried to find people who liked to work nights to staff my datacenter and did not inflict the evil of rotating schedules on my staff.

But I found that I am the lost soul crying in the wilderness. While commercial aviation flight crews have proscribed hours — the occasional air tragedy shows that this is widely abused. Train crews, hospital staff and many others find themselves in this fix. Before she retired, my wife was working part time at the local hospital — her schedule was two days on days, three days off, two days on nights (12 hour shifts), rinse, repeat. Plus being available to come in for any shifts that may come up short — if she was really exhausted I would bring her to work and collect the body afterwards. No one seemed interested as to whether this exhausted person could actually do the work. The depressing part is that there was nothing unusual about how abusive this schedule was — and had there been a problem I am sure that the ‘system’ would have burned her personally rather than looking at how people were scheduled. And from looking at how my son–in-law and daughter-in-law are scheduled, this seems totally mainstream.

The problem I have with this is that all these people are in positions of public safety. If they made a mistake then people might die. But that risk seems ok — by adopting a staff scheduling model that spreads the people very thin and makes wild assumptions about their ability to cope while exhausted. In a sense they have individually accepted that this is how they are expected to work — but if something goes wrong they personally will be liable. We have seen this model many times before. It is the private profit, public risk model. By refusing to staff and schedule at levels that would maximize worker capabilities, these groups are increasing their profits at the expense of public harm should one of these over-tired employees fail at a critical moment. Lac Megantic was just an example — the potential is everywhere.

Is this really a world that we want for ourselves or our children?

Making change from climate change — or not.

Of late there has been a flurry of opinions thrown about regarding different capitalistic models to profit from climate change. Ostensibly, if we pick the right model, we can both make money and save the planet. Right…

Similarly, in Ontario, we have a government announcing that they were intent on having 20,000 megawatts of the provincial power supply provided by ‘renewables’ — which is interesting considering that currently the published generation capability is 27,000 megawatts — 12,000 is nuclear and 8,000 is hydroelectric. Today the Province was using 17,000 megawatts and there was very little wind. But rural areas are going to be carpet-bombed with these huge pinwheels over their strenuous objections. And so far the bulk of the power produced has to be dumped at fire sale prices. And to protect the grid, the power authority has been granted the ability to pay the wind folks for not generating power based on what they could have produced if we could have used it. No, I am not making this up…

Guess the problem of saving the planet has gotten tangled up with the need for the ‘right’ people to make money off it. Reducing emissions has become entangled with pricing ‘carbon’ and produced trade-able certificates that can be bought and sold for a profit. Similarly, Enron introduced a model for electricity markets where one speculated on future power prices to protect ones’ costs — interesting that Ontario is still very quietly trying to pursue that model. May have something to do with having Enron as advisers on how to make the Ontario power system ‘modern’…

The climate has been changing since the Earth first coalesced from the dust of the solar system. Change is perhaps the only constant. Every week one reads about some new relationship being discovered that influences climate. It ranges from shifts in the orbit and planetary orientation to the sun, through variations in solar cycles, to emission of gasses that trap infrared and conspire to produce a greenhouse effect — Venus is an extreme case. These gasses include carbon dioxide — the result of human respiration and combustion processes and methane — the result of animal flatulence, permafrost decomposition and clathrate decomposition. The latter is due to methane seeps on the ocean floor that crystallize as vast field of solids under deep sea pressures and temperatures. Problem is that as the seas warm this stuff is turning to gas and joining the party. Human emissions from fires, transportation and industry are part of the problem — but only part.

Now I am reasonably sure that industrial civilization, striving to burn anything they could get their hands on, have been big contributors to this mess. But since we are not the largest contributor it is only hubris that would lead us to suggest that any one series of actions would ‘save the planet’. We fuss about not having made accurate predictions about ISIS and yet they used the internet to spread their propaganda and had lots of folks watching them. How accurate do we really think our ideas of how the climate is changing and what, specifically, we can do to influence this — given that the atmosphere is a complex product of the actions of a very complicated global system [of which I think we have at best a few guesses but no real grasp] and the action of a large pile of people who do things for their own reasons.

I am inclined to think that the best thing we can do is worry about how to help all those people who are being affected by climate change. And develop strategies for how to adapt to a warmer and drier/wetter world (depending upon where you live). And leave the ‘who’s fault is it’ and who will pay discussions for the lawyers in a later and hopefully smarter time. Burning less is always a good idea. Those petrochemicals are likely far more valuable as feedstocks for chemical synthesis. And ethanol… give me a break. Putting ethanol in gasoline was an idea from the 1930s to improve farm income in the Depression. That we do it now to save the planet is ludicrous — infernal combustion engines run more poorly on it than without. If the goal was to reduce GHG than this really is not a solution.

My concern is that with climate change we are on the brink of the largest forced migrations in human history. And putting up the concertina wire to discourage immigration simply magnifies the eventual problems. So we will chase the ghost of emission credits, alternate technologies and so forth. But one suspects that the climate will continue to change regardless.

Meanwhile, the real elephant in the room is there are just too many people. We are the ultimate invasive species. Look at where we have been — we cut down all the trees, drink all the water and dig up everything that might be profitable. And when we have wrecked that place we move on. I have seen sober analysis that suggests if there were 1 billion people the Earth could absorb whatever we do. But 7 billion or 9 billion or more? We are rapidly over-running the carrying capacity of spaceship Earth. The real climate change problem is that this invasive species (us) is consuming the planet — we need to control our numbers or go elsewhere or both. Personally, I would vote for going elsewhere — just basic monkey curiosity if nothing else. I want to see our species go to the stars — nothing else will be enough.