The Hassle Factor or Delusions in Technical Adequacy

As I get older, one of the things that strikes me is the prevalence of poor design in technology and (unfortunately) human affairs. It may just be a fundamental shortcoming of ourselves as a species — this overweaning sense of self-importance that gets attached to everything.

Take product design, for example. For a number of years I have been a robot vacuum cleaner owner. It may just be that I am lazy and more than a bit of a technophile. But having that little plastic trilobyte scuttling around and picking up the pet hair and crud seemed to be pretty cool. But this love affair has cooled with the latest model, a 500 series Roomba. This model is much more effective at collecting pet hair and dirt than its predicessor, so much so that it requries cleaning before and after every use to remove the stuff that winds around its closely spaced moving parts. It has a scheduler function but why bother with so much ongoing maintenance requried to keep it from stopping, yes, stopping, under a bed when it feels the level of crud in its innards is too high. Seems the designers in their hubris forgot why people use cleaning machinery — it is to make their lives easier. This level of fiddling is not required for either our central vac or my wife’s beloved floor cleaner. In the end, I have almost stopped using Roomba because the ongoing hassle factor of keeping it going simply overwhelms the pleasure of its use.

Computer software is another place where the self-importance of the designers overwhelms product functionality. I have piloted a number of system management applications in our little home network (as a lifelong computer geek it is probably a bit over the top for the needs of a semi-retired couple, but what the heck…). A common factor of all of them is that they demand continuous attention for their own issues to a degree that the operational simplicity that should have been delivered is overwhelmed. Why, for example, should I care if a resource sampling job that runs every 5 minutes did not respond in time once or twice during the night — how much should I care to dismiss the critical alerts that this spawned or the lengthy recovery processes that I was forced into? I just wanted to see a summary of the error logs across the systems, not be drowned in the neurotic details of the collection process. There are too many examples of this to count. It really makes me wish for the simpler days of booting off paper tape (yes, I have been screwing with computers for that long).

Similarly in the arena of human affairs, the province where I live is entertaining a new law that will allow wind farm developers to push local interests out of the way so they can dispoil endless tracts of farmland and nature reserves with 500 foot monoliths. Its called ‘green energy’ but the only thing that is green about it is the money this will transfer from the public purse into the hands of a few. Is it worth mentioning that the party in power has a number of close connections with the wind turbine industry? What I find fascinating about this is that by tagging it ‘green’ the rules of common sense are suspended. In order to connect all of this wildly variable power to the grid a big buildout of gas turbines will be requried to keep the inevitable voltage swings from taking the provincial power grid down. So in the end the greenhouse gases the wind towers will supposedly save us from will be more than offset by the waste from burning natural gas. And when the developers have gotten their money back, the municipalities will be left with a landscape covered with these huge rusting towers — guess the next round of glaciers will clear that mess.

The common factor I see in all of this is that the perpetrators have lost sight of ‘what problem are we trying to solve’ and become self-absorbed into their own solutions. And don’t dare ask about life cycle costing or whether the solution (including all the ongoing fiddling) is better or worse than the problem it is intended to solve. What would I rather have, cheap electricity from a nuclear plant down the road or the (birdless) sky filled with rumbling windmills? After all, nuclear waste will last for thousands of years… (I suppose that extinction is better?)

Just calling the later green does not make it so, pity I get to pay for it anyhow. I just wish that the folks dreaming up these solutions could lift their heads up every so often and check whether the solution they were creating was better or worse than the problem they were (supposedly) trying to solve. Instead, they seem to have fallen in love with their solutions (and themselves) and become THE PROBLEM.

Neurotic Products

Over the last few years I have attempted to incorporate a number of system administration applications into my environment (from a certain major vendor). But in every case, after a few weeks or months of frustration, I pulled it out. What I found was that in my small environment the constant demand for attention to remedy this or that perceived problem simply consumed me — so instead of being a benefit I was net worse off. And the wreckage left behind after product removal was almost too depressing for words. Oh, sure, the agents would uninstall from the console. But the registry changes left behind would block future reinstalls and interfere with other, related products that I played with later. Ouch.

To be sure what these tools purport to be doing is non-trivial and require a deep knowledge of this vendors own components. But when a problem was encountered the universal response was to flag an error, quit and wait for the human to realize that the problem was a transient one and manually restart. So constant attention was required to keep humoring the products along. And of course, many of the complaints were undocumented, and the vendor support sites only casually reviewed, so one was left trolling around in the dark — and often the only solution was to just restart.

What I found so interesting was that doing these things manually, like backups, one simply did not encounter these constant problems. So concluded that the real fault was in the vendor software and not the environment — but the approach was always to blame the user and make it as obscure as possible to sort out the real issues.

In this day of ’smart’ products one of the most depressing things was the endless repeat of software inventories. Every day, if I wanted to get a summary of the state of my environment I would get a very long email listing the software configuration of every device — everything, not changes, everything. Problem is that any real changes were buried in the endless detail — so I would have to save them all and compare side by side. No way to tell the software to do it, guess that was too hard. And to be sure, the overhead of doing this inventory hit the user device when they were logging on and trying to start work. So once again we were all net worse off.

Closely related was the issue of continuous availability for user devices. If the user device was not left on continuously and connected to the network, things failed and once again it was the messy manual restarts. Has not anyone heard of the energy crisis? Any smart business turned their gear off when it is not in use to reduce costs — not tripling them as an added cost of hosting this neurotic software system administrator.

I have started to call these products neurotic because they constantly clamor for attention like an insecure child instead of just doing their jobs. Probably reflects the personalities of the over-enthusiastic but inexperienced software engineers who write them. A pity, because the one thing any small business needs is more help so the people can concentrate on running the business. But with these products they get more demands for attention so in the end the business has to choose — feed the neurosis or stay in business.

Citizens or Consumers — The Crunch Comes

There have been some interesting comparisons of the present day to the years around the Great Depression. There certainly have been some big slides in the market and huge fortunes (and a lot of retirement funds) have been severely hit. And there is a huge wave of layoffs triggered by the collapse of international trade. And of course, like in the 1930’s there is a lot of discussion about protectionism to keep them ‘damn foreigners’ from taking our jobs, etc. But one thing that troubles me is what has changed since then and will these changes affect out ability to recover this time?

Over the last few years governments in the US and Canada have made it increasingly difficult to get unemployment and other forms of social assistance — force people back to work was one of the mantras. And the contingency funds, like UI in Canada, were raided for funds (or quietly merged into ‘general revenue’). And private companies were allowed to dig into pension fund surplusses — was just too much money sitting there. It is a rainy day and the rainy day funds have been spent.

But more important,what has happened to the civility and willingness to endure hardships for the benefit of the country? Much has been said about how psychological marketing has successfully changed citizens into consumers — for products and policies. And how the view that people were ‘entitled’ to huge homes, fancy vacations and other extravagances. But the rising home prices that they were borrowing against and the exotic financial instruments that were providing the funds have all come crashing down. But having convinced people that they were entitled to things and having broken their cohesiveness through divisive politics, how are these same business and government forces going to get people to believe that things can get better, so they can get better.

It is my view that the real problem is a loss of faith, not the consequential economic downturn. So the solution is probably more of the economic version of a revival meeting than specific patterns of government spending. But having worked so hard to destroy social cohesiveness, how can our ‘leaders’ get people to come to the tent and get their faith restored? And what is worse, what will prevent these displaced people from following their programming to get all they can for themselves from spilling into social unrest and real societal turmoil?

Microsoft Product Holes

Sometimes I can get very confused when finding ways to utilize a vendors’ product set. Of late this has been particularly true of Microsoft. My business environment is a LAN with a few workstations and laptops serviced by a Microsoft Small Business Server 2003 R2. It is a great product for a small business — provides pretty much all the services one could want at a reasonable price and amazingly trouble-free. The version we have supports exchange email, sqlserver and sharepoint. Now, I used to be an architect of database internals so I am no stranger to device level code and (gasp) assembler. But it is nice to not have to be bothered.

We had been doing backups to a network-attached 3rd party NAS. Had scheduled job on the SBS server and ran it manually on everything else. And duped the  backup savesets to a huge USB disk for portability. But I was wondering if I could do something better… then I discovered Microsoft Data Protection Manager. This is application aware so I could get much better granularity for my server-based applications and possibly old version  recovery for desktop files. But there is a catch.

The product is priced within the budget of a small business. But it won’t do laptops (unless they are on the network pretty much all the time) and it is dubious as to how well it does workstations. But the real kicker is that it wants to allocate 3x the protected storage for an uncompressed replica of the source and changes. So far, so good. Did I mention that one of my workstations (used for photographic images) has 900gb of disk? Well, DPM does not support USB or Firewire-attached disks (although Homeserver does) so one needs all that storage local or iSCSI. Has anyone priced iSCSI anything? Despite what the marketers may say, this stuff is definately not priced for small businesses. And DPM has kinky ideas about a second volume on the boot disk, so effectively one is limited to what can be accomodated on one drive. So if one has a small business LAN and relatively new machines (with huge disk drives) pretty much the only choice is what we did originally — use backup to write to a NAS disk. Pity, it seemed like such a good idea for a small environment, but the storage constraints (which it doesn’t see if virtualized, by the way) mean its not a general purpose backup solution. Which is odd considering how necessary what it does is in a small business environment and how nicely it does it (when you have the room).

The Problem of Nuclear Waste

I have followed the debate on storage of high level nuclear waste for some time. The discussions have been interminable — worse that the recently concluded US presidential campaign. The questions of safety for unimaginable terms were fascinating — especially how one would guarantee the isolation of the chosen site for many thousands of years. I think there are a number of problems with both the approach and indeed the whole concept:

1. The thesis that nuclear waste is a problem — the discarded components of nuclear fission will be releasing energy for a very long time. Somehow it seems that there is an opportunity here more than a problem. But it remains for some bright engineer to realize that this stuff could be a sustained power source and develop the technology to safely utilize it. I would not be the least surprised to find that in the future (probably long after I am with my ancestors) that our waste piles are being mined for resources.

2. That it is sensible to try and design structures that would outlast the entire history of human civilization. No civilization so far has lasted more than a few centuries before being replaced with a different way to organize society. Even China and Egypt changed over time. Nice idea but hugely expensive and no real way to prove one way or another — but I am sure that it was a nice cash stream for engineers and lawyers.

And of course, while the search for the ultimate answer was going on, the waste was simply accumulated at the generating nuclear plants — so there is a huge backlog to deal with.

My simple suggestion would be for folks to look at this stuff as an opportunity to be used rather than just a problem — it may free creative minds to a better solution. And to stop wasting money on predicting the future so far out that mankind may no longer exist.

Process Optimization and the Current Financial and Economic Crisis

One of the main mantras I have heard over my working life is the phrase — ’squeeze out the fat’. Process optimization has been a major contribution to profit margins in many businesses — the same businesses that are suffering in this current collapse.

Not only do we find companies that have moved from maintaining a cash balance to cover current payables such as procurement and payroll into more innovative approaches to fund these routine activities from a credit line or other borrowing. But places such as where my wife works that have systematically cut back on staffing and de-skilled shift roles so that even though education requirements are being raised for nurses — university is not enough, one must have advanced degrees, more jobs are being moved to less educated staff (registered practical nurses). In the past, staff levels were set based on average patient loads — so if the load dropped there was more time for training, and if it increased then folks were just really busy. Now the staff load is varied by demand — scheduled staffing is kept to a minimum and when workloads change everyone scambles and pulls in extra people, so personal lives take a toll. In the same way, many places changed their inventory practices to minimum stock and use of ‘just in time’ approaches to push the cost of supply assurance to someone else (often someplace else).

But when conditions change — increased border security slowing goods transit, sickness reducing the available staff, global credit collapse cutting off funding, the situation changes rapidly from ‘good management and cost control’ to chaos in very short order — with a liberal helping of gloom and doom. There are no reserves to draw on.

It makes me wonder if folks have forgotten what fat is all about — it is personal insurance against uncertainty, a buffer against the unknown. This is different from process inefficiency and waste, but do we discrimanate? Can we tell the difference? Seems we have lost the sense of uncertainty about life and decided to work without a net. And when life demonstrates that we are not in control, we prefer to be abused rather than dig into the storehouse.

I would hope that one of the results of the current chaos is a reinstated respect for buffers against uncertainty and why we are all provided with fat. We have survived because we are endowed with protection mechanisms against uncertainty, seems that it is time to put some of this back.

Virtualization — Problem Solver and Problem

I read with interest the article in the latest canadian computer dealer news about executive dissatisfaction with virtualization. While many companies are doing it — no doubt because the trade press screams that this is the wave of the future, etc, business executives are left wondering where the business case is for this technology. I share their concerns but will also share my experience with the technology and how I use it in the Technology Strategists computing environment.

The obvious use is as a test environment. Virtual machines for test or demonstration purposes can be setup much faster than physical hardware can be procured. There is a global limitation of virtualization in that the physical hardware must contain the virtual environment to get acceptable performance — so a 4gb memory size virtual machine will just not load in a 2gb physical machine. Copies of virtual machines can be made to test alternate configurations or capabilities — but be careful that the software licensing agreements are not violated.

Disaster recovery is another easy use — because virtual machines are files, they can be more easily replicated to offsite storage or alternate data centers. The gotcha (of course) is that the file needs to be closed to get a clean copy. Data is an issue that requires special handling — using database replication between sites to keep multiple locations in synch is the preferred approach. But be careful, standby replication schemes work but can have problems. An easier approach if the applications are not always in use is to close the VM and copy it to the second location — so the disaster copy is always one restart behind.

Using virtualization to run production loads is a tad trickier — we have done it here but find that there are issues. There are some management applications that are best kept offline to reduce their disruptive impact, starting them periodicly for a few days to check the state of the environment — then leave down so work can be accomplished. [I will not name names...] Issues we have run into fall into three groups — physical limitations of the execution environment constrain the virtual environment; software limitations of hardware emulations; and limited access to physical machine resources.

Physical limitations of the execution environment means that the machine one uses for virtualization must be larger than the sum of the virtual machines running on it. In practive, this means that older application environments could be resurrected by virtualization and consolidated, the host machine has to be bigger and faster. And hardware costs do not go up linearly as the capabilities of the box expand, unfortunately.

Software limitations of hardware emulations include problems like display constraints — have not been able to get a stable VM for applications that use DirectX-3D. And disk-intensive applications run slowly due to the double layer of IO emulation.

And no virtualization technology I have worked with appears to permit access to any more than a narrow range of physical devices — tape drives or other exotic peripherals are just not accessible.

And virtualization vendors in general are a myopic lot — not only do people want to be able to move execution environments in and out of virtualization, but they may want to move virtual machines to different virtualization environments. We dropped a promising virtualization technology when we found that there were no conversion tools available and the vendor got very vague when asked how to move existing virtual machines to their product. Its like the old days when relational databases were new — always an easy sell if the customer had nothing, but if they were already using product ‘x’, getting them to move to ‘y’ was pretty tough if there was no easy migration path.

Virtualization does solve business problems for us — it allows running of poorly written applications in isolated environments and makes it possible to have more operating systems available to use than there are physical machines. So we can get a lot more kinds of work done on limited hardware than might seem possible from an inventory of our hardware. And there is a lot that can be done with low cost software in this environment — but one does need the licenses to cover all those virtual machines.

I have seen it said that when vendors and their running dogs are insistent that a technology is inevitable, it really means just the latest application of the big lie (just like politics). Repeat something often enough and people will start to believe it is true rather than what you want them to believe. They may be right in some sense beyond their own need to ‘make the numbers’ but the business case needs to be sound for the expense to be justified.

Layering or Why Nothing Works

Having been involved with computers for 50 years now I have certainly seen some changes. As machines have become faster and memory cheaper the accessibility of increasingly powerful capabilities to average programmers has improved. But the results are not always much of an improvement — in my jaundiced view things are prettier but flakier than ever.

A couple of examples — in my retirement I dabble in stock trading as a way to earn scotch money… when there is a market stable enough to want to trade in, anyhow. I have bought a couple of analytical tools to help me, tools that are fully buzzword compliant. Problem is that funny things keep happening — printing goes all to hell and the formatting routines just ignore the settings, data paths change on their own, complex rules about what stocks to target or dump trigger on symbols with no data (see ‘path’ above). The standard vendor response is to uninstall all of the pieces and reinstall — especially the .net, .net2, .net3 libraries and so forth. The vendor has a nice tool and mostly it is reliable, but just mostly. My suspicions are that the just in time compilations these libraries depend on are less deterministic than they should be — as there is no real memory isolation the end results are probably influenced by factors outside the application. So ‘reproducing’ the problem is probably impossible.

Similar things happen with a systems management application I deployed on my server — it should make patch management easier, but no, I would spend my time chasing internal problems in this incredibly narcissistic application — over night it can log hundreds of errors as its pieces fail to work smoothly together. On the ‘right’ machine it is probably a nice tool but in a small shop there is just no time to sort out its problems. An uninstall is running as I write this.

I guess I am getting cranky in my old age — finding bloated, flaky applications and tools ever harder to tolerate. Had the latest vendor OS and office suite on one machine, pulled both as too many things were harder to do than before. Glad I am technically capable of reconfiguring stuff when it doesn’t behave. Pity it is getting to be a survival skill.

What I suspect is happening is that in too many circles the analytical technique of divide and conqueor has been applied as a general design tool. Methodologies are slavishly applied by implementors who now see that ‘we have always done it this way’. Layers of complexity are built up that should work but don’t because of linkages and dependencies that the simplifying assumptions just exclude [the terms of reference are drafted to exclude societal consequences]. Things are blithly used with no understanding at all of how they work — or fail. So applications dont work reliably, financial markets suddenly break down when factors outside the scope of the carefully designed instrument change.

The root of the problem is that we have embraced complexity as an alternative to the hard work of understanding in a general way what we are doing. So unintended consequences become the norm and things just collapse. Too many semi-skilled specialists and no where near enough generalists. And the big picture is getting fuzzier.

Greed and Market Chaos and Canadian Elections

While I am dismayed at the continual chaos in financial markets, it is interesting to note that in the US, at least, this chaos is traceable to the systematic dismantling of regulations designed to protect the investor. There has been a mantra chanted by the ’suits’ that unrestricted free markets are the universal solution to all problems. The politicians here in Canada still chant this as they systematicly sell off strategic assets — turning this once strong economy into a client state. All while telling us that we will be more competitive and better off. That was what the US voters were told while special interests pushed for the dismantling of regulations that ended up in the chaos we are all enduring.

Why is it that when politicians get a bad idea into their heads it is impossible to shake it, no matter how devastating the real world experiences are? Could it be that what they really want is to return to a feudal society where a wealth few can do what they want and the rest of us, no longer a comfortable middle class, but starving peasants labor to pay for their elaborate palaces and huddle in fear that we will be noticed? It is interesting that the consequence of all the deregulation in the US was a concentration of wealth — fewer people got more and the rest got less. It seems to happen everywhere the disease spreads — guess the politicians and their golfing buddies are expecting to stay with the ‘haves’.

Deregulation of utilities was an idea originally pushed by Enron — let the market take care of supply and everything will be rosy. Turned out they were manipulating the market to maximize their own profits. But that was not enough to cover the losses from their off-balance sheet deals that eventually brought them crashing down. In the US this idea spread and the results were always shortages and cost increases — everyone is net worse off. But re-regulating the utilities is just not mentioned.

Here in Ontario, after dismantling the provincial utility and restructuring it into a group of ‘independent’ companies a similar curve is being followed — costs are going up, money is poured into hairbrained schemes to make green electricity by devastating the rural landscape with huge wind turbines — despite the European experience that while wind farms might contribute, they are no substitute for coal or nuclear generation. And the experience of others just doesn’t apply…

My fear is that the only way this will ever be addressed is by a general economic collapse and social restructuring. Collectively, we have turned over the reins of power to the greedy and they are stealing for all they are worth. And we are all net worse off. Unfortunately, NONE of the ABOVE is not on the ballot in either the US or Canada.

Just buy the stuff and stop complaining

I wonder if vendors really care what the service life of equipment is any more (assuming they ever did)? I read in the computer press that the useful life of a computer is purported to be three years. I am sure that the hardware vendors would be delighted if we would just discard our stuff at that interval — especially if they had a side interest in selling the discards to some 3rd world country for salvage. But sometimes I think they deliberately make shortcuts to get equipment to fail early and hopefully get me to buy more.

Case in point — I have three LaCie storage units in my environment (or more to the point setting next to my environment). Two external disk drives — a 500gb and 1,000gb 1394/USB2 devices. And a 500gb NAS disk. All three are flaky — the two external drives are moody about mounting (the large one wont mount at all). The network attached storage disk seemed like such a good idea but kept logging disk errors (almost since day one) and the software update that was supposed to fix it just locked me out of seeing what the disk errors were — untill the entire web-based interface stopped working. Disk corruption, the vendor said, have to wipe the drive and start over again (3rd time).

My point is that I had not bought any of this gear as a throwaway. There are always a bunch of bulky downloads that one must keep around, and hundreds of downloaded documents and so forth. And quite frankly the only cost-effective way to do backups is disk to disk over the network.

But apparently, LaCie thinks that spending a few hundred or few thousand dollars is a throwaway and it is unreasonable to expect service lives of years. Too bad. Nice package and function at an attractive pricepoint — untill the short life is factored in. I am not buying any more of their storage devices. And what is more, recommending to anyone I talk to that their gear is not reliable.