Wednesday, March 16, 2011

Can Your Website Think?

One of the propositions of softwarephysics is that there are currently three forms of self-replicating information on Earth – genes, memes, and software – with software rapidly becoming the dominant form of self-replicating information on the planet. As outlined in Self-Replicating Information, all forms of self-replicating information have a commonality of shared properties that allow us to learn about one by examining the others. Over billions of years the genes have learned to create conscious beings through the Darwinian mechanisms of innovation and natural selection. The genes certainly did not plan on this happening, since genes are just mindless forms of self-replicating information, but it happened just the same in a purely mechanistic manner as huge neural networks developed in complex multicellular organisms such as ours. So the question becomes did the same thing happen for the memes and might it happen for software as well? Meme-complexes do seem to have a mind of their own as they respond to changes in the Zeitgeist of the times, but since memes are forms of self-replicating information that only exist in the minds of DNA survival machines to begin with, the analogy does not easily carry forward in an unambiguous manner for software, so let us focus our attention on the consciousness of the DNA survival machines that the genes have brought forth, and to see if it can be extended to software as well.

The problem of what constitutes a conscious sentient being has been troubling philosophers, theologians, and scientists from the very beginning, and has not been satisfactorily explained by any of them. Fortunately, the extreme difficulty of the subject matter seems to stimulate the field in many disciplines. Some philosophers and many researchers working on AI (Artificial Intelligence) adopt a positivistic approach to the subject in that they don’t care what consciousness “really is”; they only care about how conscious beings are observed to behave. For AI an important advance along these lines came about with Alan Turing’s paper Computing Machinery and Intelligence (1950) in which he proposed his famous test. In this paper Turing proposed having a judge ask questions of a conscious human being and a computer via two terminals. If the judge could not tell the difference between the human being and the computer from her interactions with both, then the computer would pass the Turing test for intelligence. A few months ago we saw IBM’s Watson supercomputer trounce his human opponents on Jeopardy using massively parallel processing software running on a cluster of 90 IBM Power 750 servers with a total of 2880 POWER7 processor cores and 16 Terabytes of memory. It could be argued that Watson failed the Turing test, not because he could not keep pace with his human opponents, but because nobody could possibly be that good at Jeopardy! Or perhaps Watson passed the Turing test, while his human opponents failed it. Anyway, it was quite fascinating to see such impressive levels of AI in action. In a similar manner, GPS units seem to have reached a level of AI far superior to human navigators. My GPS unit speaks to me and tells me exactly what I need to know without me even looking at the device, which is far superior to the directions you would obtain by stopping at a local gas station, even if you could persuade the reluctant attendant to come along for the ride. This AI behavior is especially handy when I decide to wander off the route planned by my GPS unit or I miss a turn. It patiently recomputes the best way to get me back on track to my destination. Once I was driving in downtown Chicago, and because of some local construction, a one-way street had been temporarily turned into a two-way street. As I proceeded to drive the “wrong” way down the one-way street, my GPS unit gently reminded me to please turn around as soon as possible, but when I did not, it finally just told me in disgust to make the proper turn when I reached my final destination. As I was driving the “wrong” way down the one-way street, I just had to wonder what my GPS unit was “thinking”. I was afraid that my GPS unit was thinking that this nut is going to get us both killed if he does not quickly turn around!

In Kinds of Minds (1996) Daniel Dennett extends some of the ideas found in his earlier work Darwin’s Dangerous Idea (1995) to explain how intelligent consciousness could have evolved from simple neural networks. He explains how sensation, the ability to respond to stimulation in a mindless sort of way, like your home thermostat, could evolve into sentience through the step-by-step Darwinian processes of innovation and natural selection. To explain this process, Dennett uses one of his favorite intellectual devices – the Intentional Stance.

Here is how it works: first you decide to treat the object whose behavior is to be predicted as a rational agent; then you figure out what beliefs that agent ought to have, given its place in the world and its purpose. Then you figure out what desires it ought to have, on the same considerations, and finally you predict that this rational agent will act to further its goals in the light of its beliefs. A little practical reasoning from the chosen set of beliefs and desires will in most instances yield a decision about what the agent ought to do; that is what you predict the agent will do. (Daniel Dennett, The Intentional Stance, p. 17)

At a very early age we all learned to adopt an Intentional Stance when dealing with our fellow human beings, like our parents and other care givers. We learned that human beings seemed to operate in a different manner than our blankets and teddy bears. Human beings behaved as if they could “think” and acted as rational agents with their own sets of beliefs and goals. For Dennett, adopting an Intentional Stance towards an external entity is one of the significant landmarks along the road from sensation to sentience. End-users of a modern-day websites certainly adopt an Intentional Stance towards websites. Not only does the website respond to requests by the end-user, it also seems to have its own beliefs, goals, and desires. Many times the website will try to up-sell or cross-sell the end-user with products or services, or entice the end-user with a special offer to go to paperless billing.

In Kinds of Minds, Dennett also explains that another landmark along the road to consciousness is the ability for an agent to spread its mind out into the world by leaving behind cues and reminders in the environment to help it achieve its goals. This can be found in the lowly ant that leaves behind a pheromone trail on the surface of the soil that can be later followed by the ant or other ants to a source of food, all the way up to human beings who have a multitude of Good Tricks, including language and the written word to leave behind cultural artifacts in the environment that can be later used by our minds or the minds of others. Dennett explains:

We Gregorian creatures are the beneficiaries of literally thousands of ... useful technologies, invented by others in the dim recesses of history or prehistory but transmitted via cultural highways, not via the genetic pathways of inheritance. We learn, thanks to this cultural heritage, how to spread our minds out in the world, where we can put our beautifully designed innate tracking and pattern-recognizing talents to optimal use.

Websites do the same thing with cookies. Cookies come in many flavors and are used to do things such as “maintain state” and “personalize” the website experience for end-users. Since the HTTP protocol does not maintain state between one page request and the next, a website must keep track of end-users between requests as if they were in a dedicated session, like somebody on a very long phone call, with a continuous give and take between the end-user and the website. In reality, the HTTP protocol essentially behaves like a series of disconnected phone calls, but through the magic of cookies, it is possible to maintain state and trick the end-user into believing that they are actually having a dedicated conversation with the website. For example, the end-user’s browser might first make a phone call to the website requesting the HTML for a particular webpage and then all of the necessary files like the .jpg and .gif files that form the images on the webpage and little JavaScript programs that are required for the HTML to render properly. The end-user’s browser then hangs up while the website server responds to all of these requests. For the next webpage that the end-user requires, the end-user’s browser once again makes another phone call to the website server for another HTML page and all of its associated files. As you can see, if the website is trying to help the end-user fill a shopping cart and place an order, there must be some mechanism to keep track of the thousands of concurrent end-users who are simultaneously making such requests. This is accomplished by dropping a session cookie on the end-user’s browser. The session cookie identifies the end-user for the duration of the total transaction, which might require many webpages to accomplish. Session cookies are short-lived cookies that usually disappear when you close your browser or timeout after a few minutes of inactivity on your part. Tracking or persistent cookies, on the other hand, last much longer and are used to identify end-users from session to session so that the website experience can be customized for the end-user. For example, you might use a weather website that remembers your home town and how you like your home page laid out when you login, and so the weather website automatically displays the forecast for your home town in the manner you like when you login to the website. This is accomplished by dropping a persistent cookie on your browser. So websites do not actually “remember” you, instead they distribute their “minds” out upon the end-user community by dropping cookies on individual browsers. Unfortunately, tracking cookies can also be used by spyware and other forms of malware to monitor your Internet activities. Your virus and malware protection software can be used to help identify such cookies, and is again an example of the need to adopt an Intentional Stance when dealing with software that really does have a set of beliefs, goals, and desires that are not in your best interest. So websites do indeed respond to stimulation, behave in a manner worthy of the Intentional Stance, and do spread their “minds” out into cyberspacetime.

Now if end-users can think of websites as “thinking” agents, it is even more true for the IT professionals who create and maintain them. As a member of the Middleware Operations group for my current employer, I constantly find myself adopting the Intentional Stance when dealing with the software that I support. For example, last week I got pulled into an install that was not going very well. Our MidOps Offshore group in Bangalore India was having problems installing a very large Websphere .ear file. For J2EE Appservers like Websphere, .ear files are used to install Applications into a particular Websphere JVM that then create dynamic HTML for a particular website. Because this was a foreign website which had already seen the Sun set for the day and most of its end-user community depart, its maintenance window just happened to include a Friday afternoon for me, and I had a ton of last-minute change tickets for the weekend to proofread and approve. Frantic programmers were calling my IP phone, leaving messages that I could not listen to, and popping instant messaging sessions on me while I was desperately trying to troubleshoot the install problem. The root cause of the problem was that the filesystem that Websphere was trying to deploy the .ear file to ran out of disk space, a very unusual problem since this critical filesystem for Websphere is normally sized with quite a bit of extra disk to avoid such problems. But this time it was not. This was further complicated by the fact that this was a low-usage website that had the Websphere Deployment Manager and a Websphere Cell with a single node all running on the same server, an unusual affair for most of the Websphere applications that MidOps supports. By the time I joined the call, there were apparently several residual orphan Unix processes from previous install attempts trying to write to the filesystem. At one time the Unix df command showed the filesystem at 100% disk, so I deleted out some old .ear files to make room, but unknown to me, other orphans must have been still trying to write to the filesystem and maxing it out. This resulted in numerous strange and aberrant behaviors by the Websphere Console. At first when I tried to login to the Websphere Console, I got the login page OK, but when I actually tried to login, I got an HTTP 500 error. I restarted the Websphere Deployment Manager a couple of times, but that did not help. What was happening was that the Deployment Manager was running OK, but it could not log my login attempt to the Websphere Console because the filesystem was at 100% disk utilization at the time. Deleting some old .ear files freed up some disk space and fixed the problem so that I could at least login to the Websphere Console. But when I tried to install the big .ear file, the Websphere Console lied to me and told me that the .ear file was not a proper .ear file and might be corrupt! That’s when it was noted that the filesystem had gone to 100% disk utilization again. This time UnixOps was engaged to increase the filesystem to 1 GB, and after that the .ear file installed properly, or at least it seemed so at first. But it turned out later that the Websphere Deployment Manager was out of sync with the Websphere node because of the filesystem going to 100% disk utilization, so the new .ear file had not been properly deployed. Resyncing the Websphere Deployment Manager with the Websphere node finally fixed the problem.

All during this install turmoil I kept thinking of the Websphere Deployment Manager and Websphere node as an unruly child pulling a tantrum in a grocery store checkout line just to get its way. I kept thinking to myself, “What is wrong with you? Why are you behaving in such a bratty manner? Did I do something wrong? What do you really want? What will it take to make you stop acting like this?”. This was a rather embarrassing moment, as all parents can attest to, because there were several other people on the install call listening to me trying to discipline my unruly child, all with the smug attitude that bystanders in such situations are infamous for. At one point I even threatened to reboot the Websphere server, the hollow threat that all IT professionals resort to, but infrequently follow through upon, but even that threat did not scare Websphere into submission! I am sure that this scenario is quite familiar to all IT professionals, and that we all routinely adopt an Intentional Stance towards the software that we create and support, and assign to it a personality of its own that we frequently try to bargain with in a vain attempt to get it to behave properly.

Comments are welcome at

To see all posts on softwarephysics in reverse order go to:

Steve Johnston