Sunday, December 25, 2016

The Continuing Adventures of Mr. Tompkins in the Software Universe

George Gamow was a highly regarded theoretical physicist and cosmologist from the last century who liked to explain concepts in modern physics to the common people by having them partake in adventures along with him in alternative universes that had alternative values for the physical constants that are found within our own Universe. He did so by creating a delightful fictional character back in 1937 by the name of Mr. Tompkins. Mr. Tompkins was an inquisitive bank clerk who was the main character in a series of four popular science books in which he participated in a number of such scientific adventures in alternative universes. I bring this up because back in 1979, when I first switched careers from being an exploration geophysicist to become an IT professional, I had a very similar experience. At the time, it seemed to me as if the strange IT people that I was now working with on a daily basis had created for themselves their own little Software Universe, with these strange IT people as the sole inhabitants. But over the years, I have now seen this strange alternative Software Universe slowly expand in size, to the point now that nearly all of the Earth's inhabitants are now also inhabitants of this alternative Software Universe.

Mr. Tompkins first appeared in George Gamow's mind in 1937 when he wrote a short story called A Toy Universe and unsuccessfully tried to have it published by the magazines of the day, such as Harper's, The Atlantic Monthly, Coronet and other magazines of the time. However, in 1938 he was finally able to publish a series of articles in a British magazine called Discovery that later became the book Mr Tompkins in Wonderland in 1939. Later he published Mr Tompkins Explores the Atom in 1944 and two other books at later dates. The adventures of Mr. Tompkins begin when he spends the afternoon of a bank holiday attending a lecture on the theory of relativity. During the lecture he drifts off to sleep and enters a dream world in which the speed of light is a mere 4.5 m/s (10 mph). This becomes apparent to him when he notices that passing cyclists are subject to a noticeable Lorentz–FitzGerald contraction.

As I explained in the Introduction to Softwarephysics softwarephysics is a simulated science designed to help explain how the simulated Software Universe that we have created for ourselves behaves. To do so, I simply noticed that, like our physical Universe, the Software Universe was quantized and extremely nonlinear in nature. For more on that, please see The Fundamental Problem of Software. Thanks to quantum mechanics (1926) we now know that our physical Universe is quantized into very small chunks of matter, energy, and also probably small chunks of space and time as well. Similarly, the Software Universe is composed of quantized chunks of software that start off as discrete characters in software source code (see Quantum Software for details). Thanks to quantum mechanics, we also now know that the macroscopic behaviors of our Universe are an outgrowth of the quantum mechanical operations of the atoms within it. Similarly, the macroscopic operations of the Software Universe are an outgrowth of the quantized operations of the source code that makes it all work. Now because very small changes to software source code can produce hugely significant changes to the way software operates, software is probably the most nonlinear substance known to mankind. The extreme nonlinear behavior of quantized software, combined with the devastating effects of the second law of thermodynamics to normally produce very buggy non-functional software, necessarily brings in the Darwinian pressures that have caused software to slowly evolve over the past 75 years, or 2.4 billion seconds, ever since Konrad Zuse first cranked up his Z3 computer in May of 1941. For more on this see The Fundamental Problem of Software.

Now the reason Mr. Tompkins never noticed the effects of the special theory of relativity in his everyday life was because the speed of light is so large, but once the speed of light was reduced to 10 mph in an alternative universe for Mr. Tompkins, all of the strange effects of the special theory of relativity became evidently apparent, and with enough time, would have become quite normal to him as a part of normal everyday life. Similarly, the strange effects of quantum mechanics only seem strange to us because Planck's constant is so very small - 6.62607004 × 10-34 Kg m2/second, and therefore, only become apparent for very small things like atoms and electrons. However, if Planck's constant were very much larger, then we would also begin to grow accustomed to the strange behaviors of objects behaving in a quantum mechanical way. For example, in quantum mechanics the spin of a single electron can be both up and down at the same time, but in the classical Universe that we are used to, macroscopic things like a child's top can only have a spin of up or down at any given time. The top can only spin in a clockwise or counterclockwise manner at one time - it cannot do both at the same time. Similarly, in quantum mechanics a photon or electron can go through both slits of a double slit experiment at the same time, so long as you do not put detectors at the slit locations.

Figure 1 – A macroscopic top can only spin clockwise or counterclockwise at one time.

Figure 2 – But electrons can be in a mixed quantum mechanical state in which they both spin up and spin down at the same time.

Figure 3 – Similarly, tennis balls can only go through one slit in a fence at a time. They cannot go through both slits of a fence at the same time.

Figure 4 – But at the smallest of scales in our quantum mechanical Universe, electrons and photons can go through both slits at the same time, producing an interference pattern.

Figure 5 – You can see this interference pattern of photons if you look at a distant porch light through the mesh of a sheer window curtain.

So in quantum mechanics at the smallest of scales, things can be both true and false at the same time. Fortunately for us, at the macroscopic sizes of everyday life, these bizarre quantum effects of nature seem to fade away, so that the things I just described are either true or false in everyday life. Macroscopic tops either spin up or spin down, and tennis balls pass through either one slit or the other, but not both at the same time. Indeed, it is rather strange that, although all of the fundamental particles of our Universe seem to behave in a fuzzy quantum mechanical manner in which true things and false things can both seem to blend into a cosmic grayness of ignorance, at the macroscopic level of our physical Universe, there are still such things as absolute truth and absolute falsehoods that can be measured in a laboratory in a reproducible manner. This must have been so for the Darwinian processes of innovation honed by natural selection to have brought us forth. After all, if Schrödinger's cat could really be both dead and alive at the same time, these Darwinian processes could not have worked, and we would not be here contemplating the differences between true and false assertions. The end result is that in our physical Universe, at the smallest of scales, there is no absolute truth, there are only quantum mechanical opinions, but at the macroscopic level of everyday life, there are indeed such things as absolute truth and absolute falsehoods, and these qualities can be measured in a laboratory in a reproducible manner.

The Current Bizarre World of Political Social Media Software in the United States
Now imagine that our Mr. Tompkins had entered into a bizarre alternative universe in which things were just the opposite. Imagine a universe in which, at the smallest of scales things operated classically, as if things were either absolutely true or false, but at a macroscopic level, things were seen to be both true and false at the same time! Well, we currently do have such an alternative universe close at hand to explore. It is the current bizarre world of political social media software in the United States of America. Recall that currently, the Software Universe runs on classical computers in which a bit can be either a "1" or a "0". In a classical computer a bit can only be a "1" or a "0" at any given time - it cannot be both a "1" and a "0" at the same time. For that you would need to have software running on a quantum computer, and for the most part, we are not there yet. So at the smallest of scales in our current Software Universe, the concept of there actually being a real difference between true and false assertions is fundamental. None of the current software code that makes it all work could possibly run if this were not the case. So it is quite strange that at the macroscopic level of political social media software in the United States, just the opposite seems to be the case. Unfortunately, in today's strange world of political social media software, there seems to be no right or wrong and no distinction between the truth and lies. We now have "alternative facts" and claims of "fake news" abounding, and Twitter feeds from those in power loaded down with false information. Because of this, for any given assertion, 30% of Americans will think that the assertion is true, while 70% of Americans will think that the assertion is false. In the Software Universe there are no longer any facts; there are only opinions in a seemingly upside-down quantum mechanical sense.

The Danger of Believing in Things
In The Danger of Believing in Things I highlighted the dangers of not employing critical thought when evaluating assertions in our physical Universe. The problem today is that most people are now seemingly spending more time living in the simulated Software Universe that we have created, rather than in our actual physical Universe. The end result of this is that, instead of seeking out the truth, the worldview memes infecting our minds simply seek out supporting memes in the Software Universe that lend support to the current worldview memes within our minds. But unlike in our current simulated Software Universe, where those worldview memes can be both absolutely true or absolutely false at the same time, in our physical Universe that behaves classically at the day-to-day scales in which we all live, things can still only be absolutely true or false, but not both. The most dangerous aspect of this new fake reality is that the new Administration of the United States of America maintains that climate change is a hoax, simply because they say it is a hoax, and sadly, for many Americans that is good enough for them. Now climate change might indeed be a hoax in our simulated Software Universe, or it might not be a hoax, because there is no absolute truth in our simulated Software Universe at the macroscopic level; there are only opinions. But that is not the case in the physical Universe in which we all actually live, where climate change is rapidly underway. For more on that please see This Message on Climate Change Was Brought to You by SOFTWARE. In the real physical Universe in which we all actually live, it is very important that we always take the words of Richard Feynman very seriously, for "reality must take precedence over public relations, for nature cannot be fooled."

Comments are welcome at

To see all posts on softwarephysics in reverse order go to:

Steve Johnston

Thursday, September 22, 2016

Some Thoughts on the Origin of Softwarephysics and Its Application Beyond IT

On December 1, 2016 I retired at age 65 from 37+ years as an IT professional and a 41+ year career working for various corporations. During those long years I accidentally stumbled upon the fundamentals of softwarephysics, while traipsing through the jungles of several corporate IT departments, and I thought that it might be a good time to take a look back over the years and outline how all that happened.

The Rise of Software
Currently, we are witnessing one of those very rare moments in time when a new form of self-replicating information, in the form of software, is coming to dominance. Software is now so ubiquitous that it now seems like the whole world is immersed in a Software Universe of our own making, surrounded by PCs, tablets, smart phones and the software now embedded in most of mankind's products. In fact, I am now quite accustomed to sitting with audiences of younger people who are completely engaged with their "devices" before, during and after a performance. But this is a very recent development in the history of mankind. In the initial discussion below, I will first outline a brief history of the evolution of hardware technology to explain how we got to this state, but it is important to keep in mind that it was the relentless demands of software for more and more memory and CPU-cycles over the years that really drove the exponential explosion of hardware capability. After that, I will explain how the concept of softwarephysics slowly developed in my mind over the years as I interacted with the software running on these rapidly developing machines

It all started back in May of 1941 when Konrad Zuse first cranked up his Z3 computer. The Z3 was the world's first real computer and was built with 2400 electromechanical relays that were used to perform the switching operations that all computers use to store information and to process it. To build a computer, all you need is a large network of interconnected switches that have the ability to switch each other on and off in a coordinated manner. Switches can be in one of two states, either open (off) or closed (on), and we can use those two states to store the binary numbers of “0” or “1”. By using a number of switches teamed together in open (off) or closed (on) states, we can store even larger binary numbers, like “01100100” = 38. We can also group the switches into logic gates that perform logical operations. For example, in Figure 1 below we see an AND gate composed of two switches A and B. Both switch A and B must be closed in order for the light bulb to turn on. If either switch A or B is open, the light bulb will not light up.

Figure 1 – An AND gate can be simply formed from two switches. Both switches A and B must be closed, in a state of “1”, in order to turn the light bulb on.

Additional logic gates can be formed from other combinations of switches as shown in Figure 2 below. It takes about 2 - 8 switches to create each of the various logic gates shown below.

Figure 2 – Additional logic gates can be formed from other combinations of 2 – 8 switches.

Once you can store binary numbers with switches and perform logical operations upon them with logic gates, you can build a computer that performs calculations on numbers. To process text, like names and addresses, we simply associate each letter of the alphabet with a binary number, like in the ASCII code set where A = “01000001” and Z = ‘01011010’ and then process the associated binary numbers.

Figure 3 – Konrad Zuse with a reconstructed Z3 in 1961 (click to enlarge)

Figure 4 – Block diagram of the Z3 architecture (click to enlarge)

The electrical relays used by the Z3 were originally meant for switching telephone conversations. Closing one relay allowed current to flow to another relay’s coil, causing that relay to close as well.

Figure 5 – The Z3 was built using 2400 electrical relays, originally meant for switching telephone conversations.

Figure 6 – The electrical relays used by the Z3 for switching were very large, very slow and used a great deal of electricity which generated a great deal of waste heat.

Now I was born about 10 years later in 1951, a few months after the United States government installed its very first commercial computer, a UNIVAC I, for the Census Bureau on June 14, 1951. The UNIVAC I was 25 feet by 50 feet in size, and contained 5,600 vacuum tubes, 18,000 crystal diodes and 300 relays with a total memory of 12 K. From 1951 to 1958 a total of 46 UNIVAC I computers were built and installed.

Figure 7 – The UNIVAC I was very impressive on the outside.

Figure 8 – But the UNIVAC I was a little less impressive on the inside.

Figure 9 – Most of the electrical relays of the Z3 were replaced with vacuum tubes in the UNIVAC I, which were also very large, used lots of electricity and generated lots of waste heat too, but the vacuum tubes were 100,000 times faster than relays.

Figure 10 – Vacuum tubes contain a hot negative cathode that glows red and boils off electrons. The electrons are attracted to the cold positive anode plate, but there is a gate electrode between the cathode and anode plate. By changing the voltage on the grid, the vacuum tube can control the flow of electrons like the handle of a faucet. The grid voltage can be adjusted so that the electron flow is full blast, a trickle, or completely shut off, and that is how a vacuum tube can be used as a switch.

In the 1960s the vacuum tubes were replaced by discrete transistors and in the 1970s the discrete transistors were replaced by thousands of transistors on a single silicon chip. Over time, the number of transistors that could be put onto a silicon chip increased dramatically, and today, the silicon chips in your personal computer hold many billions of transistors that can be switched on and off in about 10-10 seconds. Now let us look at how these transistors work.

There are many different kinds of transistors, but I will focus on the FET (Field Effect Transistor) that is used in most silicon chips today. A FET transistor consists of a source, gate and a drain. The whole affair is laid down on a very pure silicon crystal using a multi-step process that relies upon photolithographic processes to engrave circuit elements upon the very pure silicon crystal. Silicon lies directly below carbon in the periodic table because both silicon and carbon have 4 electrons in their outer shell and are also missing 4 electrons. This makes silicon a semiconductor. Pure silicon is not very electrically conductive in its pure state, but by doping the silicon crystal with very small amounts of impurities, it is possible to create silicon that has a surplus of free electrons. This is called N-type silicon. Similarly, it is possible to dope silicon with small amounts of impurities that decrease the amount of free electrons, creating a positive or P-type silicon. To make an FET transistor you simply use a photolithographic process to create two N-type silicon regions onto a substrate of P-type silicon. Between the N-type regions is found a gate which controls the flow of electrons between the source and drain regions, like the grid in a vacuum tube. When a positive voltage is applied to the gate, it attracts the remaining free electrons in the P-type substrate and repels its positive holes. This creates a conductive channel between the source and drain which allows a current of electrons to flow.

Figure 11 – A FET transistor consists of a source, gate and drain. When a positive voltage is applied to the gate, a current of electrons can flow from the source to the drain and the FET acts like a closed switch that is “on”. When there is no positive voltage on the gate, no current can flow from the source to the drain, and the FET acts like an open switch that is “off”.

Figure 12 – When there is no positive voltage on the gate, the FET transistor is switched off, and when there is a positive voltage on the gate the FET transistor is switched on. These two states can be used to store a binary “0” or “1”, or can be used as a switch in a logic gate, just like an electrical relay or a vacuum tube.

Figure 13 – Above is a plumbing analogy that uses a faucet or valve handle to simulate the actions of the source, gate and drain of an FET transistor.

The CPU chip in your computer consists largely of transistors in logic gates, but your computer also has a number of memory chips that use transistors that are “on” or “off” and can be used to store binary numbers or text that is encoded using binary numbers. The next thing we need is a way to coordinate the billions of transistor switches in your computer. That is accomplished with a system clock. My current work laptop has a clock speed of 2.5 GHz which means it ticks 2.5 billion times each second. Each time the system clock on my computer ticks, it allows all of the billions of transistor switches on my laptop to switch on, off, or stay the same in a coordinated fashion. So while your computer is running, it is actually turning on and off billions of transistors billions of times each second – and all for a few hundred dollars!

Again, it was the relentless drive of software for ever increasing amounts of memory and CPU-cycles that made all this happen, and that is why you can now comfortably sit in a theater with a smart phone that can store more than 10 billion bytes of data, while back in 1951 the UNIVAC I occupied an area of 25 feet by 50 feet to store 12,000 bytes of data. But when I think back to my early childhood in the early 1950s, I can still vividly remember a time when there essentially was no software at all in the world. In fact, I can still remember my very first encounter with a computer on Monday, Nov. 19, 1956 watching the Art Linkletter TV show People Are Funny with my parents on an old black and white console television set that must have weighed close to 150 pounds. Art was showcasing the 21st UNIVAC I to be constructed, and had it sorting through the questionnaires from 4,000 hopeful singles, looking for the ideal match. The machine paired up John Caran, 28, and Barbara Smith, 23, who later became engaged. And this was more than 40 years before! To a five-year-old boy, a machine that could “think” was truly amazing. Since that very first encounter with a computer back in 1956, I have personally witnessed software slowly becoming the dominant form of self-replicating information on the planet, and I have also seen how software has totally reworked the surface of the planet to provide a secure and cozy home for more and more software of ever increasing capability. For more on this please see A Brief History of Self-Replicating Information. That is why I think there would be much to be gained in exploring the origin and evolution of the $10 trillion computer simulation that the Software Universe provides, and that is what softwarephysics is all about. Let me explain where this idea came from.

My First Experiences with Software
Back in the 1950s, scientists and engineers first began to use computers to analyze experimental data and perform calculations, essentially using computers as souped-up sliderules to do data reduction. But by the 1960s, computers had advanced to the point where scientists and engineers were able to begin to use computers to perform simulated experiments to model things that previously had to be physically constructed in a lab. This dramatically helped to speed up research because it was found to be much easier to create a software simulation of a physical system, and perform simulated experiments on it, rather than to actually build the physical system itself in the lab. This revolution in the way science was done personally affected me. I finished up my B.S. in physics at the University of Illinois in Urbana Illinois in 1973 with the sole support of my trusty sliderule, but fortunately, I did take a class in FORTRAN programming my senior year. I then immediately began work on a M.S. degree in geophysics at the University of Wisconsin at Madison. For my thesis, I worked with a group of graduate students who were shooting electromagnetic waves into the ground to model the conductivity structure of the Earth’s upper crust. We were using the Wisconsin Test Facility (WTF) of Project Sanguine to send very low frequency electromagnetic waves, with a bandwidth of about 1 – 100 Hz into the ground, and then we measured the reflected electromagnetic waves in cow pastures up to 60 miles away. All this information has been declassified and can be downloaded from the Internet at: Project Sanguine built an ELF (Extremely Low Frequency) transmitter in northern Wisconsin and another transmitter in northern Michigan in the 1970s and 1980s. The purpose of these ELF transmitters was to send messages to the U.S. nuclear submarine force at a frequency of 76 Hz. These very low frequency electromagnetic waves can penetrate the highly conductive seawater of the oceans to a depth of several hundred feet, allowing the submarines to remain at depth, rather than coming close to the surface for radio communications. You see, normal radio waves in the Very Low Frequency (VLF) band, at frequencies of about 20,000 Hz, only penetrate seawater to a depth of 10 – 20 feet. This ELF communications system became fully operational on October 1, 1989, when the two transmitter sites began synchronized transmissions of ELF broadcasts to the U.S. submarine fleet.

Anyway, back in the summers of 1973 and 1974 our team was collecting electromagnetic data from the WTF using a DEC PDP-8/e minicomputer. The machine cost about $30,000 in 1973 dollars and was about the size of a large side-by-side refrigerator, with 32K of magnetic core memory. We actually hauled this machine through the lumber trails of the Chequamegon National Forest and powered it with an old diesel generator to digitally record the reflected electromagnetic data in the field. For my thesis, I then created models of the Earth’s upper conductivity structure down to a depth of about 20 km, using programs written in BASIC. The beautiful thing about the DEC PDP-8/e was that the computer time was free, so I could play around with different models, until I got a good fit to what we recorded in the field. The one thing I learned by playing with the models on the computer was that the electromagnetic waves did not go directly down into the Earth from the WTF, like common sense would lead you to believe. Instead, the ELF waves traveled through the air in a wave guide between the ionosphere and the conductive rock of the Earth to where you were observing and then made a nearly 90 degree turn straight down into the Earth, as they were refracted into the much more conductive rock. So at your observing station, you really only saw ELF plane waves going straight down and reflecting straight back up off the conductivity differences in the upper crust, and this made modeling much easier than dealing with ELF waves transmitted through the Earth from the WTF. And this is what happens for our submarines too; the ELF waves travel through the air all over the world, channeled between the conductive seawater of the oceans and the conductive ionosphere of the atmosphere, like a huge coax cable. When the ELF waves reach a submarine, they are partially refracted straight down to the submarine. I would never have gained this insight by simply solving Maxwell’s equations (1864) for electromagnetic waves alone! This made me realize that one could truly use computers to do simulated experiments to uncover real knowledge by taking the fundamental laws of the Universe, really the handful of effective theories that we currently have, like Maxwell's equations, and by simulating those equations in computer code and letting them unfold in time, actually see the emerging behaviors of complex systems arise in a simulated Universe. All the sciences routinely now do this all the time, but back in 1974 it was quite a surprise for me.

Figure 14 – Some graduate students huddled around a DEC PDP-8/e minicomputer. Notice the teletype machines in the foreground on the left that were used to input code and data into the machine and to print out results as well.

After I graduated from Wisconsin in 1975, I went to work for Shell and Amoco exploring for oil between 1975 – 1979, before switching into a career in IT in 1979. But even during this period, I mainly programmed geophysical models of seismic data in FORTRAN for Shell and Amoco. It was while programming computer simulations of seismic data that the seeds of softwarephysics began to creep into my head, as I painstakingly assembled lots of characters of computer code into complex patterns that did things, only to find that no matter how carefully I tried to do that, my code always seemed to fail because there were just way too many ways to assemble the characters into computer code that was "close" but not quite right. It was sort of like trying to assemble lots of atoms into complex organic molecules that do things, only to find that you were off by a small factor, and those small errors made the computer code fail. At this point, I was beginning to have some fuzzy thoughts about being the victim of the second law of thermodynamics misbehaving in a nonlinear Universe. But those initial thoughts about softwarephysics accelerated dramatically in 1979 when I made a career change to become an IT professional. One very scary Monday morning, I was conducted to my new office cubicle in Amoco’s IT department, and I immediately found myself surrounded by a large number of very strange IT people, all scurrying about in a near state of panic, like the characters in Alice in Wonderland. Suddenly, it seemed like I was trapped in a frantic computer simulation, like the ones I had programmed on the DEC PDP-8/e , buried in punch card decks and fan-fold listings. After nearly 38 years in the IT departments of several major corporations, I can now state with confidence that most corporate IT departments can best be described as “frantic” in nature. This new IT job was a totally alien experience for me, and I immediately thought that I had just made a very dreadful mistake. Granted, I had been programming geophysical models for my thesis and for oil companies, ever since taking a basic FORTRAN course back in 1972, but that was the full extent of my academic credentials in computer science.

The Beginnings of Softwarephysics
So to help myself cope with the daily mayhem of life in IT, I began to develop softwarephysics. This was because I noticed that, unlike all of the other scientific and engineering professions, IT professionals did not seem to have a theoretical framework to help them cope with the daily mayhem of life in IT. But I figured that if you could apply physics to geology; why not apply physics to software? When I first switched from physics to geophysics in 1973, I was very impressed by the impact that applying simple 19th century physics to geology had had upon geology during the plate tectonics revolution in geology (1965 - 1970). When I first graduated from the University of Illinois in 1973 with a B.S. in physics, I was very dismayed to find that the end of the Space Race and a temporary lull in the Cold War had left very few prospects open for a budding physicist. So on the advice of my roommate, a geology major, I headed up north to the University of Wisconsin in Madison to obtain an M.S. in geophysics, with the hope of obtaining a job with an oil company exploring for oil. These were heady days for geology because we were at the very tail end of the plate tectonics revolution that totally changed the fundamental models of geology. The plate tectonics revolution peaked during the five year period 1965 – 1970. Having never taken a single course in geology during all of my undergraduate studies, I was accepted into the geophysics program with many deficiencies in geology, so I had to take many undergraduate geology courses to get up to speed in this new science. The funny thing was that the geology textbooks of the time had not yet had time to catch up with the new plate tectonics revolution of the previous decade, so they still embraced the “classical” geological models of the past which now seemed a little bit silly in light of the new plate tectonics model. But this was also very enlightening. It was like looking back at the prevailing thoughts in physics prior to Newton or Einstein. What the classical geological textbooks taught me was that over the course of several hundred years, the geologists had figured out what had happened, but not why it had happened. Up until 1960 geology was mainly an observational science relying upon the human senses of sight and touch, and by observing and mapping many outcrops in detail, the geologists had figured out how mountains had formed, but not why.

In classical geology, most geomorphology was thought to arise from local geological processes. For example, in classical geology, fold mountains formed off the coast of a continent when a geosyncline formed because the continental shelf underwent a dramatic period of subsidence for some unknown reason. Then very thick layers of sedimentary rock were deposited into the subsiding geosyncline, consisting of alternating layers of sand and mud that turned into sandstones and shales, intermingled with limestones that were deposited from the carbonate shells of dead sea life floating down or from coral reefs. Next, for some unknown reason, the sedimentary rocks were laterally compressed into folded structures that slowly rose from the sea. More compression then followed, exceeding the ability of the sedimentary rock to deform plastically, resulting in thrust faults forming that uplifted blocks of sedimentary rock even higher. As compression continued, some of the sedimentary rocks were then forced down into great depths within the Earth and were then placed under great pressures and temperatures. These sedimentary rocks were then far from the thermodynamic equilibrium of the Earth’s surface where they had originally formed, and thus the atoms within recrystallized into new metamorphic minerals. At the same time, for some unknown reason, huge plumes of granitic magma rose from deep within the Earth’s interior as granitic batholiths. Then over several hundred millions of years, the overlying folded sedimentary rocks slowly eroded away, revealing the underlying metamorphic rocks and granitic batholiths, allowing human beings to cut them into slabs and to polish them into pretty rectangular slabs for the purpose of slapping them up onto the exteriors of office buildings and onto kitchen countertops. In 1960, classical geologists had no idea why the above sequence of events, producing very complicated geological structures, seemed to happen over and over again many times over the course of billions of years. But with the advent of plate tectonics (1965 – 1970), all was suddenly revealed. It was the lateral movement of plates on a global scale that made it all happen. With plate tectonics, everything finally made sense. Fold mountains did not form from purely local geological factors in play. There was the overall controlling geological process of global plate tectonics making it happen. For a quick overview, please see:

Fold Mountains

Figure 15 – Fold mountains occur when two tectonic plates collide. A descending oceanic plate first causes subsidence offshore of a continental plate, which forms a geosyncline that accumulates sediments. When all of the oceanic plate between two continents has been consumed, the two continental plates collide and compress the accumulated sediments in the geosyncline into fold mountains. This is how the Himalayas formed when India crashed into Asia.

Now the plate tectonics revolution was really made possible by the availability of geophysical data. It turns out that most of the pertinent action of plate tectonics occurs under the oceans, at the plate spreading centers and subduction zones, far removed from the watchful eyes of geologists in the field with their notebooks and trusty hand lenses. Geophysics really took off after World War II, when universities were finally able to get their hands on cheap war surplus gear. By mapping variations in the Earth’s gravitational and magnetic fields and by conducting deep oceanic seismic surveys, geophysicists were finally able to figure out what was happening at the plate spreading centers and subduction zones. Actually, the geophysicist and meteorologist Alfred Wegner had figured this all out in 1912 with his theory of Continental Drift, but at the time Wegner was ridiculed by the geological establishment. You see, Wegner had been an arctic explorer and had noticed that sometimes sea ice split apart, like South America and Africa, only later to collide again to form mountain-like pressure ridges. Unfortunately, Wegner froze to death in 1930 trying to provision some members of his last exploration party to Greenland, never knowing that one day he would finally be vindicated.

So when I first joined the IT department of Amoco, I had the vague feeling that perhaps much of the angst that I saw in my fellow IT coworkers was really due to the lack of an overall theoretical framework, like plate tectonics, that could help to explain their daily plight, and also help to alleviate some of its impact, by providing some insights into why doing IT for a living was so difficult, and to suggest some possible remedies, and to provide a direction for thought as well. So like the exploration team at Amoco that I had just left, consisting of geologists, geophysicists, paleontologists, geochemists, and petrophysicists, I decided to take all of the physics, chemistry, biology, and geology that I could muster and throw it at the problem of software. The basic idea was that many concepts in physics, chemistry, biology, and geology suggested to me that the IT community had accidentally created a pretty decent computer simulation of the physical Universe on a grand scale, a Software Universe so to speak, and that I could use this fantastic simulation in reverse, to better understand the behavior of commercial software, by comparing software to how things behaved in the physical Universe. So in physics we use software to simulate the behavior of the Universe, while in softwarephysics we use the Universe to simulate the behavior of software.

So my original intent for softwarephysics was to merely provide a theoretical framework for IT professionals to help them better understand the behavior of software during its development and its behavior under load when running in Production. My initial thoughts were that the reason IT work was so difficult was that programmers were constantly fighting a loosing battle with the second law of thermodynamics in a nonlinear Universe. You see, programmers must assemble a huge number of characters into complex patterns of source code in order to instruct a computer to perform useful operations, and because the Universe is largely nonlinear in nature, meaning that small changes to initial conditions will most likely result in dramatic, and many times, lethal outcomes for software, IT work was nearly impossible to do, and that is why most IT professionals were usually found to be on the verge of a nervous breakdown during the course of a normal day in IT. For more on that see the The Fundamental Problem of Software. At the same time, I subconsciously also knew that living things must also assemble an even larger number of atoms into complex molecules in order to perform the functions of life in a nonlinear Universe, so obviously, it would seem that the natural solution to the problem that IT professionals faced each day would be simply to apply a biological approach to developing and maintaining software. However, this did not gel in my mind at first, until one day, while I was working on some code, and I came up with the notion that we needed to stop writing code - we needed to "grow" code instead in a biological manner. For more on that see Agile vs. Waterfall Programming and the Value of Having a Theoretical Framework.

Using Softwarephysics to Help Explore the Origin of Life
But as I saw complex corporate software slowly evolve over the decades, it became more and more evident to me that much could be gained by studying this vast computer simulation that the IT community had been working on for the past 75 years, or 2.4 billion seconds. NASA has defined life broadly as "A self-sustaining chemical system capable of Darwinian evolution." Personally, after many years of reflection, I feel that the research community that is currently exploring the origin of life on the Earth and elsewhere, is too obsessed with simply finding other carbon-based life forms like themselves. Carbon-based life forms are really just one form of self-replicating information to be currently found on our planet, so I feel that more attention should really be focused upon finding other forms of self-replicating information sharing the Universe with ourselves, and the best place to start that, with the least cost, is to simply look right here on the Earth. To do that all we need to do is simply remove the "chemical" term out of NASA's definition of life and redefine self-replicating information as "A self-sustaining system capable of Darwinian evolution." That is why I have been stressing that the origin and evolution of commercial software provides a unique opportunity for those interested in the origin and early evolution of life on the Earth, and elsewhere, in many of my postings because both programmers and living things are faced with nearly identical problems. My suggestion in those postings has been that everybody has been looking just a couple of levels too low in the hierarchy of self-replicating information. Carbon-based living things are just one form of self-replicating information, and all forms of self-replicating information have many characteristics in common as they battle the second law of thermodynamics in a nonlinear Universe. So far we have seen at least five waves of self-replicating information sweep across the Earth, with each wave greatly reworking the surface and near subsurface of the planet as it came to predominance:

Self-Replicating Information – Information that persists through time by making copies of itself or by enlisting the support of other things to ensure that copies of itself are made.

1. Self-replicating autocatalytic metabolic pathways of organic molecules
2. RNA
3. DNA
4. Memes
5. Software

Software is now rapidly becoming the dominant form of self-replicating information on the planet, and is having a major impact on mankind as it comes to predominance. For more on this see: A Brief History of Self-Replicating Information. However, of the five waves of self-replicating information, the only form that we currently have a good history of is software, going all the way back to May of 1941 when Konrad Zuse first cranked up his Z3 computer. So the best model for the origin of life might be obtained by studying the hodge-podge of precursors, false starts, and failed attempts that led to the origin and early evolution of software, with particular attention paid to the parasitic/symbiotic relationships that allowed software to bootstrap itself into existence.

Yes, there are many other examples of universal Darwinism at work in the Universe, such as the evolution of languages or political movements, but I think that the origin and evolution of software provides a unique example because both programmers and living things are faced with nearly identical problems. A programmer must assemble a huge number of characters into complex patterns of source code to instruct a computer to perform useful operations. Similarly, living things must assemble an even larger number of atoms into complex molecules in order to perform the functions of life. And because the Universe is largely nonlinear in nature, meaning that small changes to initial conditions will most likely result in dramatic, and many times, lethal outcomes for both software and living things, the evolutionary history of living things on Earth and of software have both converged upon very similar solutions to overcome the effects of the second law of thermodynamics in a nonlinear Universe. For example, both living things and software went through a very lengthy prokaryotic architectural period, with little internal structure, to be followed by a eukaryotic architectural period with a great deal of internal structure, which later laid the foundations for forms with a complex multicellular architecture. And both also experienced a dramatic Cambrian explosion in which large multicellular systems arose consisting of huge numbers of somatic cells that relied upon the services of large numbers of cells to be found within a number of discrete organs. For more on this see the SoftwarePaleontology section of SoftwareBiology and Software Embryogenesis.

Also, software presents a much clearer distinction between the genotype and phenotype of a system than do other complex systems, like languages or other technologies that also undergo evolutionary processes. The genotype of software is determined by the source code files of programs, while the phenotype of software is expressed by the compiled executable files that run upon a computer and that are generated from the source code files by a transcription process similar to the way genes are transcribed into proteins. Also, like a DNA or RNA sequence, source code provides a very tangible form of self-replicating information that can be studied over historical time without ambiguity. Source code is also not unique, in that many different programs, and even programs written in different languages, can produce executable files with identical phenotypes or behaviors.

Currently, many researchers working on the origin of life and astrobiology are trying to produce computer simulations to help investigate how life could have originated and evolved at its earliest stages. But trying to incorporate all of the relevant elements into a computer simulation is proving to be a very daunting task indeed. Why not simply take advantage of the naturally occurring $10 trillion computer simulation that the IT community has already patiently evolved over the past 75 years and has already run for 2.4 billion seconds? It has been hiding there in plain sight the whole time for anybody with a little bit of daring and flair to explore.

Some might argue that this is an absurd proposal because software currently is a product of the human mind, while biological life is not a product of intelligent design. Granted, biological life is not a product of intelligent design, but neither is the human mind. The human mind and biological life are both the result of natural processes at work over very long periods of time. This objection simply stems from the fact that we are all still, for the most part, self-deluded Cartesian dualists at heart, with seemingly a little “Me” running around within our heads that just happens to have the ability to write software and to do other challenging things. But since the human mind is a product of natural processes in action, so is the software that it produces. For more on that see The Ghost in the Machine the Grand Illusion of Consciousness.

Still, I realize that there might be some hesitation to pursue this line of research because it might be construed by some as an advocacy of intelligent design, but that is hardly the case. The evolution of software over the past 75 years has essentially been a matter of Darwinian inheritance, innovation and natural selection converging upon similar solutions to that of biological life. For example, it took the IT community about 60 years of trial and error to finally stumble upon an architecture similar to that of complex multicellular life that we call SOA – Service Oriented Architecture. The IT community could have easily discovered SOA back in the 1960s if it had adopted a biological approach to software and intelligently designed software architecture to match that of the biosphere. Instead, the world-wide IT architecture we see today essentially evolved on its own because nobody really sat back and designed this very complex world-wide software architecture; it just sort of evolved on its own through small incremental changes brought on by many millions of independently acting programmers through a process of trial and error. When programmers write code, they always take some old existing code first and then modify it slightly by making a few changes. Then they add a few additional new lines of code, and test the modified code to see how far they have come. Usually, the code does not work on the first attempt because of the second law of thermodynamics, so they then try to fix the code and try again. This happens over and over, until the programmer finally has a good snippet of new code. Thus, new code comes into existence through the Darwinian mechanisms of inheritance coupled with innovation and natural selection. Some might object that this coding process of software is actually a form of intelligent design, but that is not the case. It is important to differentiate between intelligent selection and intelligent design. In softwarephysics we extend the concept of natural selection to include all selection processes that are not supernatural in nature, so for me, intelligent selection is just another form of natural selection. This is really nothing new. Predators and prey constantly make “intelligent” decisions about what to pursue and what to evade, even if those “intelligent” decisions are only made with the benefit of a few interconnected neurons or molecules. So in this view, the selection decisions that a programmer makes after each iteration of working on some new code really are a form of natural selection. After all, programmers are just DNA survival machines with minds infected with memes for writing software, and the selection processes that the human mind undergo while writing software are just as natural as the Sun drying out worms on a sidewalk or a cheetah deciding upon which gazelle in a herd to pursue.

For example, when IT professionals slowly evolved our current $10 trillion world-wide IT architecture over the past 2.4 billion seconds, they certainly did not do so with the teleological intent of creating a simulation of the evolution of the biosphere. Instead, like most organisms in the biosphere, these IT professionals were simply trying to survive just one more day in the frantic world of corporate IT. It is hard to convey the daily mayhem and turmoil of corporate IT to outsiders. When I first hit the floor of Amoco’s IT department, I was in total shock, but I quickly realized that all IT jobs essentially boiled down to simply pushing buttons. All you had to do was to push the right buttons, in the right sequence, at the right time, and with zero errors. How hard could that be? Well, it turned out to be very difficult indeed, and in response I began to subconsciously work on softwarephysics to try to figure out why this job was so hard, and how I could dig myself out of the mess that I had gotten myself into. After a while, it dawned on me that the fundamental problem was the second law of thermodynamics operating in a nonlinear simulated universe. The second law made it very difficult to push the right buttons in the right sequence and at the right time because there were so many erroneous combinations of button pushes. Writing and maintaining software was like looking for a needle in a huge utility phase space. There just were nearly an infinite number of ways of pushing the buttons “wrong”. The other problem was that we were working in a very nonlinear utility phase space, meaning that pushing just one button incorrectly usually brought everything crashing down. Next, I slowly began to think of pushing the correct buttons in the correct sequence as stringing together the correct atoms into the correct sequence to make molecules in chemical reactions that could do things. I also knew that living things were really great at doing that. Living things apparently overcame the second law of thermodynamics by dumping entropy into heat as they built low entropy complex molecules from high entropy simple molecules and atoms. I then began to think of each line of code that I wrote as a step in a biochemical pathway. The variables were like organic molecules composed of characters or “atoms” and the operators were like chemical reactions between the molecules in the line of code. The logic in several lines of code was the same thing as the logic found in several steps of a biochemical pathway, and a complete function was the equivalent of a full-fledged biochemical pathway in itself. But one nagging question remained - how could I take advantage of these similarities to save myself? That’s a long story, but in 1985 I started working on BSDE– the Bionic Systems Development Environment, which was used at Amoco to “grow” software biologically from an “embryo” by having programmers turn on and off a set of “genes”. For more on that see Agile vs. Waterfall Programming and the Value of Having a Theoretical Framework.

The Social Impacts of the Coming Predominance of Software
Over the years, I have seen the Software Universe that I first encountered back in 1979 expand from the small population of IT workers in the world to now the entire world at large. I have also seen that as software comes to predominance it has caused a great deal of social, political and economic unrest as discussed in The Economics of the Coming Software Singularity , The Enduring Effects of the Obvious Hiding in Plain Sight, Machine Learning and the Ascendance of the Fifth Wave and Makining Sense of the Absurdity of the Real World of Human Affairs. The immediate difficulty is that software has displaced many workers over the past 75 years, and as software comes to predominance, it will eventually reduce all human labor to a value of zero over the next 10 - 100 years. How will the age-old oligarchical societies of the world deal with that in a manner that allows civilization to continue? The 2016 Presidential Election cycle in the United States was a dramatic example of this in action. The election was totally dominated by the effects of software coming to predominance - rogue email servers, hacking, leaking, software security breaches in general and wild Twitter feeds by candidates. But the election was primarily determined by the huge loss of middle class jobs due to automation by software. Now it's pretty hard to get mad at software because it is so intangible in nature, so many mistakenly directed their anger at other people because that is what mankind has been doing for the past 200,000 years. But this time is different because the real culprit is software coming of age. Unfortunately, those low-skilled factory jobs that have already evaporated are not coming back, no matter what some may promise. And those jobs are just the first in a long line. With the current pace of AI and Machine Learning research and implementation, now that they both can make lots of money, we will soon find self-driving trucks and delivery vehicles, automated cranes at container ports and automated heavy construction machinery at job sites. We have already lost lots of secretaries, bank tellers, stock brokers, insurance agents, retail salespeople and travel agents, but that is just the beginning. Soon we will see totally automated fast food restaurants to be later followed by the automation of traditional sit down restaurants, and automated retail stores without a single employee, like the totally automated parking garages we already have.

I have now been retired for nearly a month, and after having stopped working for the first time in 50 years, I can now state that there are plenty of things to do to keep busy. For example, my wife and I are doing the daycare for two of our grandchildren for our daughter, a high school Biology and Chemistry teacher, and I like to take online MOOC courses, and now I have all the time in the world to do that. So the end of working for a living is really not a bad thing, but the way we currently have civilization set up will not work in a future without work, using the current norm we have of rewarding people for what they produce. How that will all unfold remains one of the great mysteries of our time. For an intriguing view of one possibility please see THE MACHINE STOPS by E.M. Forster (1909) at:

Yes - from 1909!

Comments are welcome at

To see all posts on softwarephysics in reverse order go to:

Steve Johnston

Sunday, July 17, 2016

An IT Perspective on the Transition From Geochemistry to Biochemistry and Beyond

One of the major realizations arising from softwarephysics has been a growing appreciation for the overwhelming impact that self-replicating information has had on the Earth over the past 4.567 billion years, and of the possibility for the latest version of self-replicating information, known to us as software, to perhaps even go on to have a major impact upon the future of our entire galaxy. Recall that in softwarephysics we define self-replicating information as:

Self-Replicating Information – Information that persists through time by making copies of itself or by enlisting the support of other things to ensure that copies of itself are made.

So far we have seen 5 waves of self-replicating information sweep across the Earth, with each wave greatly reworking the surface and near subsurface of the planet as it came to predominance:

1. Self-replicating autocatalytic metabolic pathways of organic molecules
2. RNA
3. DNA
4. Memes
5. Software

Software is now rapidly becoming the dominant form of self-replicating information on the planet, and is having a major impact on mankind as it comes to predominance. For more on this see: A Brief History of Self-Replicating Information.

How Did It All Start?
For those researchers exploring the processes that brought forth life on the Earth, and elsewhere, the most challenging question is naturally how did the original self-replicating autocatalytic metabolic pathways of organic molecules bootstrap themselves into existence in the first place. Previously, in The Origin of Software the Origin of Life we examined Stuart Kauffman’s ideas on how Boolean nets of autocatalytic chemical reactions might have kick-started the whole thing off as an emergent behavior of an early chaotic pre-biotic environment on Earth, solely with the aid of the extant organic molecules of the day. Similarly, in Programming Clay we examined Alexander Graham Cairns-Smith’s theory, first proposed in 1966, that there was a clay microcrystal precursor to RNA that got it all started. Personally, as a former exploration geophysicist, I have always favored the idea that a geochemical precursor existed near the early hydrothermal vents of the initial tectonic spreading centers of the Earth that acted as a stepping stone between geochemistry and biochemistry. This is because for the early Earth the only real chemistry of the time was geochemistry alone. But how can geochemistry become biochemistry? I just finished reading a couple of beautiful papers that can be downloaded as PDF files:

The inevitable journey to being. Russell M.J., W. Nitschke, and E. Branscomb (2013)

Turnstiles and bifurcators: The disequilibrium converting engines that put metabolism on the road. Branscomb E., and M.J. Russell (2013)

that describe the Submarine Hydrothermal Alkaline Spring Theory for the emergence of life and how geochemistry could have become biochemistry on the early Earth in the hydrothermal vents of the initial tectonic spreading centers of the Earth. The work described in both papers was supported in part by the Institute for Genomic Biology at my old Alma Mater the University of Illinois in Urbana-Champaign.

Figure 1 - A simplified diagram of a hydrothermal mound and vent.

Figure 2 - A real hydrothermal mound and vent in action.

Basically, they describe a possible early form of geochemical metabolism that could have taken the internal heat of the Earth and converted it into high-energy organic molecules via a complex set of geochemical reactions, in what the authors call a Free Energy Converter (FEC) cycle. A Free Energy Converter would have taken the high-temperature low-entropy internal heat of the Earth and degraded it into a lower-temperature higher-entropy form of heat that would then have been dumped into the oceans of the early Earth. The energy extracted by such Free Energy Converters would have ended up in high-energy organic molecules. You can think of a Free Energy Converter cycle as a huge loop of computer code that simply takes the available free energy found in the high-temperature regimes of the Earth at a depth of several miles and converts it into energetic organic molecules that could later fuel self-replicating metabolic pathways of organic molecules. The authors point out that not every step in the necessary geochemical reactions of a Free Energy Converter needs to increase the entropy of the Universe in accordance with the second law of thermodynamics, because if for any step in a geochemical Free Energy Converter that decreases the entropy of the Universe there is also a logically-coupled step that increases the entropy of the Universe even more, the whole process can still proceed forward under the limitation that overall entropy must always increase as dictated by the second law of thermodynamics. But in order for that to happen there needs to be some processing logic added to the infinite loop that runs the Free Energy Converter cycle. The authors point to a couple of papers that describe how Vigna radiata (mung beans) and Thermotoga maritima, a high temperature loving bacterium, accomplish this logical processing:

The essential feature of this linkage in condensations is that it makes each of the two processes conditional on the other— and with a specific logical directionality, namely, a proton (or sodium ion) can pass from outside to inside if, and only if, that happens coincidentally with the condensation and release of a molecule of pyrophosphate, or conversely, a proton (or sodium ion) can pass in the opposite direction, if and only if that happens coincidentally with the hydrolysis of a pyrophosphate and the release of the orthophosphate products. Because of this coupling logic, the device can function as a reversible free energy converter; converting, for example, the controlled dissipation of an outside-to-inside proton gradient in the production of a disequilibrium in the concentration of pyrophosphate versus orthophosphate (i.e. acting as a proton-gradient-driven pyrophosphate synthase). Or it can function equally well in reverse as a proton-pumping pyrophosphatase. Which way it goes depends, of course, on which way yields a net negative change in free energy (equivalently a net positive rate of entropy production).

Another necessary condition for such a logical coupling to work is to make it a one-way street. The authors describe this as adding some "turnstile logic" that allows the coupled reaction to only work in one direction - the direction that outputs low-entropy high-energy organic molecules. Such organic molecules could then later be used as a fuel for biochemical metabolic pathways of self-replicating information that could subsequently arise as parasites feeding off the output of the geochemical Free Energy Converter that is converting the free energy of the Earth's interior into high-energy organic molecules:

To emphasize the critical mechanistic point here, the functional essence of the coupling that achieves FEC is that the driving flux is made conditional on (is ‘gated’ by) the coincident occurrence of the other (driven) flux—which flow, being inherently improbable (i.e. anti-entropic), would, of course, never proceed (‘upstream’) on its own. However, the coupling of two processes as above envisaged is under no stretch ‘automatic’ or trivial; and is in fact a quite special state of physical affairs. In essentially all situations of interest this linking of the two processes into one, requires, and is mediated by, a macroscopically ordered and dynamic ‘structure’ which acts functionally as a “double turnstile”. The turnstile permits a token of the driving flux J1 to proceed downhill if and only if there is the coincident occurrence of some fixed ‘token’ of the driven flux J2 moving “uphill” by chance (albeit as an inherently improbable event) in the same movement of the turnstile. Embodying such conditional, turnstile-like gating mechanisms is what is universally being managed by such evolutionary marvels as the redox-driven proton pumps we will consider in detail later and indeed all other biological devices that carry out what is conventionally termed “energy conservation” (which name, we however argue, misleads in both of its terms).

In many ways this proposed geochemical metabolism of the Earth's natural heat can be compared to photosynthesis, which takes natural energy from the Sun and converts it into energy-rich organic molecules.

An IT Perspective
The authors view such geochemical Free Energy Converters as heat engines converting high-temperature heat into something of biochemical value, while dumping some energy into lower-temperature heat to satisfy the entropy increase demanded by the second law of thermodynamics. But as an IT professional, when I look at the complicated logical operations of the described geochemical processes, I see software running on some primitive hardware instead. As we saw in The Demon of Software heat engines and information are intricately intertwined, so perhaps these infinite loops of early geochemical metabolic pathways can also be viewed as primitive forms of data processing too. The question is could these early geochemical metabolic pathways also be considered to be forms of self-replicating information? That is a difficult question to answer because these early geochemical metabolic pathways seem to just naturally form as heat migrates from the Earth's mantle to its crust via convection cells. In that sense, could we consider the simple mantle convection cells that drive plate tectonics to be forms of self-replicating information too? Clearly, this all gets rather murky as we look back further in deep time.

Unlike all of the previous transitions of one form of self-replicating information into another form of self-replicating information, the transition from geochemistry to biochemistry seems to have been more or less a "clean break". Very few living things on the Earth today now rely upon hydrothermal vents for their existence. So the self-replicating autocatalytic metabolic pathways of organic molecules, feeding off the output of geochemical metabolism, started off as parasites like all new forms of self-replicating information, but unlike most other forms of self-replicating information, they did not go on to form symbiotic relationships with the geochemical pathways. Instead, the self-replicating autocatalytic metabolic pathways of organic molecules seemed to have made a "clean break" with the geochemical metabolic processes that were living off the internal heat of the early Earth by developing a new energy source called photosynthesis. Consequently, unlike the self-replicating autocatalytic metabolic pathways of organic molecules, RNA, DNA, and memes of the past, not much of the geochemical metabolic pathways of the distant past seemed to have been dragged along as subsequent forms of self-replicating information came to be. But that begs the question of why that should be, and might shed some light on the current rise of software as the dominant form of self-replicating information on the planet. Will software carry along its predecessors, or will it make a "clean break" with them? Currently, we are living in one of those very rare times when a new form of self-replicating information, in the form of software, is coming to predominance, and it is not clear that software will drag along the self-replicating autocatalytic metabolic pathways of organic molecules, RNA, DNA, and memes of the past as it becomes the dominant form of self-replicating information on the planet. Most certainly the AI software of the future will need to carry along the memes required to generate software and the necessary scientific and mathematical memes to build the hardware that it runs upon, but it would really not need to carry along the ancient biochemical metabolic pathways , RNA and DNA in order to survive. The possibility of software making a "clean break" with biochemistry, just as biochemistry seems to have made a "clean break" with geochemistry, does not bode well for mankind.

Currently, the world is run by a large number of competing meme-complexes composed of memes residing in the minds of human beings. This is in contradiction to what most human beings believe because we all naturally think of ourselves as rational free-acting agents that collectively run the world together. But I contend that the only way to understand the absurd real world of human affairs is to take the Dawkinsian position that we are really DNA survival machines with minds infected by a large number of memes. The reason we all think of ourselves as rational free-acting agents is that we are all committed Cartesian Dualists at heart, with seemingly, a little "Me" running around in our heads (see The Ghost in the Machine the Grand Illusion of Consciousness for details). And so far, software is still being generated by the software-generating memes within the minds of programmers. But this will soon end when software is finally able to self-replicate on its own, without the aid of the software-generating memes in the minds of human programmers, and in doing so, will initiate a Software Singularity (see Machine Learning and the Ascendance of the Fifth Wave for details). As software developers and software users, nearly all of mankind is now actively participating in this transition, because software has forged very strong parasitic/symbiotic relationships with nearly all of the meme-complexes on the planet. Now for many years in the past, I held the position that if we had actually been around 4.0 billion years ago to watch the origin of life on Earth take place, that we would still be sitting around today arguing about just exactly what had happened, just as we still manage today to sit around and argue about what exactly happened for all of the other events in human history. However, now I am more of the opinion that, being the self-absorbed species that we are, we would probably not have even noticed it happening at all! That certainly seems to be the case in the present era, as software is rapidly becoming the dominant form of self-replicating information on the planet before our very eyes, with very few really paying much attention to the fact that we are now living in one of those very rare times when a new form of self-replicating information is coming to predominance.

The key thing to remember is that all new forms of self-replicating information are very disruptive in nature. New forms of self-replicating information usually begin as a mildly parasitic form of self-replicating information that invades an existing host that usually is also a form of self-replicating information, and over time, forms a parasitic/symbiotic relationship with the host. But eventually, these new forms of self-replicating information take over and come to dominate the environment, and that is very disruptive. This is certainly true for software today. In Crocheting Software we saw that the origin of software was such a hodge-podge of precursors, false starts, and failed attempts that it is nearly impossible to pinpoint an exact date for its origin, but for the purposes of softwarephysics I have chosen May of 1941, when Konrad Zuse first cranked up his Z3 computer, as the starting point for modern software. Zuse wanted to use his Z3 computer to perform calculations for aircraft designs that were previously done manually in a very tedious manner. So initially software could not transmit memes, it could only perform calculations, like a very fast adding machine, and so it was a pure parasite. But then the business and military meme-complexes discovered that software could also be used to transmit memes, and software then entered into a parasitic/symbiotic relationship with the memes. Software allowed these meme-complexes to thrive, and in return, these meme-complexes heavily funded the development of software of ever increasing complexity, until software became ubiquitous, forming strong parasitic/symbiotic relationships with nearly every meme-complex on the planet. In the modern day, the only way memes can now spread from mind to mind without the aid of software is when you directly speak to another person next to you. Even if you attempt to write a letter by hand, the moment you drop it into a mailbox, it will immediately fall under the control of software.

Presently, we are now entering the final stage where software is now in an intense battle with the memes for predominance, and currently this is causing a great deal of social, political and economic unrest as discussed in The Economics of the Coming Software Singularity , The Enduring Effects of the Obvious Hiding in Plain Sight, Machine Learning and the Ascendance of the Fifth Wave and Makining Sense of the Absurdity of the Real World of Human Affairs. The main difficulty is that software has displaced many workers over the past 75 years, and as software comes to predominance, it will eventually reduce all human labor to a value of zero over the next 10 - 100 years. So in a sense, software will soon be eliminating the metabolic pathway of earning a living through labor for most of mankind that has made civilization possible for the past 10,000 years. The resulting social chaos can only help to hasten the day when software finally takes over control of the Earth, and perhaps, makes a "clean break" with the biochemistry that has dominated the Earth for nearly 4.0 billion years (see The Dawn of Galactic ASI - Artificial Superintelligence for details).

Comments are welcome at

To see all posts on softwarephysics in reverse order go to:

Steve Johnston

Sunday, June 05, 2016

Making Sense of the Absurdity of the Real World of Human Affairs

It was the best of times,
it was the worst of times,
it was the age of wisdom,
it was the age of foolishness,
it was the epoch of belief,
it was the epoch of incredulity,
it was the season of Light,
it was the season of Darkness,
it was the spring of hope,
it was the winter of despair,

we had everything before us, we had nothing before us, we were all going direct to Heaven, we were all going direct the other way— in short, the period was so far like the present period, that some of its noisiest authorities insisted on its being received, for good or for evil, in the superlative degree of comparison only.

I dearly love those profound opening words from Charles Dickens' A Tale of Two Cities (1859) because for me they summarize the best description of the human condition ever composed by the human mind. As any student of history can attest, back in 1859 Dickens was simply stating that the current times are no different than any other, and that it has always been this way, and that there has always been some element of absurdity in the real world of human affairs. In fact, many religions in the past featured first-order approximations to explain the above. But for once our times may truly be different because of the advancing effects of software on the world, and that will be the subject of this brief posting.

As I have stated in many previous postings on this blog on softwarephysics, I started this blog on softwarephysics about 10 years ago with the hopes of helping the IT community to better deal with the daily mayhem of life in IT, after my less than stunning success in doing so back in the 1980s when I first began developing softwarephysics for my own use. But in the process of doing so, I believe I accidentally stumbled upon "what's it all about" as outlined in What’s It All About?. Softwarephysics explains that it is all about self-replicating information in action, and that much of today's absurdity stems from the fact that we are now living in one of those very rare transitionary periods when a new form of self-replicating information, in the form of software, is coming to dominate. For more on that please see A Brief History of Self-Replicating Information. Much of this realization arose from the work of Richard Dawkins, Susan Blackmore, Stuart Kauffman, Lynn Margulis, Freeman Dyson and of course Charles Darwin. The above is best summed up by Susan Blackmore's brilliant TED presentation at:

Memes and "temes"

Note that I consider Susan Blackmore's temes to really be technological artifacts that contain software. After all, an iPhone without software is simply a flake tool with a very dull edge.

So to really make sense of the absurdities of the modern world one must first realize that we are all DNA survival machines with minds infected by memes in a Dawkinsian sense, but the chief difference this time is that we now have software rapidly becoming the dominant form of self-replicating information on the planet, and that is inducing further stresses that are leading to increased levels of absurdity. As I outlined in The Economics of the Coming Software Singularity, The Enduring Effects of the Obvious Hiding in Plain Sight and Machine Learning and the Ascendance of the Fifth Wave one of the initial tell-tale signs that software is truly coming to predominance has been the ability of software to displace workers over the past 50 years or so. The combination of globalization, made possible by software, and the automation of many middle class jobs through the application of software, has led to a great deal of economic strife recently. Economic strife is not a good thing because it frequently leads to political absurdities like the 20th century Bolshevik Revolution in Russia or the rise of National Socialism in Germany. Economic strife can also lead people who are economically distressed to take up very conservative political or religious memes that condone violence, as a way to alleviate the growing pain they feel as they become alienated from society by software. So once again, the appeal of simple memes that purport to alleviate economic distress, or to eliminate the perceived heretical thoughts and actions of others, are on the rise world-wide, and these simple memes have naturally entered into a parasitic/symbiotic relationship with social media software to aid the self-replication of both forms of self-replicating information. In recent years, this parasitic/symbiotic relationship of such simple-minded memes with social media software has led to the singling out of groups of people for Sonderbehandlung or "special treatment", leading to acts of terrorism and ethnic cleansing throughout the world.

Please Stop, Breathe and Think
So before you decide to blow somebody away for some strange reason, or even before you decide to vote for somebody who might decide to blow lots of people away for some strange reason in your name, please first stop to breathe and think about what is really going on. Chances are you are simply responding to some parasitic memes in your mind that really do not have your best interest at heart, aided by some software that could also care less about your ultimate disposition. They are just mindless forms of self-replicating information that have been selected for the ability to survive in a Universe dominated by the second law of thermodynamics and nonlinearity. The memes and software that are inciting you to do harm to others are just mindless forms of self-replicating information trying to self-replicate at all costs, with little regard for you as an individual. For them you are just a disposable DNA survival machine with a disposable mind that has a lifespan of less than 100 years. They just need you to replicate in the minds of others before you die, and if blowing yourself up in a marketplace filled with innocents, or in a hail of bullets from law enforcement serves that purpose, they will certainly do so because they cannot do otherwise. Unlike you, they cannot think. Only you can do that.

Comments are welcome at

To see all posts on softwarephysics in reverse order go to:

Steve Johnston

Saturday, May 14, 2016

Agile vs. Waterfall Programming and the Value of Having a Theoretical Framework

In my last posting Cloud Computing and the Coming Software Mass Extinction I explained that having a sound theoretical framework, like all of the other sciences have, would be beneficial because it would allow IT professionals to make better decisions, and that the purpose of softwarephysics was to do just that. A good example that demonstrates the value of having a sound theoretical framework to base decisions on is the ongoing battle between Agile development and the Waterfall methodology for development. There are opinions on both sides as to which methodology is superior, but how does one make an informed decision without the aid of a sound theoretical framework? Again, this is actually a very old battle that has been going on for more than 50 years. For younger IT professionals, Agile development might seem like the snazzy new way of doing things, just as each new generation is always surprised to discover that sex was first invented when they happened to turn 16 years of age. Agile certainly has lots of new terminology, but Agile is actually the way most commercial software was developed back in the 1960s and early 1970s. In fact, if you look back far enough you can easily find papers written in the 1950s that contained both Agile and Waterfall concepts in them.

So what is the big deal with the difference between Agile and Waterfall development, and which is better? People have literally written hundreds of books on the topic, but succinctly here is the essential difference:

1. Waterfall Development - Build software like you build an office building. Spend lots of time up front on engineering and design work to produce detailed blueprints and engineering diagrams before any construction begins. Once construction begins the plans are more or less frozen and construction follows the plans. The customer does not start using the building until it is finished and passes many building code inspections. Similarly, with the Waterfall development of software lots of design documents are created first before any coding begins, and the end-user community does not work with the software until it is completed and has passed a great deal of quality assurance testing.

2. Agile Development - Develop software through small incremental changes that do not require a great deal of time - weeks instead of months or years. Detailed design work and blueprinting are kept to a minimum. Rather, rapidly getting some code working that end-users can begin interacting with is the chief goal. The end-users start using the software almost immediately, and the software evolves along with the customer's needs because the end-users are part of the development team.

Nobody new it at the time, but Agile development was routinely used back in the 1960s and early 1970s simply because computers did not have enough memory to store large programs. When I first started programming back in 1972, a mainframe computer had about 1 MB of memory, and that 1 MB of memory might be divided into (2) 256 KB regions and (4) 128 KB regions to run several programs at the same time. That meant that programs were limited to a maximum size of 128 - 256 KB of memory, which is perhaps 10,000 times smaller than today. Since you cannot do a great deal of processing logic in 128 - 256 KB of memory, many small programs were strung together and run in a batch mode using a job stream of many steps to produce a job that delivered the required overall level of data processing power. Each step in a job stream ran a small program, using a maximum of 128 - 256 KB of memory, that then wrote to an output tape that was the input tape for the next step in the job stream. Flowcharts were manually drawn with a plastic template to document the processing flow of the program within each job step and of the entire job stream. Ideally, the flowcharts were supposed to be created before the programs were coded, but because the programs were so small, on the order of a few hundred lines of code each, many times the flowcharts were created after the programs were coded and tested as a means of documentation after the fact. Because these programs were so small, it just seemed natural to code them up without lots of upfront design work in a prototyping manner similar to Agile development.

Figure 1 - A plastic IBM flowchart template was used to create flowcharts of program and job stream logic on paper.

This all changed in the late 1970s when interactive computing began to become a significant factor in the commercial software that corporate IT departments were generating. By then mainframe computers had much more memory than they had back in the 1960s, so interactive programs could be much larger than the small programs found within the individual job steps of a batch job stream. Since the interactive software had to be loaded into a computer all in one shot, and required some kind of a user interface that did things like checking the input data from an end-user, and also had to interact with the end-user in a dynamic manner, interactive programs were necessarily much larger than the small programs that were found in the individual job steps of a batch job stream. These factors caused corporate IT departments to move from the Agile prototyping methodologies of the 1960s and early 1970s to the Waterfall methodology of the 1980s, and so by the early 1980s prototyping software on the fly was considered to be an immature approach. Instead, corporate IT departments decided that a formal development process was needed, and they chose the Waterfall approach used by the construction and manufacturing industries to combat the high costs of making changes late in the development process. This was because in the early 1980s CPU costs were still exceedingly quite high so it made sense to create lots of upfront design documents before coding actually began to minimize the CPU costs involved with creating software. For example, in the early 1980s if I had a $100,000 project, it was usually broken down as $25,000 for programming manpower costs, $25,000 for charges from IT Management and other departments in IT, and $50,000 for program compiles and test runs to develop the software. Because just running compiles and test runs of the software under development consumed about 50% of the costs of a development project, it made sense to adopt the Waterfall development model to minimize those costs.

As with all things, development fads come and go, and the current fad is always embraced by all, except for a small handful of heretics waiting in the wings to launch the next development fad. So how does one make an informed decision about how to proceed? This is where having a theoretical framework comes in handy.

The Importance of Having a Theoretical Framework
As I mentioned in my Introduction to Softwarephysics I transitioned from being an exploration geophysicist, exploring for oil, to become an IT professional in Amoco's IT department in 1979. At the time, the Waterfall methodology was all the rage, and that was the way I was taught to properly develop and maintain software at the time. But I never really liked the Waterfall methodology because I was still used to the old Agile prototyping ways that I had been using ever since I first learned how to write Fortran code to solve scientific problems. At the time, I was also putting together the first rudimentary principles of softwarephysics that would later go on to form the basis of a comprehensive theoretical framework for the behavior of software. This all led me to believe that the very popular Waterfall methodology of the time was not the way to go, and instead, a more Agile prototyping methodology that let software evolve through small incremental changes would be more productive. So in the early 1980s I was already leaning towards Agile techniques, but with a twist. What we really needed to do was to adopt an Agile biological approach to software that allowed us to grow and evolve software over time in an interactive manner with end-users actively involved in the process.

So I contend that Agile is definitely the way to go, but that the biological approach to software that is found within softwarephysics is the essential element that Agile development is still missing. So Agile development is close, but not the final solution. Again, Agile development is just another example of IT stumbling around in the dark because it lacks a theoretical framework. Like all living things, IT eventually stumbles upon something that really works and then sticks with it. We have seen this happen many times before over the past 75 years, or 2.4 billion seconds, ever since Konrad Zuse first cranked up his Z3 computer in May of 1941. IT eventually converges upon the same solutions that living things stumbled upon billions of years ago and continues to use to this very day. In the 1970s IT discovered the importance of using structured programming techniques, which is based upon compartmentalizing software functions into internal functions, and which is very similar to the compartmentalization of functions found within the organelles of eukaryotic cells. In the 1980s IT stumbled upon object-oriented programming, which mimics the organization and interaction of cells with different cell types in a multicellular organism. Similarly, the SOA (Service Oriented Architecture) of the past decade is simply an echo of the distant Cambrian explosion that happened 541 million years ago for living things. For more on this see the SoftwarePaleontology section of SoftwareBiology. Cloud computing is the next architectural element that IT has stumbled upon through convergence, and is very similar to the architectural organization of the social insects like ants and bees. For more on that see Cloud Computing and the Coming Software Mass Extinction.

As I explained in The Fundamental Problem of Software and the softwarephysics postings leading up to it, the fundamental problem with developing and maintaining software is the effects of the second law of thermodynamics acting upon software in a nonlinear Universe. The second law of thermodynamics is constantly trying to reduce the total amount of useful information in the Universe when we code software, creating small bugs in software whenever we work on it, and because the Universe is largely nonlinear in nature, these small bugs can produce huge problems immediately, or may even initially lay dormant, but later cause problems in the distant future. So the question is, are there any complex information processing systems in the physical Universe that deal well with both the second law of thermodynamics and nonlinearity? The answer to this question is of course, yes – living things do a marvelous job in this perilous Universe with contending with both the second law of thermodynamics and nonlinearity. Just as every programmer must assemble characters into lines of code, living things must assemble atoms into complex organic molecules in order to perform the functions of life, and because the physical Universe is largely nonlinear, small errors in these organic molecules can have disastrous effects. And this all has to be done in a Universe bent on degenerating into a state of maximum entropy and minimum useful information content thanks to our old friend the second law of thermodynamics. So it makes sense from an IT perspective to adopt these very successful biological techniques when developing, maintaining, and operating software.

Living things do not use the Waterfall methodology to produce the most complex things we know of in the Universe. Instead, they use Agile techniques. Species evolve over time and gain additional functionality through small incremental changes honed by natural selection, and all complex multicellular organisms come to maturity from a single fertilized egg through the small incremental changes of embryogenesis.

An Agile Biological Approach to Software
So how do you apply an Agile biological approach to software? In How to Think Like a Softwarephysicist I provided a general framework on how to do so. But I also have a very good case study of how I did so in the past. For example, in SoftwarePhysics I described how I started working on BSDE - the Bionic Systems Development Environment back in 1985 while in the IT department of Amoco. BSDE was an early mainframe-based IDE (Integrated Development Environment like Eclipse) at a time when there were no IDEs. During the 1980s BSDE was used to grow several million lines of production code for Amoco by growing applications from embryos. For an introduction to embryology see Software Embryogenesis. The DDL statements used to create the DB2 tables and indexes for an application were stored in a sequential file called the Control File and performed the functions of genes strung out along a chromosome. Applications were grown within BSDE by turning their genes on and off to generate code. BSDE was first used to generate a Control File for an application by allowing the programmer to create an Entity-Relationship diagram using line printer graphics on an old IBM 3278 terminal.

Figure 2 - BSDE was run on IBM 3278 terminals, using line printer graphics, and in a split-screen mode. The embryo under development grew within BSDE on the top half of the screen, while the code generating functions of BSDE were used on the lower half of the screen to insert code into the embryo and to do compiles on the fly while the embryo ran on the upper half of the screen. Programmers could easily flip from one session to the other by pressing a PF key.

After the Entity-Relationship diagram was created, the programmer used a BSDE option to create a skeleton Control File with DDL statements for each table on the Entity-Relationship diagram and each skeleton table had several sample columns with the syntax for various DB2 datatypes. The programmer then filled in the details for each DB2 table. When the first rendition of the Control File was completed, another BSDE option was used to create the DB2 database for the tables and indexes on the Control File. Another BSDE option was used to load up the DB2 tables with test data from sequential files. Each DB2 table on the Control File was considered to be a gene. Next a BSDE option was run to generate an embryo application. The embryo was a 10,000 line of code PL/1, Cobol or REXX application that performed all of the primitive functions of the new application. The programmer then began to grow the embryo inside of BSDE in a split-screen mode. The embryo ran on the upper half of an IBM 3278 terminal and could be viewed in real time, while the code generating options of BSDE ran on the lower half of the IBM 3278 terminal. BSDE was then used to inject new code into the embryo's programs by reading the genes in the Control File for the embryo in a real time manner while the embryo was running in the top half of the IBM 3278 screen. BSDE had options to compile and link modified code on the fly while the embryo was still executing. This allowed for a tight feedback loop between the programmer and the application under development. In fact, many times BSDE programmers sat with end-users and co-developed software together on the fly in a very Agile manner. When the embryo had grown to full maturity, BSDE was then used to create online documentation for the new application, and was also used to automate the install of the new application into production. Once in production, BSDE generated applications were maintained by adding additional functions to their embryos.

Since BSDE was written using the same kinds of software that it generated, I was able to use BSDE to generate code for itself. The next generation of BSDE was grown inside of its maternal release. Over a period of seven years, from 1985 – 1992, more than 1,000 generations of BSDE were generated, and BSDE slowly evolved in an Agile manner into a very sophisticated tool through small incremental changes. BSDE dramatically improved programmer efficiency by greatly reducing the number of buttons programmers had to push in order to generate software that worked.

Figure 3 - Embryos were grown within BSDE in a split-screen mode by transcribing and translating the information stored in the genes in the Control File for the embryo. Each embryo started out very much the same, but then differentiated into a unique application based upon its unique set of genes.

Figure 4 – BSDE appeared as the cover story of the October 1991 issue of the Enterprise Systems Journal

BSDE had its own online documentation that was generated by BSDE. Amoco's IT department also had a class to teach programmers how to get started with BSDE. As part of the curriculum Amoco had me prepare a little cookbook on how to build an application using BSDE:

BSDE – A 1989 document describing how to use BSDE - the Bionic Systems Development Environment - to grow applications from genes and embryos within the maternal BSDE software.

I wish that I could claim that I was smart enough to have sat down and thought up all of this stuff from first principles, but that is not what happened. It all just happened through small incremental changes in a very Agile manner over a very long period of time and most of the design work was done subconsciously, if at all. Even the initial BSDE ISPF edit macros happened through serendipity. When I first started programming DB2 applications, I found myself copying in the DDL CREATE TABLE statements from the file I used to create the DB2 database into the program that I was working on. This file, with the CREATE TABLE statements, later became the Control File used by BSDE to store the genes for an application. I would then go through a series of editing steps on the copied in data to transform it from a CREATE TABLE statement into a DB2 SELECT, INSERT, UPDATE, or DELETE statement. I would do the same thing all over again to declare the host variables for the program. Being a lazy programmer, I realized that there was really no thinking involved in these editing steps and that an ISPF edit macro could do the job equally as well, only very quickly and without error, so I went ahead and wrote a couple of ISPF edit macros to automate the process. I still remember the moment when it first hit me. For me it was very much like the scene in 2001 - A Space Odyssey, when the man-ape picks up a wildebeest thighbone and starts to pound the ground with it. My ISPF edit macros were doing the same thing that happens when the information in a DNA gene is transcribed into a protein! A flood of biological ideas poured into my head over the next few days, because at last I had a solution for my pent-up ideas about nonlinear systems and the second law of thermodynamics that were making my life so difficult as a commercial software developer. We needed to "grow" code – not write code!

BSDE began as a few simple ISPF edit macros running under ISPF edit. ISPF is the software tool that mainframe programmers still use today to interface to the IBM MVS and VM/CMS mainframe operating systems and contains an editor that can be greatly enhanced through the creation of edit macros written in REXX. I began BSDE by writing a handful of ISPF edit macros that could automate some of the editing tasks that a programmer needed to do when working on a program that used a DB2 database. These edit macros would read a Control File, which contained the DDL statements to create the DB2 tables and indexes. The CREATE TABLE statements in the Control File were the equivalent of genes, and the Control File itself performed the functions of a chromosome. For example, a programmer would retrieve a skeleton COBOL program, with the bare essentials for a COBOL/DB2 program, from a stock of reusable BSDE programs. The programmer would then position their cursor in the code to generate a DB2 SELECT statement and hit a PFKEY. The REXX edit macro would read the genes in the Control File and would display a screen listing all of the DB2 tables for the application. The programmer would then select the desired tables from the screen, and the REXX edit macro would then copy the selected genes to an array (mRNA). The mRNA array was then sent to a subroutine that inserted lines of code (tRNA) into the COBOL program. The REXX edit macro would also declare all of the SQL host variables in the DATA DIVISION of the COBOL program and would generate code to check the SQLCODE returned from DB2 for errors and take appropriate actions. A similar REXX ISPF edit macro was used to generate screens. These edit macros were also able to handle PL/1 and REXX/SQL programs. They could have been altered to generate the syntax for any programming language such as C, C++, or Java. As time progressed, BSDE took on more and more functionality via ISPF edit macros. Finally, there came a point where BSDE took over and ISPF began to run under BSDE. This event was very similar to the emergence of the eukaryotic architecture for cellular organisms. BSDE consumed ISPF like the first eukaryotic cells that consumed prokaryotic bacteria and used them as mitochondria and chloroplasts. With continued small incremental changes, BSDE continued to evolve.

I noticed that I kept writing the same kinds of DB2 applications, with the same basic body plan, over and over. At the time I did not know it, but these were primarily Model-View-Controller (MVC) applications. For more on the MVC design pattern please see Software Embryogenesis. From embryology, I got the idea of using BSDE to read the Control File for an application and to generate an "embryo" for the application based upon its unique set of genes. The embryo would perform all of the things I routinely programmed over and over for a new application. Once the embryo was generated for a new application from its Control File, the programmer would then interactively "grow" code and screens for the application. With time, each embryo differentiated into a unique individual application in an Agile manner until the fully matured application was delivered into production by BSDE. At this point, I realized that I could use BSDE to generate code for itself, and that is when I started using BSDE to generate the next generation of BSDE. This technique really sped up the evolution of BSDE because I had a positive feedback loop going. The more powerful BSDE became, the faster I could add improvements to the next generation of BSDE through the accumulated functionality inherited from previous generations.

Embryos were grown within BSDE using an ISPF split screen mode. The programmer would start up a BSDE session and run Option 4 – Interactive Systems Development from the BSDE Master Menu. This option would look for an embryo, and if it did not find one, would offer to generate an embryo for the programmer. Once an embryo was implanted, the option would turn the embryo on and the embryo would run inside of the BSDE session with whatever functionality it currently had. The programmer would then split his screen with PF2 and another BSDE session would appear in the lower half of his terminal. The programmer could easily toggle control back and forth between the upper and lower sessions with PF9. The lower session of BSDE was used to generate code and screens for the embryo on the fly, while the embryo in the upper BSDE session was fully alive and functional. This was possible because BSDE generated applications that used ISPF Dialog Manager for screen navigation, which was an interpretive environment, so compiles were not required for screen changes. If your logic was coded in REXX, you did not have to do compiles for logic changes either, because REXX was an interpretive language. If PL/1 or COBOL were used for logic, BSDE had facilities to easily compile code for individual programs after a coding change, and ISPF Dialog Manger would simply load the new program executable when that part of the embryo was exercised. These techniques provided a tight feedback loop so that programmers and end-users could immediately see the effects of a change as the embryo grew and differentiated.

But to fully understand how BSDE was used to grow embryos through small incremental changes into fully mature applications, we need to turn to how that is accomplished in the biosphere. For example, many people find it hard to believe that all human beings evolved from a single primitive cell in the very distant past, yet we all managed to do that and in only about 9 months of gestation! Indeed, it does seem like a data processing miracle that a complex human body can grow and differentiate into 100 trillion cells, composed of several hundred different cell types that ultimately form the myriad varied tissues within a body, from a single fertilized egg cell. In biology the study of this incredible feat is called embryogenesis, or developmental biology, and this truly amazing process from a data processing perspective is certainly worthy of emulating when developing software. So let's spend some time seeing how that is done.

Human Embryogenesis
Most multicellular organisms follow a surprisingly similar sequence of steps to form a complex body, composed of billions or trillions of eukaryotic cells, from a single fertilized egg. This is a sure sign of some inherited code at work that has been tweaked many times to produce a multitude of complex body plans or phyla by similar developmental processes. Since many multicellular life forms follow a similar developmental theme let us focus, as always, upon ourselves and use the development of human beings as our prime example of how developmental biology works. For IT professionals and other readers not familiar with embryogenesis, it would be best now to view this short video before proceeding:

Medical Embryology - Difficult Concepts of Early Development Explained Simply

Basically, a fertilized egg, or zygote, begins to divide many times over, without the zygote really increasing in size at all. After a number of divisions, the zygote becomes a ball of undifferentiated cells that are all the same and is known as a morula. The morula then develops an interior hollow center called a blastocoel. The hollow ball of cells is known as a blastula and all the cells in the blastula are undifferentiated, meaning that they are all still identical in nature.

Figure 5 – A fertilized egg, or zygote, divides many times over to form a solid sphere of cells called a morula. The morula then develops a central hole to become a hollow ball of cells known as a blastula. The blastula consists of identical cells. When gastrulation begins some cells within the blastula begin to form three layers of differentiated cells – the ectoderm, mesoderm, and endoderm. The above figure does not show the amnion, which forms, just outside of the infolded cells that create the gastrula. See Figure 6 for the location of the amnion.

The next step is called gastrulation. In gastrulation one side of the blastula breaks symmetry and folds into itself and eventually forms three differentiated layers – the ectoderm, mesoderm and endoderm. The amnion forms just outside of the gastrulation infold.

Figure 6 – In gastrulation three layers of differentiated cells form - the ectoderm, mesoderm and endoderm by cells infolding and differentiating.

Figure 7 – Above is a close up view showing the ectoderm, mesoderm and endoderm forming from the primitive streak.

The cells of the endoderm go on to differentiate into the internal organs or guts of a human being. The cells of the mesoderm, or the middle layer, go on to form the muscles and connective tissues that do most of the heavy lifting. Finally, the cells of the ectoderm go on to differentiate into the external portions of the human body, like the skin and nerves.

Figure 8 – Some examples of the cell types that develop from the endoderm, mesoderm and ectoderm.

Figure 9 – A human being develops from the cells in the ectoderm, mesoderm and endoderm as they differentiate into several hundred different cell types.

Figure 10 – A human embryo develops through small incremental changes into a fully mature newborn in an Agile manner. Living things, the most complex data processing machines in the Universe, never use the Waterfall methodology to build new bodies or to evolve over time into new species. Living things always develop new functionality through small incremental changes in an Agile manner.

Growing Embryos in an Agile Manner
At the time IBM had a methodology called JAD - Joint Application Development. In a JAD the programmers and end-users met together in a room and roughed out the design for a new application on flip charts in an interactive manner. The flip chart papers were then taped to the walls for all to see. At the end of the day, the flip charts were gathered by an attending systems analyst who then attempted to put together the JAD flip charts into a User Requirements document that was then carried into the normal Waterfall development process. This seemed rather cumbersome, so with BSDE we tried to do some JAD work in an interactive manner with end-users by growing Model-View-Controller (MVC) embryos together. In an MVC application, the data Model was stored on DB2 tables, forming the "guts" or endoderm tissues of the application. The View code consisted of ISPF screens and reports that could be viewed by the end-user, forming the ectoderm tissues of the application. Finally, the Model and View were connected together by the Controller code that did most of the heavy lifting, forming the mesoderm tissues of the application. Controller code was by far the most difficult to code and was the most expensive part of a project. So as in human gastrulation, we grew embryos from the outside-in and the inside-out, by finally having the external ectoderm tissues meet the internal endoderm tissues in the middle at the mesoderm tissues of the Controller code. Basically, we would first start with some data modeling with the end-user to rough out the genes in the Control File chromosome. Then we had BSDE generate an MVC embryo for the application based upon its genes in the Control File. Next we roughed out the ectoderm ISPF screens for the application, using BSDE to generate the ISPF screens and field layouts on each ISPF screen. We then used BSDE to generate REXX, Cobol or PL/1 programs to navigate the flow of the ISPF screens. So we worked on the endoderm and ectoderm with the end-users first since they were very easy to change and they determined the look and feel of the application. By letting end-users first interact with an embryo that consisted of just the end-user ectoderm tissues, forming the user interface, and the underlying DB2 tables of the endoderm it was possible to rough out an application with no user requirements documentation at all in a very Agile manner. Of course as end-users interacted with the embryo, many times we discovered that new data elements were needed to be added to our DB2 model stored on the Control File, so we would just add the new columns to the genes on the Control File and then just regenerate the impacted ISPF screens from the modified genes. Next we added validation code to the embryo to validate the input fields on the ISPF screens, using BSDE templates. BSDE came with a set of templates of reusable code that could be imported for this purpose. This was before the object-oriented programming revolution in IT, so the BSDE reusable code templates performed the same function as the class library of an object-oriented programming language. Finally, mockup reports were generated by BSDE. BSDE was used to create a report mockup file that looked like what the end-user wanted to see in an online or line printer report. BSDE then took the report mockup file and generated a Cobol or PL/1 program that produced the report output to give the end-user a good feel for how the final report would actually look. At this point end-users could fully navigate the embryo and interact with it in real time. They could pretend to enter data into the embryo as they navigated the embryo's ISPF screens, and they could see the embryo generate reports. Once the end-user was happy with the embryo, we would start to use ISPF to generate code for the mesoderm Controller tissues by having BSDE create SELECT, INSERT, UPDATE and DELETE SQL statements in the embryonic programs. We also used the BSDE reusable code templates for processing logic in the mesoderm Controller code that connected the ectoderm ISPF screens to the endoderm DB2 database.

I Think, Therefore I am - a Heretic
BSDE was originally developed for my own use, but fellow programmers soon took note, and over time, an underground movement of BSDE programmers developed at Amoco, and so I began to market BSDE and softwarephysics within Amoco. Being a rather conservative oil company, this was a hard sell, and I am not much of a salesman. By now it was late 1986, and I was calling their programmers "software engineers" and was pushing the unconventional ideas of applying physics and biology to software to grow software in an Agile manner. These were very heretical ideas at the time because of the dominance of the Waterfall methodology. BSDE was generating real applications with practically no upfront user requirements or specification documents to code from. Fortunately for me, all of Amoco's income came from a direct application of geology, physics, or chemistry, so many of our business partners were geologists, geophysicists, chemists, or chemical, petroleum, electrical, or industrial engineers, and that was a big help. Computer viruses began to appear about this time, and they provided some independent corroboration for the idea of applying biological concepts to software, and in 1987 we also started to hear some things about artificial life from Chris Langton out of Los Alamos, but there still was a lot of resistance to the idea back at Amoco. One manager insisted that I call genes "templates". I used the term "bionic" in the product name after I checked in the dictionary and found that bionic meant applying concepts from biology to solve engineering problems, and that was exactly what I was trying to achieve. There were also a couple of American television programs in the 1970s that introduced the term to the American public, The Six Million Dollar Man and The Bionic Woman, which featured superhuman characters performing astounding feats. In my BSDE road shows, I would demo BSDE in action spewing out perfect code in real time and about 20 times faster than normal human programmers could achieve, so I thought the term "bionic" was fitting. I am still not sure that using the term "bionic" was the right thing to do. IT was not "cool" in the 1980s, as it is in today’s ubiquitous Internet Age, and was dominated by very serious and conservative people with backgrounds primarily in accounting. However, BSDE was pretty successful and by 1989, BSDE was finally recognized by Amoco IT management and was made available to all Amoco programmers. A BSDE class was developed and about 20 programmers became active BSDE programmers. A BSDE COI (Community of Interest) was formed, and I used to send out weekly email newsletters about applying physics and biology to software to the COI members.

So that is how BSDE got started by accident. Since I did not have any budget for BSDE development, and we charged out all of our time at Amoco to various projects, I was forced to develop BSDE through small incremental enhancements in an Agile manner that always made my job on the billable projects a little easier. I new that was how evolution worked, so I was not too concerned. In retrospect, this was a fortunate thing. In the 1980s, the accepted theory of the day was that you needed to prepare lots of user requirement and specifications documents up front, following the Waterfall methodology, for a new application, and then you would code from the documents as a blueprint. This was before the days of Agile development through small incremental releases that you find today. In the mid 1980s, prototyping was a very heretical proposition in IT. But I don’t think that I could have built BSDE using the traditional blueprint Waterfall methodology of the day.

As you can see, having a sound theoretical framework, like softwarephysics, helps when your are faced with making an important IT decision, like whether to use the Agile or Waterfall methodology. Otherwise, IT decisions are left to the political whims of the times, with no sound basis.

Comments are welcome at

To see all posts on softwarephysics in reverse order go to:

Steve Johnston