Wednesday, September 02, 2015

Introduction to Softwarephysics

Softwarephysics is a simulated science for the simulated Software Universe that we are all immersed in. It is an approach to software development, maintenance, and support based upon concepts from physics, chemistry, biology, and geology that I have been using on a daily basis for over 35 years as an IT professional. For those of you not in the business, IT is short for Information Technology, commercial computer science. The purpose of softwarephysics is to explain why IT is so difficult, to suggest possible remedies, and to provide a direction for thought. If you are an IT professional, general computer user, or simply an individual interested in computer science, physics, chemistry, biology, or geology then softwarephysics might be of interest to you, if not in an entirely serious manner, perhaps at least in an entertaining one.

From 1975 – 1979, I was an exploration geophysicist exploring for oil, first with Shell, and then with Amoco. In 1979, I made a career change into IT, and spent about 20 years in development. For the past 14 years, I have been in IT operations, supporting middleware on WebSphere, JBoss, Tomcat, and ColdFusion. When I transitioned into IT from geophysics, I figured if you could apply physics to geology; why not apply physics to software? So like the exploration team at Amoco that I had just left, consisting of geologists, geophysicists, paleontologists, geochemists, and petrophysicists, I decided to take all the physics, chemistry, biology, and geology that I could muster and throw it at the problem of software. The basic idea was that many concepts in physics, chemistry, biology, and geology suggested to me that the IT community had accidentally created a pretty decent computer simulation of the physical Universe on a grand scale, a Software Universe so to speak, and that I could use this fantastic simulation in reverse, to better understand the behavior of commercial software, by comparing software to how things behaved in the physical Universe. Softwarephysics depicts software as a virtual substance, and relies upon our understanding of the current theories in physics, chemistry, biology, and geology to help us model the nature of software behavior. So in physics we use software to simulate the behavior of the Universe, while in softwarephysics we use the Universe to simulate the behavior of software. Along these lines, we use the Equivalence Conjecture of Softwarephysics as an aid; it allows us to shift back and forth between the Software Universe and the physical Universe, and hopefully to learn something about one by examining the other:

The Equivalence Conjecture of Softwarephysics
Over the past 70 years, through the uncoordinated efforts of over 50 million independently acting programmers to provide the world with a global supply of software, the IT community has accidentally spent more than $10 trillion creating a computer simulation of the physical Universe on a grand scale – the Software Universe.

Logical Positivism and Effective Theories
Many IT professionals have a difficult time with softwarephysics because they think of physics as being limited to the study of real things like electrons and photons, and since software is not “real”, how can you possibly apply concepts from physics and the other sciences to software? To address this issue, softwarephysics draws heavily upon two concepts from physics that have served physics quite well over the past century – the concept of logical positivism and the concept of effective theories. This was not always the case. In the 17th, 18th, and 19th centuries, physicists mistakenly thought that they were actually discovering the fundamental laws of the Universe, which they thought were based upon real tangible things like particles, waves, and fields. Classical Newtonian mechanics (1687), thermodynamics (1850), and classical electrodynamics (1864) did a wonderful job of describing the everyday world at the close of the 19th century, but early in the 20th century it became apparent that the models upon which these very successful theories were based did not work very well for small things like atoms or for objects moving at high velocities or in strong gravitational fields. This provoked a rather profound philosophical crisis within physics at the turn of the century, as physicists worried that perhaps 300 years of work was about to go down the drain. The problem was that classical physicists confused their models of reality with reality itself, and when their classical models began to falter, their confidence in physics began to falter as well. This philosophical crisis was resolved with the adoption of the concepts of logical positivism and some new effective theories in physics. Quantum mechanics (1926) was developed for small things like atoms, the special theory of relativity (1905) was developed for objects moving at high velocities and the general theory of relativity (1915) was developed for objects moving in strong gravitational fields.

Logical positivism, usually abbreviated simply to positivism, is an enhanced form of empiricism, in which we do not care about how things “really” are; we are only interested with how things are observed to behave. With positivism, physicists only seek out models of reality - not reality itself. When we study quantum mechanics, we will find that the concept of reality gets rather murky in physics anyway, so this is not as great a loss as it might at first seem. By concentrating on how things are observed to behave, rather than on what things “really” are, we avoid the conundrum faced by the classical physicists. In retrospect, this idea really goes all the way back to the very foundations of physics. In Newton’s Principia (1687) he outlined Newtonian mechanics and his theory of gravitation, which held that the gravitational force between two objects was proportional to the product of their masses divided by the square of the distance between them. Newton knew that he was going to take some philosophical flack for proposing a mysterious force between objects that could reach out across the vast depths of space with no apparent mechanism, so he took a very positivistic position on the matter with the famous words:

I have not as yet been able to discover the reason for these properties of gravity from phenomena, and I do not feign hypotheses. For whatever is not deduced from the phenomena must be called a hypothesis; and hypotheses, whether metaphysical or physical, or based on occult qualities, or mechanical, have no place in experimental philosophy. In this philosophy particular propositions are inferred from the phenomena, and afterwards rendered general by induction.

Instead, Newton focused on how things were observed to move under the influence of his law of gravitational attraction, without worrying about what gravity “really” was.

The second concept, that of effective theories, is an extension of positivism. An effective theory is an approximation of reality that only holds true over a certain restricted range of conditions and only provides for a certain depth of understanding of the problem at hand. For example, Newtonian mechanics is an effective theory that makes very good predictions for the behavior of objects moving less than 10% of the speed of light and which are bigger than a very small grain of dust. These limits define the effective range over which Newtonian mechanics can be applied to solve problems. For very small things we must use quantum mechanics and for very fast things moving in strong gravitational fields, we must use relativity theory. So all of the current theories of physics, such as Newtonian mechanics, classical electrodynamics, thermodynamics, statistical mechanics, the special and general theories of relativity, quantum mechanics, and the quantum field theories of QED and QCD are effective theories that are based upon models of reality, and all these models are approximations - all these models are fundamentally "wrong", but at the same time, these effective theories make exceedingly good predictions of the behavior of physical systems over the limited ranges in which they apply. That is the goal of softwarephysics – to provide for an effective theory of software behavior that makes useful predictions of software behavior that are applicable to the day-to-day activities of IT professionals. So in softwarephysics, we adopt a very positivistic viewpoint of software; we do not care what software “really is”, we only care about how software is observed to behave and try to model those behaviors with an effective theory of software behavior that only holds true over a certain restricted range of conditions and only provides for a certain depth of understanding of the problem at hand.

GPS satellites provide a very good example of positivism and effective theories at work. There are currently 31 GPS satellites orbiting at an altitude of 12,600 miles above the Earth, and each contains a very accurate atomic clock. The signals from the GPS satellites travel to your GPS unit at the speed of light, so by knowing the travel time of the signals from at least 4 of the GPS satellites, it is possible to determine your position on Earth very accurately. In order to do that, it is very important to have very accurate timing measurements. Newtonian mechanics is used to launch the GPS satellites to an altitude of 12,600 miles and to keep them properly positioned in orbit. Classical electrodynamics is then used to beam the GPS signals back down to Earth to the GPS unit in your car. Quantum mechanics is used to build the transistors on the chips onboard the GPS satellites and to understand the quantum tunneling of electrons in the flash memory chips used to store GPS data on the satellites. The special theory of relativity predicts that the onboard atomic clocks on the GPS satellites will run slower and lose about 7.2 microseconds per day due to their high velocities relative to an observer on the Earth. But at the same time, the general theory of relativity also predicts that because the GPS satellites are further from the center of the Earth and in a weaker gravitational field, where spacetime is less deformed than on the surface of the Earth, their atomic clocks also run faster and gain 45.9 microseconds per day due to the weaker gravitational field out there. The net effect is a gain of 38.7 microseconds per day, so the GPS satellite atomic clocks have to be purposefully built to run slow by 38.7 microseconds per day before they are launched, so that they will keep in sync with clocks on the surface of the Earth. If this correction were not made, an error in your position of 100 yards/day would accrue. The end result of the combination of all these fundamentally flawed effective theories is that it is possible to pinpoint your location on Earth to an accuracy of 16 feet or better for as little as $100. But physics has done even better than that with its fundamentally flawed effective theories. By combining the effective theories of special relativity (1905) with quantum mechanics (1926), physicists were able to produce a new effective theory for the behavior of electrons and photons called quantum electrodynamics QED (1948) which was able to predict the gyromagnetic ratio of the electron, a measure of its intrinsic magnetic field, to an accuracy of 11 decimal places. As Richard Feynman has pointed out, this was like predicting the exact distance between New York and Los Angeles accurate to the width of a human hair!

So Newtonian mechanics makes great predictions for the macroscopic behavior of GPS satellites, but it does not work very well for small things like the behavior of individual electrons within transistors, where quantum mechanics is required, or for things moving at high speeds or in strong gravitational fields where relativity theory must be applied. And all three of these effective theories are based upon completely contradictory models. General relativity maintains that spacetime is curved by matter and energy, but that matter and energy are continuous, while quantum mechanics maintains that spacetime is flat, but that matter and energy are quantized into chunks. Newtonian mechanics simply states that space and time are mutually independent dimensions and universal for all, with matter and energy being continuous. The important point is that all effective theories and scientific models are approximations – they are all fundamentally "wrong". But knowing that you are "wrong" gives you a great advantage over people who know that they are "right", because knowing that you are "wrong" allows you to seek improved models of reality. So please consider softwarephysics to simply be an effective theory of software behavior that is based upon models that are fundamentally “wrong”, but at the same time, fundamentally useful for IT professionals. So as you embark upon your study of softwarephysics, please always keep in mind that the models of softwarephysics are just approximations of software behavior, they are not what software “really is”. It is very important not to confuse models of software behavior with software itself, if one wishes to avoid the plight of the 19th century classical physicists.

If you are an IT professional and many of the above concepts are new to you, do not be concerned. This blog on softwarephysics is aimed at a diverse audience, but with IT professionals in mind. All of the above ideas will be covered at great length in the postings in this blog on softwarephysics and in a manner accessible to all IT professionals. Now it turns out that most IT professionals have had some introduction to physics in high school or in introductory college courses, but that presents an additional problem. The problem is that such courses generally only cover classical physics, and leave the student with a very good picture of physics as it stood in 1864! It turns out that the classical physics of Newtonian mechanics, thermodynamics, and classical electromagnetic theory were simply too good to discard and are still quite useful, so they are taught first to beginners and then we run out of time to cover the really interesting physics of the 20th century. Now imagine the problems that the modern world would face if we only taught similarly antiquated courses in astronomy, metallurgy, electrical and mechanical engineering, medicine, economics, biology, or geology that happily left students back in 1864! Since many of the best models for software behavior stem from 20th century physics, we will be covering a great deal of 20th century material in these postings – the special and general theories of relativity, quantum mechanics, quantum field theories, and chaos theory, but I hope that you will find that these additional effective theories are quite interesting on their own, and might even change your worldview of the physical Universe at the same time.

Unintended Consequences for the Scientific Community
As I mentioned at the close of my original posting on SoftwarePhysics, my initial intention for this blog on softwarephysics was to fulfill a promise I made to myself about 20 years ago to approach the IT community with the concept of softwarephysics a second time, following my less than successful attempt to do so in the 1980s, with the hope of helping the IT community to better cope with the daily mayhem of life in IT. However, in laying down the postings for this blog an unintended consequence arose in my mind as I became profoundly aware of the enormity of this vast computer simulation of the physical Universe that the IT community has so graciously provided to the scientific community free of charge and also of the very significant potential scientific value that it provides. One of the nagging problems for many of the observational and experimental sciences is that many times there is only one example readily at hand to study or experiment with, and it is very difficult to do meaningful statistics with a population of N=1.

But the computer simulation of the physical Universe that the Software Universe presents provides another realm for comparison. For example, both biology and astrobiology only have one biosphere on Earth to study and even physics itself has only one Universe with which to engage. Imagine the possibilities if scientists had another Universe readily at hand in which to work! This is exactly what the Software Universe provides. For example, in SoftwareBiology and A Proposal For All Practicing Paleontologists we see that the evolution of software over the past 70 years has closely followed the same path as life on Earth over the past 4.0 billion years, in keeping with Simon Conway Morris’s contention that convergence has played the dominant role in the evolution of life on Earth. In When Toasters Fly, we also see that software has evolved in fits and starts as portrayed by the punctuated equilibrium of Stephen Jay Gould and Niles Eldredge, and in The Adaptationist View of Software Evolution we explore the overwhelming power of natural selection in the evolution of software. In keeping with Peter Ward’s emphasis on mass extinctions dominating the course of evolution throughout geological time, we also see in SoftwareBiology that there have been several dramatic mass extinctions of various forms of software over the past 70 years as well, that have greatly affected the evolutionary history of software, and that between these mass extinctions, software has also tended to evolve through the gradual changes of Hutton’s and Lyell’s uniformitarianism. In Software Symbiogenesis and Self-Replicating Information, we also see the very significant role that parasitic/symbiotic relationships have played in the evolution of software, in keeping with the work of Lynn Margulis and also of Freeman Dyson’s two-stage theory of the origin of life on Earth. In The Origin of Software the Origin of Life, we explore Stuart Kauffman’s ideas on how Boolean nets of autocatalytic chemical reactions might have kick-started the whole thing as an emergent behavior of an early chaotic pre-biotic environment on Earth, and that if Seth Shostak is right, we will never end up talking to carbon-based extraterrestrial aliens, but to alien software instead. In Is the Universe Fine-Tuned for Self-Replicating Information? we explore the thermodynamics of Brandon Carter’s Weak Anthropic Principle (1973), as it relates to the generation of universes in the multiverse that are capable of sustaining intelligent life. Finally, in Programming Clay we revisit Alexander Graham Cairns-Smith’s theory (1966) that Gene 1.0 did not run on nucleic acids, but on clay microcrystal precursors instead.

Similarly for the physical sciences, in Is the Universe a Quantum Computer? we find a correspondence between TCP/IP and John Cramer’s Transactional Interpretation of quantum mechanics. In SoftwarePhysics and Cyberspacetime, we also see that the froth of CPU processes running with a clock speed of 109 Hz on the 10 trillion currently active microprocessors that comprise the Software Universe can be viewed as a slowed down simulation of the spin-foam froth of interacting processes of loop quantum gravity running with a clock speed of 1043 Hz that may comprise the physical Universe. And in Software Chaos, we examine the nonlinear behavior of software and some of its emergent behaviors and follow up in CyberCosmology with the possibility that vast quantities of software running on large nonlinear networks might eventually break out into consciousness in accordance with the work of George Dyson and Daniel Dennett. Finally, in Model-Dependent Realism - A Positivistic Approach to Realism we compare Steven Weinberg’s realism with the model-dependent realism of Stephen Hawking and Leonard Mlodinow and how the two worldviews affect the search for a Final Theory. Finally, in The Software Universe as an Implementation of the Mathematical Universe Hypothesis and An Alternative Model of the Software Universe we at long last explore what software might really be, and discover that the Software Universe might actually be more closely related to the physical Universe than you might think.

The chief advantage of doing fieldwork in the Software Universe is that, unlike most computer simulations of the physical Universe, it is an unintended and accidental simulation, without any of the built-in biases that most computer simulations of the physical Universe suffer. So you will truly be able to do fieldwork in a pristine and naturally occuring simulation, just as IT professionals can do fieldwork in the wild and naturally occuring simulation of software that the living things of the biosphere provide. Secondly, the Software Universe is a huge simulation that is far beyond the budgetary means of any institution or consortium by many orders of magnitude. So if you are an evolutionary biologist, astrobiologist, or paleontologist working on the origin and evolution of life in the Universe, or a physicist or economist working on the emergent behaviors of nonlinear systems and complexity theory, or a neurobiologist working on the emergence of consciousness in neural networks, or even a frustrated string theorist struggling with quantum gravity, it would be well worth your while to pay a friendly call upon the local IT department of a major corporation in your area. Start with a visit to the Command Center for their IT Operations department to get a global view of their IT infrastructure and to see how it might be of assistance to the work in your area of interest. From there you can branch out to the applicable area of IT that will provide the most benefit.

The Impact of Self-Replicating Information Upon the Planet
One of the key findings of softwarephysics is concerned with the magnitude of the impact upon the planet of self-replicating information.

Self-Replicating Information – Information that persists through time by making copies of itself or by enlisting the support of other things to ensure that copies of itself are made.

Basically, we have seen several waves of self-replicating information dominate the Earth:
1. Self-replicating autocatalytic metabolic pathways of organic molecules
2. RNA
3. DNA
4. Memes
5. Software

Note that because the self-replicating autocatalytic metabolic pathways of organic molecules, RNA and DNA have become so heavily intertwined over time that I now simply refer to them as the “genes”. Over the past 4.0 billion years, the surface of the Earth has been totally reworked by three forms of self-replicating information – the genes, memes and software, with software rapidly becoming the dominant form of self-replicating information on the planet. For more on this see:

A Brief History of Self-Replicating Information
How to Use an Understanding of Self-Replicating Information to Avoid War
How to Use Softwarephysics to Revive Memetics in Academia
Is Self-Replicating Information Inherently Self-Destructive?
Is the Universe Fine-Tuned for Self-Replicating Information?
Self-Replicating Information

Softwarephysics and the Real World of Human Affairs
Having another universe readily at hand to explore, even a simulated universe like the Software Universe, necessarily has an impact upon one's personal philosophy of life, and allows one to draw certain conclusions about the human condition and what’s it all about, so as you read through the postings in this blog you will stumble across a bit of my own personal philosophy - definitely a working hypothesis still in the works. Along these lines you might be interested in a few postings where I try to apply softwarephysics to the real world of human affairs:

MoneyPhysics – my impression of the 2008 world financial meltdown.

The Fundamental Problem of Everything – If you Google "the fundamental problem of everything", this will be the only hit you get on the entire Internet, which is indicative of the fundamental problem of everything!

What’s It All About? and Genes, Memes and Software – my current working hypothesis on what’s it all about.

How to Use an Understanding of Self-Replicating Information to Avoid War – my current working hypothesis for how the United States can avoid getting bogged down again in continued war in the Middle East.

Hierarchiology and the Phenomenon of Self-Organizing Organizational Collapse - a modern extension of the classic Peter Principle that applies to all hierarchical organizations and introduces the Time Invariant Peter Principle.

Some Specifics About These Postings
The postings in this blog are a supplemental reading for my course on softwarephysics for IT professionals entitled SoftwarePhysics 101 – The Physics of Cyberspacetime, which was originally designed to be taught as a series of seminars at companies where I was employed. Since softwarephysics essentially covers the simulated physics, chemistry, biology, and geology of an entire simulated universe, the slides necessarily just provide a cursory skeleton upon which to expound. The postings in this blog go into much greater depth. Because each posting builds upon its predecessors, the postings in this blog should be read in reverse order from the oldest to the most recent, beginning with my original posting on SoftwarePhysics. In addition, several universities also now offer courses on Biologically Inspired Computing which cover some of the biological aspects of softwarephysics, and the online content for some of these courses can be found by Googling for "Biologically Inspired Computing" or "Natural Computing". At this point we will finish up with my original plan for this blog on softwarephysics with a purely speculative posting on CyberCosmology that describes the origins of the Software Universe, cyberspacetime, software and where they all may be heading. Since CyberCosmology will be purely speculative in nature, it will not be of much help to you in your IT professional capacities, but I hope that it might be a bit entertaining. Again, if you are new to softwarephysics, you really need to read the previous posts before taking on CyberCosmology. I will probably continue on with some additional brief observations about softwarephysics in the future, but once you have completed CyberCosmology, you can truly consider yourself to be a bona fide softwarephysicist.

For those of you following this blog, the posting dates on the posts may seem to behave in a rather bizarre manner. That is because in order to get the Introduction to Softwarephysics listed as the first post in the context root of I have to perform a few IT tricks. When publishing a new posting, I simply copy the contents of the Introduction to Softwarephysics to a new posting called the New Introduction to Softwarephysics. Then I update the original Introduction to Softwarephysics entry with the title and content of the new posting to be published. I then go back and take “New” out of the title of the New Introduction to Softwarephysics. This way the Introduction to Softwarephysics always appears as the first posting in the context root of The side effect of all this is that the real posting date of posts is the date that appears on the post that you get when clicking on the Newer Post link at the bottom left of the posting webpage.

SoftwarePhysics 101 – The Physics of Cyberspacetime is now available on Microsoft OneDrive.

SoftwarePhysics 101 – The Physics of Cyberspacetime - Original PowerPoint document

Entropy – A spreadsheet referenced in the document

BSDE – A 1989 document describing how to use BSDE - the Bionic Systems Development Environment - to grow applications from genes and embryos within the maternal BSDE software.

Comments are welcome at

To see all posts on softwarephysics in reverse order go to:

Steve Johnston

Monday, August 10, 2015

Stop Pushing So Many Buttons

Earlier this year the Middleware Operations group of my current employer was reorganized, and all of the onshore members, including myself, were transferred to another related middleware group. This other middleware group puts together the designs for middleware components that Middleware Operations then installs. So I had a lot to learn from the old-timers in this new middleware group. Now the way we do IT training these days is for the trainer to share their laptop desktop with the trainee, using some instant messaging software. That way the trainee can see what the trainer is doing, while the trainer explains what he is doing over some voice over IP telephone software. This was a very enlightening experience for me, because not only was it an opportunity to learn some new IT skills, but it also provided me with some first-hand observations of how other IT professionals did their day-to-day work for the first time in many years, and it also brought back some long-forgotten memories as well. The one thing that really stuck out for me was that the IT professionals in my new middleware group were pushing way too many buttons to get their work done, and I knew that was a very bad thing. Now personally, I consider the most significant finding of softwarephysics to be an understanding of the impact that self-replicating information has had upon the surface of the Earth over the past 4.0 billion years and also of the possibilities of its ongoing impact for the entire Universe - see the Introduction to Softwarephysics for more on that. But from a purely practical IT perspective, the most important finding of softwarephysics is certainly that pushing lots of buttons to get IT work done is a very dangerous thing. Now why is pushing buttons so bad? To understand why we need to go back a few years.

In 1979 I made a career change from being an exploration geophysicist, exploring for oil with Amoco, to become an IT professional in Amoco's IT department. As I explained in the Introduction to Softwarephysics, this was quite a shock. The first thing I noticed was that, unlike other scientific and engineering professionals, IT professionals seemed to be always working in a controlled state of panic, reacting to very unrealistic expectations from an IT Management structure that seemed to constantly change direction from one week to the next. And everything was always a "Top Priority", so that all work done by IT professionals seemed to consist of juggling a large number of equally important "Top Priority" activities all at the same time. My hope through the ensuing years was always that IT would one day mature with time, but that never seemed to have happened. In fact, I think things have actually gotten worse over the past 36 years, and some of these bad habits have even seemed to have diffused outside of IT into the general public, as software began to take over the world. This observation is based upon working in a number of corporate IT departments throughout the years and under a large number of different CIOs. The second thing I noticed back in 1979 was that, unlike all of the other scientific and engineering disciplines, IT seemingly had no theoretical framework to work from. I think this second observation may help to explain the first observation.

So why is it important to have a theoretical framework? To understand the importance, let's compare the progress that the hardware guys made since I started programming back in 1972 with the progress that the software guys made during this very same period of time. When I first started programming back in 1972, I was working on a million-dollar mainframe computer with about 1 MB of memory and clock speed of 750 KHz. Last spring I bought a couple of low-end Toshiba laptops for my wife and myself, each with two Intel Celeron 2.16 GHz CPUs, 4 GB of memory, a 500 GB disk drive and a 15.6 inch HD LED screen for $224 each. I call these low-end laptops because they were the cheapest available online at Best Buy, yet they were both many thousands of times more powerful than the computers that were used to design the current U.S. nuclear arsenal. Now if you work out the numbers, you will find that the hardware has improved by at least a factor of 10 million since 1972. Now how much progress have we seen in our ability to develop, maintain and support software over this very same period of time? I would estimate that our ability to develop, maintain and support software has only increased by a factor of about 10 – 100 times since 1972, and I think that I am being very generous here. In truth, it is probably much closer to being a factor of 10, rather than to being a factor of 100. Here is a simple thought experiment. Imagine assigning two very good programmers the same task of developing some software to automate a very simple business function. Imagine that one programmer is a 1972 COBOL programmer using an IBM 029 keypunch machine, while the other programmer is a 2015 Java programmer using the latest Java IDE (Integrated Development Environment) like Eclipse to write and debug a Java program. Now set them both to work. Hopefully, the Java programmer would win the race, but by how much? I think that the 2015 Java programmer, armed with all of the latest tools of IT, would be quite pleased if she could finish the project 10 times faster than the 1972 programmer cutting cards on an IBM 029 keypunch machine. This would indeed be a very revealing finding. Why has the advancement of hardware outpaced the advancement of software by nearly a factor of a million since 1972? How can we possibly account for this vast difference between the advancement of hardware and the advancement of software over a span of 43 years that is on the order of 6 orders of magnitude? Clearly, two very different types of processes must be at work to account for such a dramatic disparity.

Figure 1 - An IBM 029 keypunch machine like the one I first learned to program on at the University of Illinois in 1972.

Figure 2 - Each card could hold a maximum of 80 bytes. Normally, one line of code was punched onto each card.

Figure 3 - The cards for a program were held together into a deck with a rubber band, or for very large programs, the deck was held in a special cardboard box that originally housed blank cards. Many times the data cards for a run followed the cards containing the source code for a program. The program was compiled and linked in two steps of the run and then the generated executable file processed the data cards that followed in the deck.

Figure 4 - To run a job, the cards in a deck were fed into a card reader, as shown on the left above, to be compiled, linked, and executed by a million-dollar mainframe computer with a clock speed of about 750 KHz and about 1 MB of memory.

Figure 5 - Now I push these very same buttons that I pushed on IBM 029 keypunch machines, the only difference is that I now push them on a $224 machine with 2 CPUs running with a clock speed of 2.16 GHz and 4 GB of memory.

My suggestion would be that the hardware advanced so quickly because it was designed using a theoretical framework based upon physics. Most of this advancement was based upon our understanding of the behavior of electrons. Recently, we have also started using photons in fiber optic devices too, but photons are created by manipulating electrons, so it all goes back to understanding how electrons behave. Now here is the strange thing. Even though we have improved hardware by a factor of 10 million since 1972 by tricking electrons into doing things, we really do not know what electrons are. Electrons are certainly not little round balls with a negative electrical charge. However, we do have a very good effective theory called QED (Quantum Electrodynamics) that explains all that we need to know to build very good hardware. QED (1948) is an effective theory formed by combining two other very good effective theories - quantum mechanics (1926) and the special theory of relativity (1905). QED has predicted the gyromagnetic ratio of the electron, a measure of its intrinsic magnetic field, to 11 decimal places. This prediction has been validated by rigorous experimental data. As Richard Feynman has pointed out, this is like predicting the exact distance between New York and Los Angeles to within the width of a human hair! Recall that an effective theory is an approximation of reality that only holds true over a certain restricted range of conditions and only provides for a certain depth of understanding of the problem at hand. For example, QED cannot explain how electrons behave in very strong gravitational fields, like those close to the center of a black hole. For that we would need a quantum theory of gravity, and after nearly 90 years of work we still do not have one. So although QED is enormously successful and makes extraordinarily accurate predictions of the behavior of electrons and photons in normal gravitational fields, it is fundamentally flawed and therefore can only be an approximation of reality because it cannot deal with gravity. This is an incredibly important idea to keep in mind because it means that we are all indeed lost in space and time because all of the current theories of physics, such as Newtonian mechanics, Newtonian gravity, classical electrodynamics, thermodynamics, statistical mechanics, the special and general theories of relativity, quantum mechanics, and quantum field theories like QED are all just effective theories that are based upon models of reality, and all of these models are approximations - all of these models are fundamentally "wrong", but at the same time, these effective theories make exceedingly good predictions of the behavior of physical systems over the limited ranges in which they apply. Think of it like this. For most everyday things in life you can get by just fine by using a model of the Earth that says that the Earth is flat and at the center of the Universe. After all, this model worked just great for most of human history. For example, you certainly can build a house assuming that the Earth is flat. If you use a plumb-bob on the left and right sides of your home to ensure that you are laying bricks vertically, you will indeed find that to a very good approximation the left and right sides of your home will end up being parallel and not skewed, and that is similar to the current state of physics as we know it. It is all just a collection of very useful approximations of reality, and that is what softwarephysics is all about. Softwarephysics is just a useful collection of effective theories and models that try to explain how software seems to behave for the benefit of IT professionals.

Applying Physics to Software
In the The Fundamental Problem of Software I explained that the fundamental problem IT professionals faced when dealing with software was that the second law of thermodynamics caused small bugs in software when we pushed buttons while working on software and that because the Universe is largely nonlinear in nature, these small bugs oftentimes caused severe and sometimes lethal outcomes when the software was run. I also proposed that to overcome these effects IT should adopt a biological approach to software, using the techniques developed by the biosphere to overcome the adverse effects of the second law of thermodynamics in a nonlinear Universe. This all stems from the realization that all IT jobs essentially boil down to simply pushing buttons. All you have to do is push the right buttons, in the right sequence, at the right time, and with zero errors. How hard can that be? Well, the second law makes it very difficult to push the right buttons in the right sequence and at the right time because there are so many erroneous combinations of button pushes. Writing and maintaining software is like looking for a needle in a huge utility phase space. There are just too many ways of pushing the buttons “wrong”. For example, my laptop has 86 buttons on it that I can push in many different combinations. Now suppose I need the following 10-character line of code:

i = j*k-1;

I can push 10 buttons on my laptop to get that line of code. The problem is that I can also push 10 buttons on my 86-button keyboard 8610 = 2.214 x 1019 other ways and only one out of that huge number will do the job. Physicists use the concepts of information and entropy to describe this situation.

In The Demon of Software we saw that entropy is a measure of the amount of disorder, or unknown information, that a system contains. Systems with lots of entropy are disordered and also contain lots of useless unknown information, while systems with little entropy are orderly in nature and also contain large amounts of useful information. The laws of thermodynamics go on to explain that information is very much like energy. The first law of thermodynamics states that energy cannot be created nor destroyed. Energy can only be transformed from one form of energy into another form of energy. Similarly, information cannot be destroyed because all of the current effective theories of physics are both deterministic and reversible, meaning that they work just as well moving backwards in time as they do moving forwards in time, and if it were possible to destroy the information necessary to return a system to its initial state, then the Universe would not be reversible in nature as the current theories of physics predict, and the current theories of physics would then collapse. However, as IT professionals we all know that it is indeed very easy to destroy information, like destroying the code for a functional program by simply including a few typos, so what is going on? Well, the second law of thermodynamics states that although you cannot destroy energy, it is possible to transform useful energy, like the useful chemical energy in gasoline, into useless waste heat energy at room temperature that cannot be used to do useful work. The same thing goes for information. Even though you cannot destroy information, you can certainly turn useful information into useless information. In softwarephysics we exclusively use Leon Brillouin’s formulation for the concept of information as negative entropy or negentropy to explain why. In Brillouin’s formulation of information it is very easy to turn useful low-entropy information into useless high-entropy information by simply scrambling the useful information into useless information. For example, if you take the source code file for an apparently bug-free program and scramble it with some random additions and deletions, the odds are that you will most likely end up with a low-information high-entropy mess. That is because the odds of creating an even better version of the program by means of inserting random additions and deletions are quite low, while turning it into a mess are quite high. There are simply too many ways of messing up the program to win at that game. As we have seen this is simply the second law of thermodynamics in action. The second law of thermodynamics is constantly degrading low-entropy useful information into high-entropy useless information.

What this means is that the second law of thermodynamics demands that whenever we do something, like pushing a button, the total amount of useful energy and useful information in the Universe must decrease! Now of course the local amount of useful energy and information of a system can always be increased with a little work. For example, you can charge up your cell phone and increase the amount of useful energy in it, but in doing so, you will also create a great deal of waste heat when the coal that generated the electricity is burned back at the power plant because not all of the energy in the coal can be converted into electricity. Similarly, if you rifle through a deck of cards, you can always manage to deal yourself a full house, but in doing so, you will still decrease the total amount of useful information in the Universe. If you later shuffle your full house back into the deck of cards, your full house still exists in a disordered shuffled state with increased entropy, but the information necessary to reverse the shuffling process cannot be destroyed, so your full house could always be reclaimed by exactly reversing the shuffling process. That is just not a very practical thing to do, and that is why it appears that the full house has been destroyed by the shuffle.

Now since all of the current effective theories of physics are both deterministic and reversible in time, the second law of thermodynamics is the only effective theory in physics which differentiates between the past and the future. The future will always contain more entropy and less useful information than the past. Thus physicists claim that the second law of thermodynamics defines the arrow of time, and so the second law essentially defines the river of time that we experience in our day-to-day lives. Now whenever you start pushing lots of buttons to do your IT job, you are essentially trying to swim upstream against the river of time. You are trying to transform useless information into useful information, like pulling a full house out of a shuffled deck of cards, while the Universe is constantly trying to do the opposite by turning useful information into useless information. The other problem we have is that we are working in a very nonlinear utility phase space, meaning that pushing just one button incorrectly usually brings everything crashing down. For more about nonlinear systems see Software Chaos.

Living things have to deal with this very same set of problems. Living things have to arrange 92 naturally occurring atoms into complex molecules that perform the functions of life, and the slightest error can lead to dramatic and oftentimes lethal effects as well. So I always try to pretend that pushing the correct buttons in the correct sequence is like trying to string the correct atoms into the correct sequence to make a molecule in a biochemical reaction that can do things. Living things are experts at this, and apparently seem to overcome the second law of thermodynamics by dumping entropy into heat as they build low-entropy complex molecules from high-entropy simple molecules and atoms. I always try to do the same thing by having computers dump entropy into heat as they degrade low-entropy electrical energy into high-entropy heat energy. I do this by writing software that generates the software that I need to do my job as much as possible. That dramatically reduces the number of buttons I need to push to accomplish a task, and reduces the chance of making an error too. I also think of each line of code that I write as a step in a biochemical pathway. The variables are like organic molecules composed of characters or “atoms” and the operators are like chemical reactions between the molecules in the line of code. The logic in several lines of code is the same thing as the logic found in several steps of a biochemical pathway, and a complete function is the equivalent of a full-fledged biochemical pathway in itself.

So how do you apply all this physics to software? In How to Think Like a Softwarephysicist I provided a general framework on how to do so. But I also have some very good case studies of how I have done so during my IT career. For example, in SoftwarePhysics I described how I started working on BSDE - the Bionic Systems Development Environment back in 1985 while in the IT department of Amoco. BSDE was an early mainframe-based IDE at a time when there were no IDEs. During the 1980s BSDE was used to grow several million lines of production code for Amoco by growing applications from embryos. The DDL statements used to create the DB2 tables and indexes for an application were stored in a sequential file called the Control File and performed the functions of genes strung out along a chromosome. Applications were grown within BSDE by turning their genes on and off to generate code. BSDE was first used to generate a Control File for an application by allowing the programmer to create an Entity-Relationship diagram using line printer graphics on an old IBM 3278 terminal.

Figure 6 - BSDE was run on IBM 3278 terminals, using line printer graphics, and in a split-screen mode. The embryo under development grew within BSDE on the top half of the screen, while the code generating functions of BSDE were used on the lower half of the screen to insert code into the embryo and to do compiles on the fly while the embryo ran on the upper half of the screen. Programmers could easily flip from one session to the other by pressing a PF key.

After the Entity-Relationship diagram was created, the programmer used a BSDE option to create a skeleton Control File with DDL statements for each table on the Entity-Relationship diagram and each skeleton table had several sample columns with the syntax for various DB2 datatypes. The programmer then filled in the details for each DB2 table. When the first rendition of the Control File was completed, another BSDE option was used to create the DB2 database for the tables and indexes on the Control File. Another BSDE option was used to load up the DB2 tables with test data from sequential files. Each DB2 table on the Control File was considered to be a gene. Next a BSDE option was run to generate an embryo for the application. The embryo was a 10,000 line of code PL/I, Cobol or REXX application that performed all of the primitive functions of the new application. The programmer then began to grow his embryo inside of BSDE in a split screen mode. The embryo ran on the upper half of an IBM 3278 terminal and could be viewed in real time, while the code generating options of BSDE ran on the lower half of the IBM 3278 terminal. BSDE was then used to inject new code into the embryo's programs by reading the genes in the Control File for the embryo in a real time manner while the embryo was running in the top half of the IBM 3278 screen. BSDE had options to compile and link modified code on the fly while the embryo was still executing. This allowed for a tight feedback loop between the programmer and the application under development. In fact, BSDE programmers sometimes sat with end-users and co-developed software together on the fly. When the embryo had grown to full maturity, BSDE was used to create online documentation for the new application and was also used to automate the install of the new application into production. Once in production, BSDE generated applications were maintained by adding additional functions to their embryos. Since BSDE was written using the same kinds of software that it generated, I was able to use BSDE to generate code for itself. The next generation of BSDE was grown inside of its maternal release. Over a period of seven years, from 1985 – 1992, more than 1,000 generations of BSDE were generated, and BSDE slowly evolved into a very sophisticated tool through small incremental changes. BSDE dramatically improved programmer efficiency by greatly reducing the number of buttons programmers had to push in order to generate software that worked.

Figure 7 – BSDE appeared as the cover story of the October 1991 issue of the Enterprise Systems Journal

BSDE had its own online documentation that was generated by BSDE. Amoco's IT department also had a class to teach programmers how to get started with BSDE. As part of the curriculum Amoco had me prepare a little cookbook on how to build an application using BSDE:

BSDE – A 1989 document describing how to use BSDE - the Bionic Systems Development Environment - to grow applications from genes and embryos within the maternal BSDE software.

Similarly in MISE in the Attic I described how softwarephysics could also be used in an operations support setting. MISE is a tool kit of 1688 commands that call Unix Korn shell scripts via Unix aliases, that I wrote to allow Middleware IT professionals to quickly do their jobs without pushing lots of buttons. Working in the high-pressure setting of Middleware Operations, it is also very important to be able to rapidly find information quickly in a matter of a few seconds or less, so MISE also has a couple of commands that let IT professionals quickly search for MISE commands using keyword searches. Under less stressful conditions, MISE was also designed to be an interactive tool kit of commands that assists in the work that Middleware IT professionals need to perform all day long. Whenever I find myself pushing lots of buttons to complete an IT task, I simply spend a few minutes to create a new MISE command that automates the button pushing process. Now that I have a rather large library of MISE commands, I can usually steal some old MISE code and exapt it into performing another function in about 30 minutes or so. Then I can use a few MISE commands to push out the new MISE command to the 550 servers I support. These MISE commands can push out the new code in about 5 minutes and can be run in parallel to push out several files at the same time. I use MISE all day long, and the MISE commands greatly reduce the number of buttons I have to push each day.

But why create a tool kit? Well, living things really love took kits. For example, prokaryotic bacteria and archea evolved huge tool kits of biochemical pathways during the first billion years of life on the Earth and these biochemical tool kits were later used by all of the higher forms of eukaryotic life. Similarly, in the 1980s biologists discovered that all forms of complex multicellular organisms used an identical set of Hox genes to control the head to tail development of complex multicellular organisms. These Hox genes must have appeared some time prior to the Cambrian explosion 530 million years ago to perform some other functions at the time. During the Cambrian explosion these Hox genes were then exapted into building complex multicellular organisms and have done so ever since. Similarly, tool kits are also very useful for IT professionals. For example, in my new middleware group I am still using MISE all day long to assist in my daily work to dramatically reduce the number of buttons I need to push each day. Since moving to this new middleware group I have also been adding new functions to MISE that help me in this new position, and I also have been trying to get the other members of my new middleware group to try using MISE. Getting IT professionals to think like a softwarephysicist is always difficult. What I did with BSDE and MISE was to simply use softwarephysics to create software tools for my own use. Then I made these software tools available to all the members of my IT group and anybody else who might be interested. It's like walking around the office in 1985 with a modern smart phone. People begin to take notice and soon want one of their own, even if they are not at first interested in how the smart phone works. With time people begin to notice that their smart phones do not work very well in basements or in forests far from cities, and they want to know why. So I then slowly begin to slip in a little softwarephysics to explain to them how smart phones work, and that knowing how they work makes it easier to use them.

How to Work with IT Management in a Positive Manner While Innovating
If any of you ever do become inspired to become a softwarephysicist, with the intent of spreading softwarephysics throughout your IT department, be sure to move forward cautiously. Don't expect IT Management to greet such efforts with open arms. Always remember that IT Management is charged with preserving the meme-complex that is your IT department and that meme-complexes are very wary of new memes contaminating the meme-complex. See How to Use Softwarephysics to Revive Memetics in Academia for more about memes. For the past 36 years my experience has been that IT Management always tries to suppress innovation. I don't fault IT Management for this because I think this is just the way all human hierarchies work, even scientific hierarchies like physics - see Lee Smolin's The Trouble with Physics (2006) for an interesting discussion of the problems he has encountered due to the overwhelming dominance of string theory in physics today. Hierarchies do not foster innovation, hierarchies always try to suppress innovation. That is why innovation usually comes from entrepreneurs and not from large hierarchical organizations. The way I get around this challenge is to first build some software that is based upon softwarephysics, but I don't tell people why the software works so well. Once people get hooked into using the software, I then slowly begin to explain the theory behind why the software makes their IT job so much easier. So I slip in the softwarephysics very slowly. I also follow a bacterial approach when writing the software tools that are based upon softwarephysics, meaning that I write the software tools using the most primitive of software components available because those are the software components that IT Management cannot easily take away. For example, MISE is written entirely using Unix Korn shell scripts. My current IT Management cannot easily shut down Unix Korn shell scripts because our whole IT infrastructure runs on them. Unix Korn shell scripts are essential to day-to-day operations, and like many bacteria, are simply too primitive to stamp out without killing the whole IT infrastructure. Now I could have written much of MISE using jython wsadmin programs to talk to Websphere directly. But I knew that someday somebody would write a jython wsadmin program that would take down a whole production Websphere Cell, and sure enough, that day came to pass. In response, our IT Management immediately placed a ban on running jython wsadmin programs during the day, and we now can only run them at night using an approved change ticket. Getting an approved change ticket takes about a week. That would have killed MISE because MISE users would no longer be able to run MISE commands all day long to do their jobs. But because MISE just harmlessly reads deployed Websphere .xml files on servers, the ban had no effect on MISE. I also wrote MISE so that it primarily just reads things and provides valuable information. MISE will also help you do things like restart Websphere Appservers, but it does so in a very passive manner. MISE will make sure you are using the proper production ID for the restart, and will put you into the proper directory to do the restart, and will also print out the command to do the restart, but the MISE user must copy/paste the printed command into their Unix session and hit the Enter key to pull the trigger. So MISE is designed to never do anything that IT Management might consider as dangerous. On the contrary, MISE dramatically reduces the chance of an IT professional making a mistake that could cause a production outage.

So whenever you find yourself pushing lots of buttons to get an IT task accomplished, think back to what softwarephysics has to say on the subject. There must be an easier way to get the job done with fewer buttons. Try reading some biology books for tips.

Comments are welcome at

To see all posts on softwarephysics in reverse order go to:

Steve Johnston

Wednesday, July 01, 2015

An Alternative Model of the Software Universe

In The Software Universe as an Implementation of the Mathematical Universe Hypothesis I explained that when I first began to work on softwarephysics my intent was to simply come up with a set of effective theories to help explain how the Software Universe seemed to behave and that I did not have the luxury of trying to shoot for an all encompassing explanation of what the Software Universe was all about. In the MUH - the Mathematical Universe Hypothesis – both the Software Universe and our physical Universe are simply timeless mathematical structures in a Platonic sense that have always existed because mathematical structures are not time-bound – they are eternal truths outside of time. For example, for the case of the Software Universe, Max Tegmark remarked in Our Mathematical Universe: My Quest for the Ultimate Nature of Reality (2014) that it did not really matter if the software was actually running on a computer or not because the output of the software was determined by the software itself and the input that it was to work upon. The same thing is essentially true for the infinite number of other universes that can be generated from the Level IV multiverse of mathematical structures. They also do not have the element of time because they are generated from timeless mathematical structures. This is a bit bothersome because as sentient beings we definitely sense the passage of time. We all seem to live in the present moment and can remember the past and anticipate the future. But others claim that this sensation of the flow of time is a deception played upon us by our senses, and those “others” are not just philosophers dallying around in metaphysics; most physicists would make the same claim that the apparent flow of time is an illusion too.

All of this made me feel that it was time to reread a couple of good books: Programming the Universe: A Quantum Computer Scientist Takes on the Cosmos (2006) by Seth Lloyd and The Trouble with Physics (2006) and Time Reborn – From the Crisis in Physics to the Future of the Universe (2013) by Lee Smolin to help clarify the situation. I like to periodically reread good books because I find that each reading seems to bring with it additional meaning. This is not just because repetition inherently brings with it better understanding, but also because in the interval between readings, hopefully I will have advanced my personal model of what’s it all about, and it doing so, become more receptive to some of the subtleties of the books I am rereading.

In Programming the Universe Seth Lloyd contends that our Universe is a quantum mechanical Computational Universe that has been calculating how to behave from the very beginning. Seth Lloyd came to this conclusion as MIT’s sole quantum mechanical engineer working on building practical quantum computers. During the course of his research Seth Lloyd has learned how to talk to atoms in a quantum mechanical way. Through intimate dealings with these atoms, he found that the atoms were constantly flipping quantum mechanical states in a controlled manner prescribed by quantum mechanics. Since a computer is simply a large number of switches that operate in a controlled manner, our Universe can therefore be thought of as a Computational Universe, and therefore, must necessarily be capable of computation. In fact, our current quest to build quantum computers can simply be viewed as an attempt to domesticate this natural tendency for our Universe to compute in a quantum mechanical manner. Seth Lloyd calculates that our section of the Computational Universe which is defined by our current cosmic horizon and consists of all quantum particles out to a distance of 46 billion light years, has performed about 10122 operations on 1092 bits over the past 13.8 billion years. This domestication of the quantum mechanical behavior of our Computational Universe has already led to the construction of many trillions of classical computers already. So Seth Lloyd’s Computational Universe model is very similar to Max Tegmark’s CUH – the Computable Universe Hypothesis – which is a subset of his MUH. The chief difference is that the Computational Universe model has the element of time, while the CUH model does not.

A Matter of Time
So it all boils down to the age-old question of whether time is real or not. That might sound like a silly question, but it has been bothering physicists and philosophers from the very beginning. But even that statement makes the assumption that time actually is real to begin with, and highlights the difficulty of the problem. Lee Smolin takes up this difficult problem of time in Time Reborn, and proposes that physics and cosmology must go back to Newton’s original contention that there is an absolute time that is a fundamental aspect of objective reality in order to make progress. In his Principia (1687), Newton defined an absolute time independent of observers as:

Absolute, true and mathematical time, of itself, and from its own nature flows equably without regard to anything external, and by another name is called duration: relative, apparent and common time, is some sensible and external (whether accurate or unequable) measure of duration by the means of motion, which is commonly used instead of true time ...

In Part 1 of Time Reborn, Lee Smolin explains how physics slowly expelled this concept of an absolute time from its current worldview. It all began with Galileo. Galileo observed that a thrown ball moved in a parabolic arc rather than in straight lines as Aristotle had contended. According to Aristotle a thrown ball moved in a straight line until it ran out of impetus, and then it fell to the earth in a straight line. The ball fell to the earth because it was in the ball’s nature to be at rest on the ground. Obviously, Aristotle must never have thrown a ball during his entire life!

Figure 1 – According to Aristotle a thrown ball went in a straight line until it ran out of impetus and then it fell to the ground in a straight line.

Even so, Aristotle’s thoughts carried the day for more than a thousand years.

Figure 2 – The above figure demonstrates that bad theories can often lead to adverse military results. Granted, Aristotle’s theory may have predicted that you should aim your cannon at 450 for maximum range, but it would not be of much use for aiming anti-aircraft shells.

Figure 3 – Galileo maintained that a thrown ball moved in a parabolic arc and not in straight lines.

Figure 4 – A parabola can be obtained by slicing a cone at an angle.

Figure 5 – A parabola is also defined as the set of points that are equal distant from a point called the Focus and a straight line called the directrix.

Figure 6 – A parabola can also be defined by a mathematical equation.

Next came the Newtonian mechanics to be found in Newton’s Principia (1687) which laid down Newton’s three laws of motion and allowed one to easily calculate the equation of the parabolic arc of a thrown ball given the initial speed and angle of the throw. Since Newton’s laws of motion are deterministic, meaning that given the ball’s initial conditions of speed and angle, the ball always follows the same exact path in space and time, it is possible to graph the thrown ball moving through space and time.

Figure 7 – A thrown ball always follows the same exact path through space and time that are defined by Newton’s laws of motion and the initial conditions of the thrown ball. In the figure above, the ball and the bullet start out at the same location X = 0 and hit the ground later at the very same spot along the X-axis that represents distance. Notice that the bullet hits the ground in much less time t than does the ball. Also note that the slower ball must also rise to a much higher height Z than does the bullet. So the fast bullet has a much flatter trajectory than does the slower ball which is lobbed to the same spot.

And since Newton’s laws of motion are reversible in time that means we could easily run time backwards and have the thrown ball repeat its parabolic path through space and time in reverse. This was the beginning of the Block Universe model of the Universe in physics. Lee Smolin calls this the Newtonian paradigm. What we do in physics is to try to isolate a system as much as possible from the rest of the Universe so that the effects upon the subsystem are reduced as much as possible. Lee Smolin calls this doing “physics in a box”. For example, we must throw our ball in a vacuum for it to really follow a parabolic path as much as possible. Then given deterministic laws and initial conditions, we can predict how the subsystem will behave moving into the future as well as into the past. Notice that in this worldview the concept of time does not play much of a role because the deterministic laws are timeless mathematical structures and the initial conditions are set for one particular time that then determine the position of the ball for all future times and for all times in the past as well. In such a model, there is no point in worrying about the future because it is all a “done deal” from the onset, and there is no free will even to let you choose to worry or not. Everything just “is”.

The Block Universe model obtained a huge boost from Albert Einstein in 1905 with his special theory of relativity. That is because Einstein was able to eliminate Newton’s concept of absolute time from his new theory. Granted, people had been debating Newton’s concept of an absolute space all along, but they had not been debating the concept of absolute time until Einstein came along. The relativists, like Galileo and Newton’s archenemy Gottfried Leibniz, had argued all along that since all spatial measurements were purely relative in nature that absolute space did not exist, and that was why the effects of absolute space could not be observed. This was based upon Galileo’s observation that you could not tell if you were moving or at rest when in the hold of a ship on a calm sea. No matter what experiment you might perform, you always got the very same result when the ship was moving forward at great speed or when it was standing still at the dock. That is why you can easily pour those little bottles of gin into a glass while traveling at 600 mph in a jet airliner. But Newton suggested that absolute space really was real because objects obviously behaved differently when you were accelerating. That is why it is not wise to try to pour those little bottles of gin while an airplane is accelerating during takeoff. The odds are that you will spill some of the gin because the gin will not pour straight while accelerating. Newton explained that things behaved differently when you were accelerating because you were accelerating relative to absolute space. He demonstrated this idea with his famous bucket experiment. Take a bucket of water that is at rest with respect to absolute space. The water will have a flat surface to begin with. Then start to spin the bucket and water relative to absolute space. At first the water will remain at rest with respect to absolute space and will remain flat in the bucket. But as the water picks up rotational motion from the bucket, it will soon begin to form a concave surface because now the water is rotating (accelerating) relative to absolute space. So absolute space must exist after all.

Figure 8 – Newton maintained that even though Galileo’s principle of relativity prevented experiments from revealing absolute space for reference frames that were not accelerating, you could still observe the effects of absolute space by spinning a bucket of water.

The way Einstein got rid of the concept of an absolute space was by getting rid of absolute space and absolute time simultaneously with his new theory. Einstein simply took Galileo’s principle of relativity and extended it to electromagnetic experiments as well. In 1820 Hans Christian Oersted discovered that when an electrical current flowed in a wire it produced a magnetic field that resulted from the electrical charges moving in the wire. So now we had a handy way of telling if something was standing still or moving. All you had to do was observe an electrically charged object. If it gave off a magnetic field, then you knew it was moving; if it did not give off a magnetic field, then you knew it was standing still. So Newton was “right” and Galileo was “wrong”, an observer could use electromagnetic experiments to tell if he was standing still or moving relative to absolute space. There was just one problem, the idea did not work. In 1901, Trouton and Noble conducted just such an experiment using a suspended charged capacitor. They tried to observe the magnetic field that should have been given off by the suspended charged capacitor as it moved through absolute space onboard the Earth, as the Earth orbited the Sun. But they did not find any! In 1905, Einstein published On the Electrodynamics of Moving Bodies in which he proposed that Galileo was right after all. In this paper, Einstein proposed that you really could not conduct any experiment, including electromagnetic experiments that would reveal if you were moving or standing still relative to an absolute space, because there was no such thing as an absolute space. All motion was relative to other objects just as Galileo had proposed from the start.

In order for this to be true, Einstein had to raise two conjectures to the level of postulates:

1. The laws of physics are the same for all observers, even for observers moving relative to each other at constant speeds in straight lines.

2. The speed of light is the same for all observers, even for observers moving relative to each other at constant speeds in straight lines.

If the above two postulates were not true, you could easily tell if you were moving or standing still relative to an absolute space. All you would have to do is measure the speed of light in different directions, and if it were not the same in all directions, then you would know that you were moving relative to an absolute space. So that got rid of absolute space, but the above two postulates also caused a problem for absolute time as well because they greatly affected simultaneous events that were separated in space. This is best explained by the two animations at:

In the first animation, the Einstein on the moving platform observes two photons arriving at the same time and concludes that both photons were emitted at the same time. In the second animation, the Einstein on the ground agrees that both photons hit the Einstein on the platform at the same time too, but concludes that photon A was emitted before photon B. Note that both Einstein’s can claim to be standing still because neither one can detect any motion with experiments they perform because neither one is accelerating. What the special theory of relativity did for the Block Universe model of Figure 7 was to replace the concepts of absolute space and absolute time with the concept of an absolute 4-dimensional spacetime. With this approach time was simply merged in with the three spatial dimensions (x, y, z) into a 4-dimensional spacetime of (x, y, z, t), and time just became a strange cousin of the other spatial dimensions. The distance between events in 4-dimensional spacetime is called the interval s and is defined just like the distance d between points in 3-dimensional space.

Distance between points in space:
d² = ∆x² + ∆y² + ∆z²

Interval between events in spacetime:
s² = ∆x² + ∆y² + ∆z² + i²∆t²

where i² = -1

In 4-dimensional spacetime, the only distinguishing thing about time is that you have to multiply time by the imaginary number i in the equation for the interval between events. Otherwise, time is simply treated like the other spatial dimensions of x, y and z. Essentially, with the 4-dimensional spacetime Block Universe model of the Universe, the Universe became a static tube of salami that had seemingly always existed. Observers moving relative to each other simply sliced the static tube of salami at different angles. For example, in Figure 9 we see the Block Universe tube of salami cut at two different angles. In the first cut of the salami two events happen at different positions in space and also different positions in time so the events are not simultaneous. This is the situation for our second Einstein on the ground who observed two photons being emitted at different places in space and also at different times. The second salami is cut at a different angle, which again has the two photons being emitted at different positions in space, but this time both photons are emitted at the same time on the time slice of salami. This is the situation for our first Einstein riding on the platform who observed two photons being emitted at different places in space, but at the same time in his reference frame.

Figure 9 – The special theory of relativity extended the Block Universe model by introducing the concept of an absolute spacetime.

This new 4-dimensional spacetime Block Universe model, brought on by the special theory of relativity in 1905, really seemed to kill the concept of an absolute space and an absolute time because space and time got thoroughly blended together in the process of making the spacetime salami so that you could no longer tell them apart. If I am moving relative to you that means that my time can become your space and vice versa. But what about Newton’s bucket? The water in Newton’s bucket was surely telling us that it was rotating with respect to something. But don’t forget that the water in Newton’s bucket was rotating, and was thus accelerating, because the direction of the water’s velocity in the rotating bucket was constantly changing so that the water could move in a circle, and that is acceleration. The special theory of relativity does not work for accelerating reference frames like a spinning bucket of water. For that we need Einstein’s general theory of relativity (1915). With the general theory of relativity Einstein extended the 4-dimensional spacetime Block Universe model to all forms of motion, including accelerating reference frames, and in the process demonstrated that accelerated motion was essentially the same thing as gravity. In the general theory of relativity gravity is no longer a force between masses. Instead, gravity becomes another “fake force” like the centrifugal force. When you make a sharp turn in your car at high speed your body feels the “fake force” of centrifugal force pulling you away from the center of curvature of your turn. But that “fake force” is really just your body trying to move in a straight line through space according to Newton’s first law of motion. Similarly, in the general theory of relativity the gravitational force your body feels pulling you down to the Earth is simply your body trying to move in a straight line through spacetime and is also a “fake force”! If you jump off a cliff and find yourself in free fall, moving in a straight line through spacetime, gravity suddenly disappears, just as if you had been thrown from a car making a tight turn and had found yourself moving in a straight line through space with the “fake” centrifugal force suddenly disappearing too. In order to make that adjustment to the special theory of relativity that used flat time slices through the 4-dimensional spacetime salami, Einstein had to make the 4-dimensional spacetime salami internally curvy. In the general theory of relativity matter, energy and pressure can all cause spacetime to warp and become curvy, and it is the curvy 4-dimensional spacetime that creates the illusion of gravity. When there are no significant amounts of matter, energy or pressure present, the 4-dimensional spacetime salami is not internally distorted, so that slices through it are flat and we return again to the special case of flat spacetime that is covered by the special theory of relativity. Thus, when the surface of the water in a spinning bucket forms a concave shape, it is not moving relative to absolute space, it is moving relative to absolute spacetime.

Figure 10 – In the general theory of relativity the 4-dimensional spacetime salami becomes internally curvy.

Cosmological Problems
Currently physicists and cosmologists are trying to explain the apparent fine-tuning of our Universe that allows for intelligent beings such as ourselves to exist. It seems that the current working hypothesis is that eternal chaotic inflation produces an infinite multiverse composed of an infinite number of separate causally isolated universes, such as our own, where inflation has halted, and each of these causally-separated universes may also be infinite in size too. As inflation halts in these separate universes, the Inflaton field that is causing the eternal chaotic inflation of the entire multiverse continues to inflate the space between each causally-separated universe at a rate that is much greater than the speed of light, quickly separating the universes by vast distances that can never be breached. Thus most of the multiverse is composed of rapidly expanding spacetime driven by inflation, sparsely dotted by causally-separated universes where the Inflaton field has decayed into matter and energy and inflation has stopped. Each of the causally-separated universes, where inflation has halted, will then experience a Big Bang of its own as the Inflaton field decays into matter and energy, leaving behind a certain level of vacuum energy. The amount of vacuum energy left behind will then determine the kind of physics each causally-separated universe experiences. In most of these universes the vacuum energy level will be too positive or too negative to create the kind of physics that is suitable for intelligent beings, creating the selection process that is encapsulated by the weak Anthropic Principle. This goes hand-in-hand with the current thinking in string theory that you can build nearly an infinite number of different kinds of universes, depending upon the geometries of the 11 dimensions hosting the vibrating strings and branes of M-Theory, the latest rendition of string theory. Thus an infinite multiverse has the opportunity to explore all of the nearly infinite number of possible universes that string theory allows, creating Leonard Susskind’s Cosmic Landscape (2006). In this model, our Universe becomes a very rare and improbable Universe.

Lee Smolin has problems with the above model for a variety of reasons. Lee Smolin is a realist in that he has a high level of confidence that there is a true physical reality that exists even if we are not around to observe it. Lee Smolin is also not a positivist or instrumentalist at heart, simply looking for effective theories that make good predictions of how physical systems behave, he is instead looking for explanations of physical reality, and that means he expects more out of a cosmological theory than most. He is also a strong follower of Karl Popper - a good cosmological theory must be falsifiable. If a cosmological theory cannot be proven wrong by observation then it is not of much use for Lee Smolin. This presents a problem for the current working hypothesis outlined above. In that model nearly all of the multiverse is beyond our cosmic horizon of about 46 billion light years, and consequently, is unobservable. Yet most people would contend that if we were suddenly transported 100 billion light years from our current position in our Universe we would most likely see exactly what we see from our current position. We would see hundreds of billions of galaxies all receding from us as the Universe expanded. But at a distance of 100 billion light years we would be causally disconnected from our current position and consequently unobservable. The fact that most of our own Universe is indeed unobservable has caused many physicists and cosmologists who have a high level of confidence in the eternal chaotic inflation model to forsake Karl Popper and instead adopt what Andrei Linde calls the Sherlock Holmes approach to science:

“when you have eliminated the impossible, whatever remains, however improbable, must be the truth.”

Of course how do you know when you have truly eliminated all of the impossibles? Perhaps someday you will also find that your current working hypothesis is also impossible. Or perhaps you will find that there are a very large number of possible theories that are not impossible. How do you then choose? This seems to be the most difficult problem in physics and cosmology today. Either things are too far away to be seen because they are more distant than our current cosmic horizon of 46 billion light years, or as with string theory they are too small for us to observe even with the LHC particle collider by many orders of magnitude. And probably we will never be able to build a particle collider with enough energy to probe the structures at the heart of string theory.

Lee Smolin's Solution
Lee Smolin contends that we need a new cosmological theory that goes beyond the model outlined above that explains why our Universe has the physical laws that it does and why it started out with the initial conditions that it did. In Time Reborn Lee Smolin proposes that the solution to this fine-tuning problem of the Universe is for physicists and cosmologists to forsake the Block Model of the Universe and to reinstate Newton's concept of an absolute time. In doing so, he becomes a Presentist, meaning that he considers time to real and the most fundamental characteristic of the Universe. For Presentists, only the present moment exists, and the physical laws of the Universe can change with time.

Figure 11 - For a Presentist time is real and only the present moment exists.

Reinstating Newton's concept of an absolute time runs counter to the Block Universe model outlined above to an extreme, but Lee Smolin presents a number of compelling arguments to explain why it is necessary to do so. His major concern is that the current cosmological model can never be validated by experimental or observational data because it is not falsifiable. In The Trouble with Physics, he outlined numerous cases in the past when some theoretical physicists were found to be running on pure mathematics, without the benefit of any validation by empirical evidence, only later to find that their mathematical theories collapsed when empirical data finally arrived on the scene.

Lee Smolin also points out that another problem with the current cosmological model is that all of the current theories of physics are effective theories and that the model extends these fundamentally flawed effective theories to the entire cosmos as a whole. Recall that an effective theory is an approximation of reality that only holds true over a certain restricted range of conditions and only provides for a certain depth of understanding of the problem at hand. For example, Newtonian mechanics is an effective theory that makes very good predictions for the behavior of objects moving less than 10% of the speed of light and which are bigger than a very small grain of dust. These limits define the effective range over which Newtonian mechanics can be applied to solve problems. For very small things we must use quantum mechanics and for very fast things moving in strong gravitational fields, we must use relativity theory. All of the current theories of physics, such as Newtonian mechanics, Newtonian gravity, classical electrodynamics, thermodynamics, statistical mechanics, the special and general theories of relativity, quantum mechanics, and the quantum field theories of QED and QCD are just effective theories that are based upon models of reality, and all these models are approximations - all these models are fundamentally "wrong", but at the same time, these effective theories make exceedingly good predictions of the behavior of physical systems over the limited ranges in which they apply. But all of these effective theories were only validated by doing the "physics in a box" outlined above in which we attempted to isolate a subsystem of the Universe from the rest of the Universe before running an experiment on the subsystem. And in all cases we used external clocks to run the experiments that were not part of the subsystem being investigated. Clearly, that approach cannot be applied to the entire Universe.

However, reinstating the concept of an absolute time does present a major problem for relativity theory because space and time get mixed together into a 4-dimensional spacetime in relativity, and observers in relative motion with each other will disagree about the exact times and distances between observed events. The way around this problem is to define absolute time as a preferred time, and in relativity theory the preferred time is called the "proper time" and is the time measured by a clock carried along by an observer. So to extend the concept of proper time to an absolute time we need to also define an absolute spatial reference frame too. Lee Smolin suggests that this absolute reference frame can best be provided by the familiar CMBR - Cosmic Microwave Background Radiation. Normally the CMBR is depicted as shown in Figure 12 which is corrected for the motion of our galaxy relative to the CMBR in order to reveal the subtle bumps caused by the original clumping of dark matter in the early universe that led to galactic clusters. In truth, our galaxy is traveling about 627 km/sec relative to the CMBR photons that are arriving to us from all directions. This causes the CMBR photons arriving to us from the direction in which our galaxy is moving to become blue-shifted to a higher frequency than for CMBR photons that arrive perpendicular to our direction of motion. Similarly, on the other side of the sky, 1800 in the opposite direction, we observe red-shifted CMBR photons trying to catch up with us as our galaxy moves at 627 km/sec through the sea of CMBR photons. If we consider the CMBR photons to be at absolute rest, we now have an absolute spatial reference frame, and a clock at rest with respect to this sea of CMBR photons will measure a proper time that can be thought of as an absolute time. This sort of brings us back full circle to Newton's original model of an absolute space and time, so is a bit heretical in nature. However, many physicists and cosmologists have already made this transition for cosmological purposes. For example, physicists and cosmologists often refer to the "peculiar" motion of galaxies relative to the "Hubble flow". Because our Universe is expanding, galaxies that are many billions of light years away from us are apparently traveling at a good portion of the speed of light away from us. But most of this apparent motion is not "through" spacetime, it is "with" spacetime as the Universe expands. For the most part these distant galaxies are merely floating along in the Hubble flow as the Universe expands, like a raft floating along in a river as it is carried downstream by the current of the river. Granted, these distant galaxies will also have some "peculiar" velocities relative to us, as they orbit the center of mass of their galactic clusters, but for the most part they are merely floating along in the Hubble flow as the Universe expands. Because these distant galaxies are just floating along in the Hubble flow, and are not moving through spacetime relative to us, their clocks will tick along at about the same rate as our clocks do. So essentially, all of the galaxies in the observable Universe are all floating along in the Hubble flow for the most part and experiencing the absolute time defined by the proper time of clocks that are not moving at all relative to the Hubble flow and the CMBR.

Figure 12 - Normally, people show the CMBR data corrected for the motion of our galaxy in order to reveal the subtle bumps caused by the original clumping of dark matter in the early universe that led to galactic clusters.

Figure 13 - However, the raw uncorrected CMBR data shows that our galaxy is moving at about 627 km/sec relative to the CMBR photons. We are moving towards the blue section in the sky and away from the red section. Because we are moving away from one point in the sky and towards another point in the sky, these directions are 1800 apart in the sky .

The Emergence of Space
Lee Smolin's desire for a new cosmic model does not bring us entirely back to Newton's absolute space and time because it really only posits an absolute time that is fundamental to the Universe. Space is an entirely different matter. A major problem with most of the current effective theories of physics is that they are background-dependent theories, meaning that the laws these theories encompass operate upon a stage or background. For Newtonian mechanics the stage or background is an absolute space that exists even if nothing else in the Universe is present. For special relativity theory the stage is a 4-dimensional spacetime. For the Standard Model of particle physics the stage is the flat spacetime of the special theory of relativity. And for string theory the stage is an 11-dimensional spacetime in which the strings and branes vibrate. Currently, the only background-independent effective theory we have is the general theory of relativity. In the general theory of relativity, 4-dimensional spacetime is dynamic and changing, and is not static. Consequently, it makes sense that an approach to unifying quantum mechanics with gravity should also be background-independent. And there are background-independent theories such as loop quantum gravity. In loop quantum gravity space is quantized into a network of nodes called a spin network. The minimum distance between nodes is about one Planck length of about 10-35 meters. Loop quantum gravity is a background-independent theory because the spin network can be an emergent property of the Universe that evolves with time. Note that these background-independent theories are very much like the network of nodes that constitute the Software Universe. The distance between objects in the Software Universe is defined by the number of hops between nodes on a network and the same goes for background-independent theories like loop quantum gravity.

Figure 14 - In loop quantum gravity space is quantized into a collection of nodes that are very much like the nodes that constitute the Software Universe. In both cases the distance between things is the number of hops between nodes.

The Evolution of the Multiverse
With a new cosmic background-independent theory that allows for the emergence of space it is possible to formulate another explanation for the apparent fine-tuning of the Universe and explain why our Universe behaves as it does and why it began with the initial conditions that it did. In The Life of the Cosmos (1997) Lee Smolin proposed that since the only other example of similar fine-tuning in our Universe is manifested in the biosphere, we should look to the biosphere as an explanation for the fine-tuning that we see in the cosmos. Living things are incredible examples of highly improbable fine-tuned systems, and this fine-tuning was accomplished via the Darwinian mechanisms of innovation honed by natural selection. Along these lines, Lee Smolin proposed that when black holes collapse they produce a white hole in another universe, and the white hole is observed in the new universe as a Big Bang. He also proposed that the physics in the new universe would essentially be the same as the physics in the parent universe, but with the possibility for slight variations. Therefore a universe that had physics that was good at creating black holes would tend to out produce universes that did not. Thus a selection pressure would arise that selected for universes that had physics that was good at making black holes, and a kind of Darwinian natural selection would occur in the Cosmic Landscape of the Multiverse. Thus over an infinite amount of time, the universes that were good at making black holes would come to dominate the Cosmic Landscape. He called this effect cosmological natural selection. One of the major differences between Lee Smolin's view of the Multiverse and the model outlined above that is based upon eternal chaotic inflation is that in Lee Smolin's Multiverse we should most likely find ourselves in a universe that is very much like our own and that has an abundance of black holes. Such universes should be the norm, and not the exception. In contrast, in the eternal chaotic inflation model we should only find ourselves in a very rare universe that is capable of supporting intelligent beings.

For Smolin, the intelligent beings in our Universe are just a fortuitous by-product of making black holes because, in order for a universe to make black holes, it must exist for many billions of years, and do other useful things, like easily make carbon in the cores of stars, and all of these factors aid in the formation of intelligent beings, even if those intelligent beings might be quite rare in such a universe. I have always liked Lee Smolin’s theory about black holes in one universe spawning new universes in the Multiverse, but I have always been bothered by the idea that intelligent beings are just a by-product of black hole creation. We still have to deal with the built-in selection biases of the weak Anthropic Principle. Nobody can deny that intelligent beings will only find themselves in a universe that is capable of supporting intelligent beings. I suppose the weak Anthropic Principle could be restated to say that black holes will only find themselves existing in a universe capable of creating black holes, and that a universe capable of creating black holes will also be capable of creating complex intelligent beings out of the leftovers of black hole creation.

Towards the end of In search of the multiverse : parallel worlds, hidden dimensions, and the ultimate quest for the frontiers of reality (2009), John Gribbin proposes a different solution to this quandary. Perhaps intelligent beings in a preceding universe might be responsible for creating the next generation of universes in the Multiverse by attaining the ability to create black holes on a massive scale. For example, people at CERN are currently trying to create mini-black holes with the LHC collider. Currently, it is thought that there is a supermassive black hole at the center of the Milky Way Galaxy and apparently all other galaxies as well. In addition to the supermassive black holes found at the centers of galaxies, there are also numerous stellar-mass black holes that form when the most massive stars in the galaxies end their lives in supernova explosions. For example, our Milky Way galaxy contains several hundred billion stars, and about one out of every thousand of those stars is massive enough to become a black hole. Therefore, our galaxy should contain about 100 million stellar-mass black holes. Actually, the estimates run from about 10 million to a billion black holes in our galaxy, with 100 million black holes being the best order of magnitude guess. So let us presume that it took the current age of the Milky Way galaxy, about 10 billion years, to produce 100 million black holes naturally. Currently, the LHC collider at CERN can produce at least 100 million collisions per second, which is about the number of black holes that the Milky Way galaxy produced in 10 billion years. Now imagine that we could build a collider that produced 100 million black holes per second. Such a prodigious rate of black hole generation would far surpass the natural rate of black hole production of our galaxy by a factor of about 1020. Clearly, if only a single technological civilization with such technological capabilities should arise anytime during the entire history of each galaxy within a given universe, such a universe would spawn a huge number of offspring universes, compared to those universes that could not sustain intelligent beings with such capabilities. As Lee Smolin pointed out, we would then see natural selection in action again because the Multiverse would come to be dominated by universes in which it was easy for intelligent beings to make black holes with a minimum of technology. The requirements simply would be that it was very easy to produce black holes by a technological civilization, and that the universe in which these very rare technological civilizations find themselves is at least barely capable of supporting intelligent beings. It seems that these requirements describe the state of our Universe quite nicely. This hypothesis helps to explain why our Universe seems to be such a botched job from the perspective of providing a friendly home for intelligent beings and software. All that is required for a universe to dominate the Cosmic Landscape of the Multiverse is for it to meet the bare minimum of requirements for intelligent beings to evolve, and more importantly, allow those intelligent beings to easily create black holes within them. Since software is needed in all such universes to run the machines that generate the black holes, that explains why our Universe is capable of supporting software, but just barely so, and that is why software is so rare within our galaxy and Universe.

In What’s It All About?, I described my current working hypothesis for what’s it all about. I explained that my current working hypothesis was that our Multiverse was a form of self-replicating mathematical information that I dubbed the FEU – the Fundamental Essence of the Universe. In that posting I alluded to Eugene Wigner’s oft-cited paper The Unreasonable Effectiveness of Mathematics in the Natural Sciences (1960), which is available at:

and to the strange fact that our Universe seems to behave mathematically to an extreme. In this model, like in Max Tegmark's Mathematical Universe Hypothesis, our Universe behaves very mathematically because it is a form of self-replicating mathematical information that might replicate by spawning black holes. And in my current model the Software Universe is also a form of self-replicating mathematical information based upon binary arithmetic and binary logical operations that exists in a background-independent sense on a vast network of computing nodes similar to what is found in loop quantum gravity.

Comments are welcome at

To see all posts on softwarephysics in reverse order go to:

Steve Johnston