Wednesday, April 08, 2020

Scenes From the COVID-19 and Y2K Pandemics

In my last posting, A Structured Code Review of the COVID-19 Virus we took a detailed look at the COVID-19 virus and the RNA software within that makes it work. We also discussed the power of self-replicating information to quickly and dramatically rework the surface of an entire planet. In this posting, I would like to continue on with that analysis by comparing the worldwide preparations that were made for the current COVID-19 pandemic with the worldwide preparations that were made for the Y2K pandemic of the year 2000. As outlined in Life in Postwar America After Our Stunning Defeat in the Great Cyberwar of 2016, the United States of America is now left with a feckless leader without the basic scientific knowledge to be of much use. This leader is now blaming the World Health Organization and a national stockpile with "empty shelves" as the guilty culprits. This, after being in office for more than three years and having a CIA and a Department of Homeland Security at hand! However, many informed people, like Bill Gates, knew better:

Bill Gates TED presentation March 2015 - The next outbreak? We're not ready
https://www.ted.com/talks/bill_gates_the_next_outbreak_we_re_not_ready?language=dz#t-119430

My Personal Experiences With the Y2K Pandemic
In many respects, a viral respiratory pandemic is much like the Y2K bug of the late 1990s that the worldwide IT community successfully fixed. My son was born early in 1981, and later that year I was having a discussion with my stockbroker about funding his college education. My broker explained that I could buy discounted stripped U.S. Treasury bonds for something like $125 that would pay out $1,000 in my son’s college years. Because inflation was raging at over 11% per year in those days, I could essentially lock in zero-coupon bonds that were insured by the Federal government at a guaranteed rate of 12% interest and which could not be called before they matured in the far distant future. So I bought a bunch of stripped U.S. Treasury bonds that matured in 1999, 2000, 2001, and 2002. While I was on the phone with my broker, I commented that it seemed so strange to be talking about years like 2000, 2001 and 2002 because both of us had only dealt with 20th century years like 1965, 1966 and 1967 for our entire lives. When I got off the phone, I had one of those “uh-oh” moments, as I thought about all the code that I had written with two-digit years that read and wrote files containing dates like 101265 for October 12, 1965. Doing arithmetic in my code, like 65 – 51 = 14 years, worked just great in the 20th century, but would not work for 2002 – 1998 because 02 – 98 was going to yield -96 years instead of 4 years! That’s when the Y2K bug first hit me, but this was back in 1981, so I figured that somebody else would surely fix it all in the far distant future. So not to worry.

Now scroll forward to late 1996. I got a call from Amoco’s first Y2K Czar asking me if I would like to join Amoco’s Y2K project. I had written Amoco’s Application Portfolio System with BSDE back in the mid-1980s, so I was familiar with collecting lots of data on Amoco’s systems, and that’s how I got drafted for the Y2K project. Amoco’s first Y2K Czar brought in a consulting company to scan our source code libraries. The initial scans revealed that we indeed had a very serious problem. Like all the other IT departments around the world, we had strewn our systems with millions of logic bombs all set to go off at the same time as we approached the year 2000. We then realized that the Y2K bug represented a true worldwide IT pandemic that could bring down the entire world over a 24 hour period on January 1, 2000, as the Earth slowly turned on its axis. The consulting company we brought in then racked up some pretty serious bills scanning our code, but unfortunately, it was totally clueless about how to fix the code because all of the affected systems were intimately tied together into a huge IT processing knot. You could not simply fix one system at a time because the systems exchanged data with each other via files and databases, so you had to carefully remediate groups of related systems at the same time. That required an intimate knowledge of Amoco's systems that the consulting company did not have. Meanwhile, our Y2K group could not get much help from the Applications Development groups because they were all just trying to survive through the day and were not thinking much past their next install weekend. Besides, our Y2K group had that pricy consulting company that was going to do all the work and fix everything for them! People were still pretty much in denial about the Y2K bug back in 1996.

This all might sound rather petty, but there was a lot of money on the line. At the time, all of Amoco's IT costs were charged out to the Amoco business units, and the Amoco business units all strove to show a profit to the Amoco Corporation holding company. Those business unit managers who could not show a profit frequently chose to spend more time with their families! The Amoco business units always complained about their very expensive IT charges for manpower and computer time. So the Amoco business units were not in any mood to spend millions of dollars on fixing the Y2K bug. The Amoco business units wanted to spend their IT budgets on new IT development projects that could reduce their business costs or increase their business revenues. The Amoco business units viewed the Y2K bug as an example of gross negligence on the part of Amoco's IT department! To resolve this budgeting problem the Amoco Corporation holding company had to set up a special "slush fund" budget for Y2K remediation that did not come out of the pockets of the individual Amoco business units. As Cyndi Lauper wisely noted, "Money Changes Everything". Money is also a very important issue in the current COVID-19 pandemic. For example, the current leader of the United States of America has chosen to push the remediation of the COVID-19 virus down to the state governors to handle. But the state governments of the United States of America are mostly broke or in debt and cannot afford to do the COVID-19 remediation. The state governors will need lots of federal money to bring back the American economy.

All along, our first Y2K Czar kept telling our CIO that the glass was half full, but that we were making steady progress. But in 1997 the Y2K bug began to appear in the IT trade rags, and that got our CIO’s attention. Suddenly our CIO realized that his glass was not half full, it was actually half empty! So our first Y2K Czar was summarily terminated for cause and escorted from the building! Things were a little tense back in those days on the Y2K projects around the world. A few blocks away from Amoco’s headquarters, at CNA insurance, two members of their Y2K team got into a fistfight outside of an elevator and were immediately terminated under CNA’s zero-tolerance policy! Amoco’s second Y2K Czar then came in with both barrels blazing. He immediately fired the old consulting company and hired one of the big-gun consulting companies to take its place and come in and finish (really start) the job. Suddenly there were hundreds of young kids swarming all over our Applications Development groups trying to apply the consulting company’s brand-new Y2K methodology. The burn rate for this effort was a little over $2 million/month, and after a few months of that, our CIO decided that our second Y2K Czar should spend some more time with his family. Amoco’s third Y2K Czar came in with a completely different attitude. Instead of charging off in a mad rush in the arms of a consulting company, we spent about a month just sitting around trying to figure out how we could get ourselves out of this mess all by ourselves, without using consulting companies at all. By then we had finally figured out that the consulting companies really did not know how to fix the Y2K bug at all. Out of these brain-storming sessions, we developed an overall strategy to push the fixing of the Y2K bug down to the grass-roots level of the Applications Development groups within Amoco because they were the only ones who knew Amoco’s systems.

So we split up Amoco along its subsidiary lines. I had the Amoco Corporation holding company and its Amoco Production Company subsidiary, for a total of about one third of Amoco’s total number of applications. That came to managing the Y2K remediation plans for about 1500 major corporate applications. I had about a dozen Y2K sub-coordinators under me and each of them had a Y2K sub-coordinator under them in each of the Applications Development groups. Our Y2K group then set up the policies and procedures to do the Y2K remediation of Amoco’s software and provided the tools to do the work, but it was the responsibility of each Applications Development group to remediate the software that they supported. We also researched many of the Y2K conversion tools that were coming on the market and came up with a list of software tools to help programmers convert Cobol, PL/1, Fortran and C programs to Y2K-compliant code.

Like COVID-19, the degree of susceptibility of software to the Y2K bug varied greatly. Lots of scientific and engineering software did not do any date processing at all. Such applications could go through the Y2K century rollover without displaying any symptoms at all. Other applications could go through the Y2K century rollover because they just displayed dates on screens or reports. Depending on the programming language, the code just called a date() function to get a date and then would substring the last two characters of the year. So the software would display dates like 05-12-99 or 05-12-00 just fine. For such applications, the Amoco business unit manager and their corresponding IT Applications Development manager could sign a Y2K waiver. The real problem was locating the software that actually did date calculations and harbored the Y2K bug.

The Y2K group offered two remediation options for those applications that had a real problem with the Y2K bug. The preferred option was called "date expansion". With date expansion, all code, files and databases were modified to use a date format with a full 4-character year like 1965. That meant that code, files and databases using a date format like 010565 for January 5, 1965, had to change to handle a new date format of 01051965. That might sound easy, but it was not. The second option was to use "century-windowing". With century-windowing, the code, files and databases could still use a 2-character year. The century-windowing trick was to introduce subroutines into the old code that did proper date manipulations for a century window of 1950 - 2049. Since computers did not really roll into corporations until the mid-1950s, that meant that hardly any files or databases existed with dates having years earlier than 1950. So these century-windowing subroutines could do things like calculate that 2002 – 1998 was 4 years instead of -96 years. However, nearly all of Amoco's applications were made Y2K-compliant using the date expansion approach.

The Amoco Y2K group was also responsible for ensuring that Y2K-compliant vendor hardware and software products were installed at all sites. This was not too difficult because, by 1997, most vendors were marketing the need for all to upgrade to their latest Y2K-compliant hardware and software products. That also took lots of money that nobody wanted to spend. We even began to see things like Y2K-compliant cables being sold by vendors preying upon the Y2K hysteria that was seen in all IT departments around the world! There even was a joke going around our IT department that the Whiting Refinery insisted on only buying Y2K-compliant sand for construction projects.

One of the tools our Y2K group provided was a database application to keep track of the Y2K remediation efforts for each application. The first thing we did was to classify each application by criticality – High, Medium, or Low and then to focus on the High and Medium applications first. For Y2K certification testing, we created a mainframe LPAR and a Y2K lab filled with Unix and Windows servers. Applications that needed Y2K remediation were first remediated by their Applications Development group and then spent two weeks of testing in our Y2K lab to obtain Y2K-certification. A total of 32 critical Y2K-dates were tested by changing the system clocks on the mainframe LPAR and the Unix and Windows servers during a two week period. Y2K-certified applications then had antibodies for the Y2K bug and could be returned to Production.

The Y2K lab was hermetically sealed and completely isolated from Amoco's IT Production processing facilities. There was a great fear that Y2K contaminated software or data might escape from the Y2K lab that might infect Amoco's Production environment. The first date was for the infamous September 9, 1999, or “090999” problem. You see, lots of programmers came up with the brilliant trick of using a date of “090999” to signify something special, like the last record in a file, so we had to test for that condition. Naturally, the date January 1, 2000 (010100) was in the list of dates.

This new Y2K strategy of Amoco doing its own Y2K remediation and using Applications Development to do the work in a massively parallel manner really worked and Amoco finished up its Y2K remediation by early 1999, just in time for its take over by BP! Again, doing things in a massively parallel manner is a hallmark of taking a biological approach to solving IT problems as outlined in How to Think Like a Softwarephysicist.

Y2K Finally Arrives
However, in 1999 I was not completely confident that the rest of the world had been as successful in Y2K remediation as Amoco. After all, even at Amoco, we had a very rocky start for our Y2K remediation project. So I set up a Y2K "fallout shelter" in my basement for my family similar to the fallout shelters of the 1950s and 1960s as outlined in Cyber Civil Defense. My Y2K "fallout shelter" had a large stockpile of candles, matches, bottled water, canned and dried foods, a gasoline-powered camp stove, batteries, flashlights and a battery-operated radio. For water, I had about 100 gallon milk jugs filled with tap water. To each jug, I added 8 drops of chlorine bleach as a preservative. Since the Y2K century rollover would be occurring in the middle of a Chicago winter for us, the plan was to move down to the basement if we lost electricity and heat. The natural warmth from the ground would make our basement the warmest part of the house. I also bought some gold and silver American Eagle coins in case paper money lost its value. This may sound a bit alarmist, but at the time, I had no idea what was going to happen on January 1, 2000.

When Y2K finally arrived, I was in the IT department of United Airlines supporting a Tuxedo client/server application that was used by the http://www.united.com/ website to display flight information. Like most members of the IT department of United Airlines, I was on call for the century rollover on December 31, 1999. The United Airlines IT Command Center was carefully monitoring all of our worldwide IT operations as the century rollover proceeded slowly around the entire planet. Not only did we have to worry about our own software, but we also had to worry about all of the other software in the world being able to properly handle the century rollover. To our great relief, nothing happened! As I recall, United Airlines only had a little problem with some signage displaying wrong dates in the Denver airport.

To the surprise of nearly all, the entire worldwide IT infrastructure came through Y2K with very little damage. The electrical grid stayed up, the worldwide financial system did not collapse, grocery store shelves did not empty, the stock market did not crash, the unemployment rate did not rise to 30% and water still came out of the tap when you turned on a faucet. All of that could have happened if it were not for the efforts of millions of IT professionals working on the Y2K bug for several years in advance of the century rollover. But because the Y2K rollover went so smoothly around the whole world, people actually started to make jokes about the Y2K bug after the fact. Some people actually began to claim that the Y2K bug was some kind of IT "hoax" designed to line the pockets of IT workers. They claimed that the Y2K bug was a "fake" disaster concocted by IT workers. It seems that no good deed goes unpunished.

Always Be Prepared
Now compare the aftermath of the Y2K pandemic that really did not happen with the death and devastation currently being caused by the current worldwide COVID-19 pandemic. Y2K could have caused the same level of damage that COVID-19 is currently inflicting, but Y2k did not do so because of a great deal of preparation by the global IT community. The global IT community saw a great threat in advance and spent the necessary time and money to fix the problem in advance before it could cause worldwide damage. As Bill Gates pointed out in his TED Talk listed above, the leaders of the world and the public health organizations and public health workers of the world need to come together now to build a robust system to handle the rogue parasitic viral RNA and DNA molecules that have plagued mankind for so long and that could produce an even worse pandemic in the future.

Comments are welcome at scj333@sbcglobal.net

To see all posts on softwarephysics in reverse order go to:
https://softwarephysics.blogspot.com/

Regards,
Steve Johnston

No comments: