The Impact of GPT-3 AI Text Generation on the Development and Maintenance of Computer Software

If you are a busy IT Development professional, you probably have not had much time to keep up with the advances being made in AI text generation. But since software source code is just text, you should be paying attention, especially with the arrival of GPT-3 last May. Since Microsoft has since purchased the exclusive rights to the GPT-3 source code, you should definitely be paying attention to how AI-generated computer source code may affect your job in the near future. In order to do that, please view the following two YouTube videos.

From Essays to Coding, This New A.I. Can Write Anything
https://www.youtube.com/watch?v=Te5rOTcE4J4

The next one features an interview with GPT-3 itself in a Turing-Test-like manner. It is a little bit scary.

What It's Like To be a Computer: An Interview with GPT-3
https://www.youtube.com/watch?v=PqbB07n_uQ4

How Does GPT-3 Do It?
GPT-3 uses Machine Learning neural networks with tremendous Deep Learning capabilities to produce coherent text of all kinds, including computer source code. All Machine Learning techniques essentially use the concepts of Universal Darwinism - inheritance, innovation and natural selection to work their wonders. All Machine Learning techniques take some input data and then apply an initial model to the data that tries to explain the data. Next, a selection process is applied by first measuring how well the model explains the data and then keeping those models around that do well. The surviving models are then mutated slightly with some innovation. The mutated models are then inherited by the Machine Learning processes into a new generation of models that are then subjected to the same Machine Learning processes all over again to see if any improvements have been made. This Darwinian process of inheritance, innovation and natural selection is then repeated over and over again until a final model is output.

Developers writing computer source code manually today also use these same concepts of Universal Darwinism to produce code. For example, developers never code up the source code for new software from scratch. Instead, developers take old existing code from previous applications, or from the applications of others in their development group, or perhaps, even from the Internet itself as a starting point and then use the Darwinian processes of inheritance, innovation and natural selection to evolve the software into the final product. The developer currently does this with a very tedious manual process of:

Borrow some old code → modify code → test → fix code → test → Borrow some more old code → modify code → test → fix code → test ....

To understand how GPT-3 simulates the above process, begin with some very excellent YouTube videos by Luis Serrano. If you are totally new to Machine Learning, begin with this first video which covers all of the major machine learning approaches currently being followed:

A Friendly Introduction to Machine Learning
https://www.youtube.com/watch?v=IpGxLWOIZy4

The next video covers neural networks and the Deep Learning that large neural networks can perform. GPT-3 relies heavily on many Deep Learning neural networks that have basically read everything on the Internet and have taught themselves how to talk all by themselves, just like a human toddler does by listening to the people around it.

A Friendly Introduction to Deep Learning and Neural Networks
https://www.youtube.com/watch?v=BR9h47Jtqyw

The next video explains how recurrent neural networks use feedback loops, like the feedback loops found in electronic circuits and biochemical reactions, to stabilize output.

A Friendly Introduction to Recurrent Neural Networks
https://www.youtube.com/watch?v=UNmqTiOnRfg

The next video explains how generative neural networks with positive feedback can be paired with neural networks with negative feedback. This same mechanism is used to control the expression of genes stored in DNA.

A Friendly Introduction to Generative Adversarial Networks (GANs)
https://www.youtube.com/watch?v=8L11aMN5KY8

Now with the above background, next take a look at the following excellent posts by Jay Alammar.

Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)
https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/

The Illustrated Transformer
https://jalammar.github.io/illustrated-transformer/

Finally, read this last post that explains how GPT-3 uses all of the above to work.

How GPT3 Works - Visualizations and Animations
http://jalammar.github.io/how-gpt3-works-visualizations-animations/

How Will GPT-3 Affect My Programming Job?
I don't think that GPT-3 has brought us to the Software Singularity yet, but we are getting closer. Here is a YouTube video that seems to have a good take on the subject:

GPT3 - Will AI replace programmers?
https://www.youtube.com/watch?v=u5MmL3nqvfE

One commenter wisely noted, "I'm not scared about GPT-3. I'm scared about GPT-5.".

But I can imagine Microsoft incorporating GPT-3 into their Visual Studio IDE. Since GPT-3 is much too large to run on a single PC and needs to run in the Cloud, Microsoft Visual Studio will need to run in a metered manner with Cloud support for GPT-3, GPT-4, GPT-5... Also, GPT-3 can be fine-tuned for specific computer languages to vastly improve performance. Currently, GPT-3 is just generating code based on the code that it randomly read on the Internet by accident. So I think that GPT-3 will only slightly alter the normal development cycle by removing lots of the tedious aspects of coding that really do not require actual thinking. It will probably look more like this:

Generate some code with GPT-3 → modify code → test → fix code → test → Generate some code with GPT-3 → modify code → test → fix code → test ....

The reason I say this is because, back in 1985, I began working on an IDE with a built-in code generator that was called BSDE - the Bionic Systems Development Environment. This was while I was a programmer in the IT department of Amoco. BSDE was an early mainframe-based IDE (Integrated Development Environment like Eclipse or Microsoft Visual Studio) at a time when there were no IDEs. During the 1980s BSDE was used to grow several million lines of production code for Amoco by growing applications from embryos in an interactive mode. BSDE would first generate an embryo application for a programmer by reading the genes for the application. The application genes were stored on a sequential file containing the DDL statements for the DB2 tables and indexes that the application was going to use to store data. The embryo was then grown within BSDE by the programmer injecting reusable BSDE skeleton code segments into the embryo and by turning on the embryo's genes to generate SQL code on the fly. This continued on until the application grew to maturity and BSDE delivered the completed application into Production. About 30 programmers in Amoco's IT department used BSDE to put several million lines of code into Production. The development process looked something like this:

Generate some code with BSDE → modify code → test → fix code → test → Generate some code with BSDE → modify code → test → fix code → test ....

For more about BSDE take a look at Agile vs. Waterfall Programming and the Value of Having a Theoretical Framework.

How Will GPT-3 Affect the Rest of Society?
Since most white-collar jobs deal primarily with reading and writing text and that text does not have to be as perfect as computer source code has to be, other white-collar jobs, beyond those in IT, are probably at more risk from AI text generators than are IT jobs. As I pointed out in Is it Finally Time to Reboot Civilization with a New Release?, Oligarchiology and the Rise of Software to Predominance in the 21st Century and The Danger of Tyranny in the Age of Software society will soon need to come to grips with trying to run civilizations in which most people do not work as we currently know it. In that regard, perhaps your next job interview will be conducted by a GPT-3 avatar!

Comments are welcome at scj333@sbcglobal.net

To see all posts on softwarephysics in reverse order go to:
https://softwarephysics.blogspot.com/

Regards,
Steve Johnston

Agile Development of Molten Salt Nuclear Reactors at Copenhagen Atomics

In Last Call for Carbon-Based Intelligence on Planet Earth, I explained that we have just about run out of time in regards to climate change possibly putting an end to carbon-based Intelligence on the Earth. This conclusion is an outgrowth of my Null Result Hypothesis explanation for Fermi's Paradox.

Fermi’s Paradox - If the universe is just chock full of intelligent beings, why do we not see any evidence of their existence?

Briefly stated:

Null Result Hypothesis - What if the explanation to Fermi's Paradox is simply that the Milky Way galaxy has yet to produce a form of interstellar Technological Intelligence because all Technological Intelligences are destroyed by the very same mechanisms that bring them forth?

By that, I mean that the Milky Way galaxy has not yet produced a form of Intelligence that can make itself known across interstellar distances, including ourselves. I then went on to propose that the simplest explanation for this lack of contact could be that the conditions necessary to bring forth a carbon-based interstellar Technological Intelligence on a planet or moon were also the same kill mechanisms that eliminated all forms of carbon-based Technological Intelligences with 100% efficiency. One of those possible kill mechanisms could certainly be for carbon-based Technological Intelligences to mess with the carbon cycle of their home planet or moon. For more on that see The Deadly Dangerous Dance of Carbon-Based Intelligence. But in Last Call for Carbon-Based Intelligence on Planet Earth, I also explained that we still had a chance to stop pumping carbon dioxide into our atmosphere by using molten salt nuclear reactors to burn the 250,000 tons of spent nuclear fuel, 1.2 million tons of depleted uranium and the huge mounds of thorium waste from rare earth mines. From the perspective of softwarephysics, this is important because the carbon-based Intelligence on this planet is so very close to producing a silicon-based Intelligence to carry on with exploring our galaxy and making itself known to other forms of Intelligence that might be out there.

Figure 1 – A ball of thorium or uranium smaller than a golf ball can fuel an American lifestyle for 100 years. This includes all of the electricity, heating, cooling, driving and flying that an American does in 100 years. We have already mined enough thorium and uranium to run the whole world for thousands of years. There is enough thorium and uranium on the Earth to run the world for hundreds of thousands of years.

How To Make Molten Salt Nuclear Reactors a Reality
Over the past decade, many organizations have begun research and development programs to produce molten salt nuclear reactors. China and India have both adopted a government-led NASA-type approach to the problem using Waterfall-like methodologies. A number of private concerns throughout the world, most notably ThorCon, are also using a Waterfall approach to produce a design for a large-scale molten salt nuclear reactor that can attain certification by governmental regulatory bodies like the NRC. Once certification by a governmental regulatory body has been attained, it is hoped that it should be possible to find the necessary private funding to build molten salt nuclear reactors on a massive scale.

Copenhagen Atomics Takes an Agile Approach
However, in this posting, I would like to showcase the efforts of a small startup called Copenhagen Atomics. Unlike the other organizations working on molten salt nuclear reactors, Copenhagen Atomics is using an Agile approach. Copenhagen Atomics is applying the same Agile research and development techniques to the production of molten salt nuclear reactors that SpaceX is currently using for the research and development of private spaceflight - only without the benefit of huge amounts of money. For more on the amazing efforts of SpaceX see Agile Project Management at SpaceX. Below is a recent video of Thomas Jam Pedersen, one of the founding members of Copenhagen Atomics, at TEAC10 explaining the Agile techniques that Copenhagen Atomics is using.

CA @ TEAC10, Oak Ridge - Thomas Jam Pedersen
https://www.youtube.com/watch?v=9nN4txzie8M

Rather than spending a great deal of time and effort on Waterfall design documents, Copenhagen Atomics is spending their time and effort on actually "building stuff" in an Agile trial and error manner. For example:

1MW Molten-Salt Test Reactor by Copenhagen Atomics - Aslak Stubsgaard @ ORNL MSRW 2020 https://www.youtube.com/watch?v=A9zfYTWjZqk

Now compare the Copenhagen Atomics' presentations to the 1969 film produced by the Oak Ridge National Laboratory for the United States Atomic Energy Commission that describes the Molten Salt Reactor Experiment (MSRE) and how Alvin Weinberg's team of 30 nuclear scientists built the very first experimental molten salt nuclear reactor from scratch in an Agile manner with only $10 million during the period 1960 - 1965 and then ran it for 20,000 hours from 1965 - 1969 without a hitch. As I mentioned in Agile vs. Waterfall Programming and the Value of Having a Theoretical Framework and Agile Project Management at SpaceX, back in the 1950s and early 1960s technical people used a more Agile approach to solving complex problems rather than using the Waterfall approach that arose in the 1970s.

The Molten-Salt Reactor Experiment
https://www.youtube.com/watch?v=tyDbq5HRs0o#t-1

More Copenhagen Atomics YouTube videos are available on their YouTube Channel:

Copenhagen Atomics YouTube Channel
https://www.youtube.com/c/CopenhagenAtomics

Funding the Copenhagen Atomics Approach
Like SpaceX, Copenhagen Atomics is partially funding its Agile research and development program by developing and building component products for molten salt nuclear reactors that can then be sold to other organizations.

Figure 2 – Copenhagen Atomics is partially funding their research and development efforts by developing and building component products for molten salt nuclear reactors. Above we see some team members sitting on portable test molten salt loops that can be used to test the interaction of molten salts with test components. Since working on molten salt nuclear reactors basically involves dealing with high-temperature plumbing, much research can be conducted with molten salts that are not even radioactive.

The ultimate goal of Copenhagen Atomics is to produce small 50 MW molten salt nuclear reactors the size of a 40-foot shipping container that can burn spent nuclear fuel.

Figure 3 – A schematic for a 50 MW molten salt nuclear waste burner.

Figure 4 – These proposed 50 MW molten salt nuclear waste burners would be the size of a 40-foot shipping container and would be constructed on assembly lines ready for shipment to customers by ship and truck.

Figure 5 – These proposed 50 MW molten salt nuclear waste burners would use a thorium-232 fuel cycle.

Figure 6 – The neutrons from plutonium-239 and plutonium-240 from spent nuclear fuel rods would be used to get the thorium-232 fuel cycle going. Additional spent nuclear fuel could also be added to provide additional energy and to turn the spent fuel into fission products that only need to be stored for 10 half-lives or 300 years.

Time For Action
Last fall, I met with Congressman Raja Krishnamoorthi of Illinois about molten salt nuclear reactors and the possibility of using them to turn the 10,000 tons of spent nuclear fuel currently being stored at nuclear reactors in Illinois. Instead of being a burden, these 10,000 tons of spent nuclear fuel could be turned into electricity and fission products that only need to be stored for 300 years. I explained that currently, Illinois is committed to storing the 10,000 tons of spent nuclear fuel for about 200,000 years. Of the 535 members of the 116th Congress of the United States of America, we only have 11 members with a background in engineering or science. Fortunately for me, Congressman Raja Krishnamoorthi received a B.S. in mechanical engineering from Princeton and a law degree from Harvard and has had some experience with solar energy and is also very concerned about climate change. I also passed on my concerns about these issues to both of my senators.

Comments are welcome at scj333@sbcglobal.net

To see all posts on softwarephysics in reverse order go to:
https://softwarephysics.blogspot.com/

Regards,
Steve Johnston