On LLMs and Software Engineering

It's only reasonable for me to talk about LLMs given its enormous impact on the space. I have a lot of thoughts on the effects of LLMs on society, but for this essay, I’d like to focus on software engineering specifically.

I started coding when I was in middle school, and, ever since, I’ve been genuinely passionate about computers. I have spent much time thinking about and studying computers. I thoroughly acknowledge what can be described as a rapid transformation of a deeply human and established practice — especially in the context of a practice that many people including myself have built a livelihood around. I feel the space is in a great deal of chaos in terms of how to make sense of what is happening, and I feel I am well-suited to explain and provide what I view as a clear-eyed perspective of the situation. I also feel obligated to remind others that the future is not fixed and that society has the right of self-determination — for better or for worse, but nonetheless sovereign.

As always, I think it’s illuminating to understand the fundamentals and reason deductively. If you look at the foundation of mathematics and computer science, it is about language and models — and it’s no coincidence that these two words appear in the acronym LLM. Language is used to model anything by representing that concept as a token/symbol and inference/rules allow you to manipulate those concepts. For example the number 1, represents the concept of one. Physics is a study of how matter behaves often described by the language of mathematics. LLMs have been modeled and trained on the relationships between all of human language and have formed an algorithm that produces very cogent sequences of text. They don’t seem to have an absolute ability to model any system internally as evidenced by their inability to strictly follow the rules of chess or it’s need to rely on Lean to extend deductions in mathematical proofs. But, it does have the ability to generate code, and with delegation to pre-existing programs, is able to synthesize information between executions to compensate. I can only assume that by being trained on the corpus of all publicly available text that it has the uncanny ability to produce sequences of text that strongly resonate with people and, through execution, produce outcomes which can be deemed correct. Within that system, I assume, there is an underlying structure/relationship that models software patterns and also how natural language maps to those concepts.

And so that brings me to software engineering — in the context of businesses, what value do engineers bring to the table and how has that changed? The answer has always been understanding how a software system works and the consequences of decisions within those systems while bound by the standard constraints of a business. But the unfortunate truth is that decisions that have lower risk in a business have historically been delegated to “junior” engineers because their understanding of a system maybe insufficient to avoid potential consequences. But, writing that code still took a significant amount of time and effort, and decisions had to be made in some form despite its lower impact on the business. In addition, teams often have divisions that naturally “sandbox” responsibilities and scope. Now, businesses are all too willing to allow LLMs to make decisions with software that are low risk in the eyes of those with authority for the sake of expediency. And slowly, it creeps up the chain as people realize that either their mental model of the system is sufficient for getting what they need out of a LLM using natural language or they pass it to another person who has a different mental model. In the extreme case, based on accounts I’ve read, a change process workflow has been established but humans have removed themselves entirely from the loop and they hope that the system is stable enough to adapt to changing business requirements with effectively no engineering input — at least in theory. But programming languages are exact while natural languages are ambiguous and that difference means that either an engineer or a LLM will need to resolve that ambiguity in a system. Ultimately, engineers know how to resolve such ambiguity in a way that is maintainable and sane for humans. LLMs will resolve such ambiguity in potentially inscrutable ways attempting to satisfy the request. I see similarities with the controversy surrounding OpenAI’s claimed proof of the Navier-Stokes existence and smoothness problem. With formal verification, LLMs are able to extend proofs consistently but not necessarily in any fashion which is digestible to humans. In cases where formal verification is viable, the inscrutability is irrelevant if the goal is simply the result. For engineers, the practice is often a means in itself. For business, it’s a means to an end. But otherwise, a software system modified solely by LLMs using natural language without human verification would likely struggle to keep consistent within an informal problem domain. How would such a system handle natural language requests that are fundamentally contradictory? Should another LLM instance simply decide on behalf of the organization? If no person is bothering to resolve this, then the group has collectively decided to let the machine pull the lever. Despite this constraint, some systems within organizations maybe stable under current operating conditions until the time-honored tradition of having to rewrite and migrate the codebase when no one can continue to make progress in changing the system without violating constraints.

Ultimately, in the context of people, it’s shifting dependence and trust on an engineer’s mental model to a machine’s ephemeral internal model fused with a user’s prompts in the hope that the latter is stable and sufficient to operate a business. That’s tough because you don’t get to build a relationship with a LLM where you have a shared understanding, accountability, and solidarity. Instead, you’re working with an entity that is unnervingly confident and completely unaccountable. LLMs sever trust relationships within teams, because you’re no longer guaranteed that the other party has put in the time to do the work and understand what has been produced. Before, a review showed exactly how a developer understood a problem and their consistent patterns and growth could give confidence on what problems they were comfortable with and what they were not. If they made a mistake, they were given grace. Now, people are being pushed to trust their livelihood on the output of a machine that allegedly may also kill all of humanity. That sounds like insanity.

I have strong skepticism about this approach in terms of overall efficiency gains within existing organizations. I have general reservations about LLMs in society that I’ll detail in a later essay.

But in the meantime, if LLMs are here to stay, people should recognize software engineering should be reorganized in a more humane way and not have LLMs’ inhuman ability to generate code from natural language remove the joy and sanity in engineer’s lives.