Many years ago (over a decade, in fact), in the space of this very blog, I spoke about why Kodak lost is way (along with its market share) by focusing on ‘the how’ versus ‘the what’. Insisting that they were a company that made photographic emulsion films rather than a company that embraced their founder’s goal of capturing memories, Kodak allowed other companies to market their ideas, take their customers, and, eventually, almost completely marginalize a once titan of industry.
In the years that have followed that post, I have become more and more convinced of the brutal truth behind the Kodak lesson. Keeping ‘the what’ separate from the ‘the how’ and then treating both of them appropriately (without mixing ideas and methods designed for one with the other) is vital to the success of any endeavor as the next two stories show. In the first vignette, we’ll see a cautionary tale about how slavish devotion to a ‘how’ undermined ‘what’ a software project was trying to do. The second vignette, plucked from the most recent political headlines, weaves the story of how a quasi-government agency, with grandiose dreams of achieving a ‘what’, failed to focus on the ‘how’ which ultimately doomed it to failure.
Story #1 – Too Much ‘How’
A few days ago. I sat down to eat lunch. And, as is my wont, I fired up YouTube on my phone and decided to watch a short video. One particular thumbnail (mysteriously suggested by the algorithm) caught my eye:
The title of the video, Why Spaghetti Code Beat Clean Architecture, was nearly as provocative and the whole story, told by one Eric Roby, was riveting for the entire 15 minutes run time. Eric’s story is about a colossal failure he was forced to be a party of when he was a junior developer working for an online retailer with an annual revenue of roughly $50 million.
Eric was assigned to a team that was going to clean up the old, ‘horrible’, ‘spaghetti’, legacy code into some brilliant, shiny new ‘clean code’ example. Steven, the team lead, was a subject matter expert brought in to set the vision (based on his “why code quality is your competitive advantage” message), motivate the team, and supervise the development and deployment of a new system. Central to Steven’s approach was the ‘clean code’ idea in which the level of abstraction and esthetic look-and-feel of the code were of paramount importance. The code needed to have “good patterns” and “solid test coverage”.
Eric says that he was initially impressed by Steven’s approach but the lead’s way of presenting seemed to be “confidence that borders on dismissiveness” – a foreshadowing of things to come.
Eric began looking at the legacy code as directed by Steven. He found that while it is true that the old system's code was nearly impossible to read and maintain without a huge investment in a learning curve it was also true that it worked really, really well. Over the previous Black Friday, the legacy code handled 85,000 concurrent users, fulfilled over 77,000 orders, all with a latency of only about a quarter of a second, and an uptime of 99.4%. In addition to this sterling performance, the legacy system provided the development team a huge number of metrics to monitor and diagnose system health. When he shared these observations, Steven brushed them aside saying that all that infrastructure was the legacy system’s way of compensating for the bad way it was written and that clean code didn’t need that.
As development and testing proceeded, Eric again tried to sway Steven on two subsequent occasions. First, Eric meticulously documented many side-by-side comparisons between the clean and legacy codes which showed that the legacy code had more metrics and more observability, indicating that the legacy code was more resilient. Steven’s response was that Eric was over-engineering. Later Eric pointed out that factory testing used a way too small and far too sanitized group of users to be a valid test. He was told the “don’t let perfect be the enemy of good-enough” line.
As is often said, pride foes before the fall, and the fall came in Autumn as the firm was beta testing in advance of Black Friday. The beta test consisted of running the clean code for 5% of the users (maximum of maybe 1,000) and monitoring its performance. For a very brief time, the system looked like it might be working but soon reports came in from the help desk of long checkout times and other errors. Shortly after it became clear that the clean code version was a bust when it consumed all available resources and effectively crashed. This embarrassing fail was further compounded by the fact that without ample metrics there was no clear way of knowing where the problems was. The firm reverted back to a sole dependence on the legacy code, essentially admitting that their clean code experiment was a colossal failure.
Eric explains this failure on the fact that Steven emphasized one and only one ‘pillar’ (clean code) of the four he believes need to be addressed – the others being infrastructure & scaling, observability, operational culture. Of these, Eric emphasizes the failures in the last pillar, the lack of an operational culture. The firms culture “treated [Steven’s] confidence as expertise”, “rewarded elegance over resilience”, and “treated production readiness as a nice-to-have”. In other words, the firm focused on ‘the how’ and not on ‘the what’ and paid the price.
Story #2 – Too Much ‘What
Our second story comes from one of the most controversial political situations to emerge from the Trump 2.0 administration: The Department of Government Efficiency, aka DOGE. Just after President Trump's election in November of 2024, there was this incredible enthusiasm and energy associated with having the Department of Government Efficiency being formed, being led by Elon Musk and Vivek Ramaswamy, and that this agency was going to go and clean house across the government.
Having worked around the federal government for over 30 years and being frustrated by the sclerotic pace of the bureaucracy, I actually considered the possibility of volunteering for DOGE. After all, I knew where all the bodies were buried and I had very strong ideas about how to make government efficient, having learned how to navigate through various useless processes and procedures. I ultimately demurred because of the possible conflict of interest, but I was enthusiastic that DOGE would be able to produce the kind of fiscal responsibility I believe the government needs, or at least some fiscal responsibility.
The country’s initial reaction to DOGE just after President Trump’s inauguration can be best described as ‘shock and awe’, borrowing a phrase from Desert Storm. DOGE claimed to be routing out billions of dollars of waste, well on its way to the ultimate goal of 2 trillion dollars.
However, if one were to query current headlines a mere 8 months later using the search string “Musk DOGE”, one comes up with a host of articles with headlines that essentially say quote Elon Musk saying “DOGE somewhat efficient, but I wouldn't do it again”.
Skimming any of the articles, looking past all the bloviating and padding the standard reporter does in order to get their word count up, one sees that each article contains two basic points: 1) DOGE was nowhere near as efficient or as sweeping in its changes as Musk had hoped it to be and 2) Musk had anticipated it would have worked properly because it had worked in all his companies in the past.
What is the reason for this even more colossal failure than the one in Story #1 above? Put simply, DOGE focused on ‘the what’ of reducing governmental bloat without ever stopping to ask themselves how they would recognize the bloat when they see it. When Musk took over Twitter, as big as that company might have been, as many computers and employees and mocha lattes and yoga rooms that it might have had under Jack Dorsey, it was still fundamentally a simple company with a simple single vision. A user types something stupid or wise or fun or mean or whatever and then tweets it to the universe. That was it. That's all there was to it. SpaceX also has only one simple, solitary vision. Put some high explosives underneath a payload made of metal and plastic and silicon and ignite them to push the payload off the Earth (and hopefully not blow it up). Simple, clean, straightforward. In other words, even if Musk had a million people working for him, he only had to navigate a few corporate visions.
The federal government, in contrast, has not only a million people literally working for it, but it has millions of visions, many of them often competing with or in contradiction to other visions by the very nature of the democratic process. Musk's team failed to take into account the nature of what they were trying to change. They failed to take into account ‘how’ they would implement those changes. Being charitable, all they wanted to do was make government smaller and more efficient, a laudable goal, but they failed miserably because they were unable to understand how to make the changes they claimed they wanted.
To close, what is the lesson we can draw from these tales of woe and incompetence? The answer is it's important in any endeavor to sit and think clearly about ‘what’ is being done and ‘how’ it should be accomplished. The more complicated the task is, the greater amount of care that is needed in making these distinctions. Focusing too much on a particular ‘how’ ends up with the project falling into the trap best described by the old adage that when one only has a hammer every problem tends to look like a nail. Likewise, overly focusing on ‘what’ runs an equally dangerous risk best thought of as reaching into the toolbox indiscriminately for any heavy object, say a wrench, when one needs to pound a nail. Neither of these two approaches is a good idea.