complicated software and you, defense edition

Military planners in the U.S. are fretting about the success of the F-35 jets and for good reason since they’re slated to be the backbone of all military aviation over the next half century, so over the next 50 years, more than a trillion dollars will be spend building, testing, and maintaining a vast fleet of them. Most of the technology they’re using is brand new and has never quite been put together that way before so there are definitely bugs a plenty in the initial stages, but since a lot of technology tends to be seen as black magic by politicians and reporters, there are angry rumbles about why the jets aren’t performing up to par and freakouts about the 10 million lines of C++ code between the three software blocks (which we’d call versions in the civilian world). It’s an impressive number and it’s great that software running on jets is getting the attention is really deserves for once, but what exactly does that number mean and is it really that important?

First and foremost, just to set the proper mood, allow me to share one of my professors’ favorite lines about programming. Software is a lot like sausage. Just like good sausage is delicious but you wouldn’t want to see exactly how it’s made firsthand, software can be impressive and useful, but you may not want to look into its source code. I have never seen new technology come off the line or be deployed completely successfully and error free first time out in the field, and have never met or heard of someone who has. Were you to look into any sizeable company’s audit of where its tech projects are going, you’d see countless lamentations of how nothing works right, everything is slow and broken, and that the systems just won’t do what they want them to do. It’s the natural state of new development. Often times people aren’t even sure what they want to do but the minute you show them a proposal they know it’s not what you just demonstrated and you will end up going through iteration after iteration to finally nail down the requirements.

In lower visibility projects this iterative chaos with a method behind it tends to fly well under the radar, but the F-35 is anything but low visibility. Were we to hear nothing but stellar reports of its perfect performance first time out, I’d be concerned since to me it would say that it was not being properly tested or something was being covered up to make the jet look good. So when I hear hyperventilation about the Block III software and its 8.5 million lines of code, it’s almost like music to my ears since it tells me that this crucial part of the process is really being taken seriously. At the same time, we could do without the focus on the number of lines of code because it’s good for conveying scale and little else. Much of this code is probably going to be very routine, meant to get and set data. A relatively small portion of it is going to handle that data in a complex ways and a good chunk is going to be responsible for optimizing performance critical pieces and can always be tweaked when we know the key functions and methods work. It’s those bits we need to worry about rather than whisper “10 million lines” on the way to the fainting couch.

Now, I don’t have access to the F-35 source, otherwise I would obviously not be writing about it, but these hunches are based on my experience with large applications. Most of a huge piece of code has to be CRUD because it needs to pull in so much data from all the relevant parts of the system to make complex choices. The same thing is happening with the F-35. It has to pull in all sorts of data from its sensors, balance it against the pilot’s input, and run its options against the complex logic that tells it what the plane can and cannot do, which means that I could defend my educated guess here in the comments. So with that in mind, freaking out about the size of Block III makes it rather easy for journalists to sound like they researched the software issues, but in reality, it’s a meaningless stat that obscures the real concern. Lockheed could lighten its code blocks with machine learning, but the military doesn’t yet trust robots to make decisions with little or no human input, so the code has to be deterministic rather than probabilistic. We could talk about what that means too, but we’d just be getting into the gritty details of software design…