complicated software and you, defense edition

September 21, 2012

future stealth bomber

Military planners in the U.S. are fretting about the success of the F-35 jets and for good reason since they’re slated to be the backbone of all military aviation over the next half century, so over the next 50 years, more than a trillion dollars will be spend building, testing, and maintaining a vast fleet of them. Most of the technology they’re using is brand new and has never quite been put together that way before so there are definitely bugs a plenty in the initial stages, but since a lot of technology tends to be seen as black magic by politicians and reporters, there are angry rumbles about why the jets aren’t performing up to par and freakouts about the 10 million lines of C++ code between the three software blocks (which we’d call versions in the civilian world). It’s an impressive number and it’s great that software running on jets is getting the attention is really deserves for once, but what exactly does that number mean and is it really that important?

First and foremost, just to set the proper mood, allow me to share one of my professors’ favorite lines about programming. Software is a lot like sausage. Just like good sausage is delicious but you wouldn’t want to see exactly how it’s made firsthand, software can be impressive and useful, but you may not want to look into its source code. I have never seen new technology come off the line or be deployed completely successfully and error free first time out in the field, and have never met or heard of someone who has. Were you to look into any sizeable company’s audit of where its tech projects are going, you’d see countless lamentations of how nothing works right, everything is slow and broken, and that the systems just won’t do what they want them to do. It’s the natural state of new development. Often times people aren’t even sure what they want to do but the minute you show them a proposal they know it’s not what you just demonstrated and you will end up going through iteration after iteration to finally nail down the requirements.

In lower visibility projects this iterative chaos with a method behind it tends to fly well under the radar, but the F-35 is anything but low visibility. Were we to hear nothing but stellar reports of its perfect performance first time out, I’d be concerned since to me it would say that it was not being properly tested or something was being covered up to make the jet look good. So when I hear hyperventilation about the Block III software and its 8.5 million lines of code, it’s almost like music to my ears since it tells me that this crucial part of the process is really being taken seriously. At the same time, we could do without the focus on the number of lines of code because it’s good for conveying scale and little else. Much of this code is probably going to be very routine, meant to get and set data. A relatively small portion of it is going to handle that data in a complex ways and a good chunk is going to be responsible for optimizing performance critical pieces and can always be tweaked when we know the key functions and methods work. It’s those bits we need to worry about rather than whisper "10 million lines" on the way to the fainting couch.

Now, I don’t have access to the F-35 source, otherwise I would obviously not be writing about it, but these hunches are based on my experience with large applications. Most of a huge piece of code has to be CRUD because it needs to pull in so much data from all the relevant parts of the system to make complex choices. The same thing is happening with the F-35. It has to pull in all sorts of data from its sensors, balance it against the pilot’s input, and run its options against the complex logic that tells it what the plane can and cannot do, which means that I could defend my educated guess here in the comments. So with that in mind, freaking out about the size of Block III makes it rather easy for journalists to sound like they researched the software issues, but in reality, it’s a meaningless stat that obscures the real concern. Lockheed could lighten its code blocks with machine learning, but the military doesn’t yet trust robots to make decisions with little or no human input, so the code has to be deterministic rather than probabilistic. We could talk about what that means too, but we’d just be getting into the gritty details of software design…

Share on FacebookTweet about this on TwitterShare on RedditShare on LinkedInShare on Google+Share on StumbleUpon
  • Bruce Coulson

    Every new technology ever developed, throughout history, has had spectacular failures in the early years of use. That’s sort of the nature of the universe. People who are learning a new skill make a lot of mistakes while they’re learning.

  • Kelly

    Thanks for a good article.

    I have never worked on F-35, but I have worked on several similar products over the past 20+ years.

    8.5 MSLOC is a big project, especially when you consider the level of scrutiny this code receives (design reviews, code reviews, unit testing with full path/decision coverage, and unit test reviews). Every line of that code has probably had a dozen pairs of eyes on it.

    That said, my experience has always been that the real fun begins once you actually start to integrate the various parts. No matter how many coordination meetings you hold, or interface specifications you write, there will always be cases where the components don’t play nice. These can be timing issues, unit mismatches, format problems, etc. The 8.5 MSLOC in the F-35 is spread out over a lot of separate components (ie flight controls, navigation, weapons, comms)

    On top of that, HW/SW integration is a challenge because you frequently don’t have suitable HW available at startup and end up doing a lot of SW simulation, which is not always hi-fidelity. You need to build a lot of custom TE (test equipment, both HW & SW) to provide stimulus to exercise all of this code and capture/analysize the results. The development of the HW, SW, and TE generally happen in parallel due to schedule pressures and the integration of these items is frequently as challenging as integration of the SW components themselves.

    On top of all that, a lot of the engineering management in these companies comes from the HW side of the equation, and don’t always understand SW. SW estimation/scheduling is hard, even for experienced people.

    So, to summarize, I am a bit concerned about the delays on this job, but neither really worried nor surprised. A job like this is not exactly the same as writing Excel.

  • Brett

    Congress and the military didn’t exactly make things easier by trying to make it a “do everything plane” capable of all kinds of air missions. They should have just made variants on the F-35 for different roles (and kept the F-22s for the air superiority role).

  • Paul451

    Completely off-topic aside:

    “Just like good sausage is delicious but you wouldn’t want to see exactly how it’s made firsthand”

    I’ve always found this saying odd. If there’s anything in the sausage making process that would put you off, they are probably not “good sausages”, or you are way too squeamish.

  • N123

    Murphy’s law (AFAIK unpublished):

    If it works from start, there is something very wrong with it.