Good, Fast, Cheap

 By Shamus Jun 15, 2007 44 comments

I need to make a new category for my blog, “What he said”. If I did, I could use it to file this fantastic post from Jay Barnson on why Why Software Design Isn’t Like Architecture.

Except few other “controllable, predictable” disciplines are as subject to inadequate specifications and changing requirements as software engineering. I mean, can you imagine a civil engineer trying to build a bridge for an unknown location and environmental conditions, only being told that it should be made of steel, cross a body of water that is “less than 2000 feet” in width, and be capable of expanding to any kind of load they want to put on it (and in any distribution)?

Or, as the man says, read the whole thing.

20204Feeling chatty? There are 44 comments.


  1. Teague says:

    I’m a surveyor working for a large civil engineering firm. We actually get clients who do approach us with the sort of criteria mentioned in the blurb Shamus posted here, and they expect us to give them a proposal based on that kind of info. It makes for our version of the stories IT people get to tell about their clients… :)

  2. Loki says:

    As someone who relies on web design for a sizable chunk of his income I must applaud this one! Excellent stuff.

    Speaking of web design, I am really impressed with your implementation of WordPress. I’d love to pick your brain for some tips as I am hoping to redeploy Planejammer to WordPress and consolidate the various PC Diaries and othersuch for my game.

    I’m especially fond of the dice that count how many comments a post has….

  3. Roxysteve says:

    Well, cry me a river.

    I can understand the chagrin, having worked in what used to be called programming but now has a fancier title (and don’t ask any chartered engineer what they think of “software engineers”) for over thirty years and seen more than my fair share of scope creep, but 90% of this problem lies firmly in the hands of the people producing the software.

    Failure to properly analyse the business process adequately (few specialised systems analysts exist today: the software engineers “do” that function themselves), failure to properly identify user requirements and failure to insist on a proper change lifecycle for increasing scope of products lie squarely in the crosshairs of this rant.

    For what it’s worth, we often ask real materials engineers to do exactly what Jay Barnson says we don’t. Your beloved sopranos wouldn’t have been deliverable if we hadn’t, for example. And yes, the results of futzing about with a once-good design to incorporate last-minute and ill-advise “enhancements” not approved by the original architect has caused exactly the kind of catastrophic failure that we see in lots and lots of software. See:Taccoma Narrows Bridge.

    Perhaps if more people understood the business process they are trying to develop software for, they would anticipate much of the trouble beforehand and build to withstand it.

    Next up: C programmers explain why they didn’t simply look at Cobol and figure out why it has a special currency type designed over 50 years ago before they ran off and replaced the bank’s software with “proper language” programs doing real number arithmetic that has cost millions in binary point droppoff. No doubt this will be the client’s fault too, when enough buzz gets generated about it. Can’t blame the “software engineer” after all. He/she can’t be expected to know anything about the real world business process, can they?

    Sheesh.

    Steve.

  4. Shamus says:

    Someone hit a never there, Roxy? Where did THAT come from?

  5. Sem says:

    Great coincidence. I am posting this from my work and together with a coworker we are doing overtime for a project. Surprisingly, the initial version of it was on time but the client drenched us in a torrent of change requests -_- and you know the drill : it must be done or heads will roll. I don’t think I will get away before 9 – 10 o’clock in the evening :( instead of my normal 4 o’clock. I really love my job but some clients drive me crazy.

  6. Wes says:

    Roxy is correct of course. However, the comment makes it sound like “just do real engineering” is the solution.

    It would be the solution if the problem was rooted in the engineering side. It isn’t rooted there though, it is rooted in the fact that software is cost sensitive and plastic. When this is removed from the environment, software can be *highly* reliable. To wit, look at the software that operates the aerospace industry. There is an excellent article about the team that does the space shuttle’s software:

    http://www.fastcompany.com/online/06/writestuff.html

    The takeaway point is that software can be part of the engineering process when the specifications are tightly created and the target application is fixed. All aerospace software (well, all critical systems… all you have to do is look at the crashed “in flight entertainment systems” to see that they cut corners where they can) have to meet stringent requirements and target a well understood solution space.

    So, if you are willing to pay aerospace software product prices, you could have bulletproof software on your desktop in about five years. This is where the cost sensitive part comes in; most consumers won’t pay $55,000 for Word: Bulletproof Edition.

    Beyond this is the expectation that our day to day software perform an infinite number of novel tasks instead of a small number of well defined tasks. This is the plastic aspect: we expect Word to write letters to grandma, and template invoice generation to customers. Aerospace software isn’t expected to be a template language and scripting tool: it only is responsible for keeping the vehicle in the air.

    Further, business software is only useful if it models the business processes (or you change your processes to those of the software: that’s what a SAP guy will ask you to do). Reality in the business word is that change is the only constant (to use a tired phrase and mean it). If it takes you five years to implement bulletproof software that models five year old, replaced, processes, you have failed. If it takes you a year to implement this years processes and then you can keep in in real time, you have provided value, even if the software is not bulletproof. This can’t be taken *too* far; if the software causes enough trouble it erases whatever business value through support costs and work interruptions.

    Most business software lives in this space of being:
    a) implemented fast enough to be worth doing
    b) stable enough to not erase the value brought
    c) flexible enough to change rapidly as the business does

    This looks nothing like the requirements I had in aerospace, where were:
    a) meets or exceeds every requirement… really
    b) redundancy to handle failure modes of both software *and* hardware
    c) run on processors that are durable enough to take massive abuse (read: slow 386-486 generation hardware)

    In conclusion, there are software “engineers”. They mostly work in aerospace or other lives-on-the-line industries. Everyone else nickle and dimes prices down for software with ill-defined specifications to the point you get what you get.

  7. Wes says:

    Grrr, just ignore the grammar errors (like “in in” instead of “up in”) and the stray commas. Text boxes are not optimized for composing long thoughts.

    One last point though: some might wonder why having bulletproof software wouldn’t be a competitive advantage. It would be if it could be created at a “reasonable cost” (where “reasonable cost” is defined as “the cost of making the garbage we live with today”). If you spend more than your competitor does on software, you had better be sure you are getting that much more value from the bulletproof nature of the software. As a battle scarred veteran of trying to bring high quality to businesses, I can safely say “good luck with that”. The finance guys can easily to the math that says 95% uptime and 99.999% uptime don’t bring enough value to the table to pay for it… and prove it with a spreadsheet, even counting lost productivity.

    My current project has a 99.9% uptime promise and we keep it. We have so few support calls that people freak out at our small support staff. It can be done. However, my current project is also a financial failure. Yay reliability.

  8. Roxy sounds like every person who is ignorant of actual art of programming.

    I been a programmer for over 20 years mostly creating, maintaining, and update a CAD/CAM program for metal cutting machines in northwest PA. (http://www.plasma-automation.com) I am still deluged with change requests. So are easy to accommodate and some are not.

    The art of programming has not stood still. There are proven algorithims for many defined problems. Object oriented programming has developed its own form of algorithms called Design Patterns. There is a technique called refactoring that been developing that allows you do make reliable design changes in your software.

    However there hasn’t been a single magic bullet. This is because programming (and computers) are so general purpose that it is hard for even the most competent programmer to properly define all the specifications.

    For example I worked on a new classes of metal cutting machines called a coil-line. It’s motion control requirements and feature list is a LOT narrower then our regular line of metal cutter. (It does one type of metal cutting really FAST hence its appeal) Despite years of experiences the machine designers had with competitors machine, I am still getting “O by the way, we need this.” Not to say all problems are caused by a lack of specifications. There are the occasionally outright bugs and misinterpretations.

    For my particular application most of the problems had do with diagnostic. The actual core functions of the software worked pretty well since Day 1. However when something went wrong with the machine, the initial diagnostics were woefully inadequate. This is hard to capture in specifications. The only way to know what to do is to actually have a machine running the initial version of the software for a while.

    However there is one episode of specifications that came up. We have two ways of powering this machine, one uses a hydraulic motor and the other uses a single axis servo motor. The hydraulic is cheaper but controlling it is really sloppy. All you can do is turn it off and on and watch a counter to mark where you are. A servo has precision control and you can really get accurate.

    Now this machine is good enough if you are getting 1/32″ of an inch. The first couple were built with hydraulic and people were very happy with the repeatability and accuracy compared to the competition. A few were sold with hydraulics and finally we had the servo version ready. The servo version really knocked their socks off.

    So they sold a few of those as well. But then six months later we started to get hydraulic orders again because in some sales they just needed the cheaper price. I started get complaints how crappy the hydraulics were and the software was broken.

    After a long struggling process basically it wound up that much of the issue was due to the fact they were used to the accuracy of the servo and couldn’t understand why the hydraulics didn’t do the same. But we did uncover some better control algorithms for controlling hydraulic motors so they are better then the first version we shipped.

  9. Shamus says:

    I’ve worked for enough clients to see the truth Jay mentioned: Many people have no concept of what goes into software development. Most people can grasp the concept that if you make the building higher or the bridge longer it will cost more, but those same people have no understanding of the tradeoffs involved in requesting new features. It’s not just development and testing time, it’s memory usage, interface clutter, security, maint. costs, and performance.

    Sometimes you have to *educate* customers before they can even begin to make these kinds of decisions, and customers are not all that keen on being educated by their clients. When they tell you to add feature X, where X will create annoying complexity and cause headaches down the road, you can do it or let them take their business elsewhere.

  10. Shamus says:

    And I ended my comment before I’d made my point:

    A lot of this is that the clients I’ve been around have only been around computers for the last decade or so. Right now we have a generation of kids who have never known a world without the internet. They will grow up with a firmer understanding of what software is and where it comes from. (Many will even write a few lines of code during their schooling.) This will probably make the world of software design less crazy.

  11. Shamus says:

    And while I’m spamming my own comment thread, I’ll add this: I agree with the point Wes made about swiss army knife software: Things like Word, which do a hundred things poorly, slowly, and confusingly, instead of doing one or two things well. I’ve always thought that if Moore’s law halted for a few years a lot of software would tighten up. You couldn’t just cram more features in and expect computers to catch up by the time you released it. You’d have to actually evaluate some tradeoffs.

  12. wumpus says:

    Yow, I’m gonna have to wade into this…

    First off:

    The modeler just didn’t get why it was such a big deal. He said, “it is funny how I cant understand how hard it is to get it up the y axis.”

    This is someone who should be fired. S/he was given specific instructions on how to create models and, apparently, decided to ignore them and consequently spent a bunch of time (==money) creating beautiful but unuseable sets. And sh/e considers this to be ‘funny’.

    Next up: On software engineering and its relationship to ‘real’ engineering. When I was getting my MSEE, many of the MEng students took the (state, I think?) exam to become ‘licensed’ engineers. This exam was considered a joke, as it tested ‘general’ engineering knowledge, and thus was so basic and generic that pretty much anyone could pass it. But you had to have an ‘engineering’ degree to take it. (Not a ‘science’ one like I got.) The MEng’s were talking about getting hammered before they went to take the test.

    Also, I’ve met a few ‘programmers’ – they know programming languages, maybe some CS. A lot of people who work with the web fall into this designation. I, however, am an engineer. I have a background that includes high level math, general science, and specific, directed engineering – things like signal processing, control theory, circuit design, etc. Yes, I use programming languages to do my work, but you couldn’t put a ‘programmer’ in my chair and hope to have him to do my job (unless you gave him a bunch of very, very specific ‘requirements’, or rather unless I gave them to him).

    Finally:

    Next up: C programmers explain why they didn’t simply look at Cobol and figure out why it has a special currency type designed over 50 years ago before they ran off and replaced the bank’s software with “proper language” programs doing real number arithmetic that has cost millions in binary point droppoff.

    Hmmm… Perhaps the ‘engineers’ who wrote the Cobol can step up and explain why their precious, heavily engineered type is documented only by source code? When I need to use a protocol or interface and the reply to “Where is the specification?” is “Just source dive the personalized object-oriented assembly for this obscure processor,” I know I’m dealing with ‘engineers’ who have been allowed to run amuck. Documentation (and design) are parts of the job (of engineering), just like implementation (coding) and debugging (and maintenance).

    Alex

  13. Roxysteve says:

    wumpus Says:

    Hmmm… Perhaps the ‘engineers’ who wrote the Cobol can step up and explain why their precious, heavily engineered type is documented only by source code?

    Missed the point by a mile there, wumpus, just like the twerps who wrote the (very real world) code in C because it was “better”: The point being the rush to replace precluded the proper consideration that in any decimal-intensive computational environment such as oooh, I dunno, let’s pick a really obscure and rare one, MONEY you are a damned fool miseranble hacker if you can’t figure out that a floating BINARY point can never do the job properly.

    Or to put it another way: The “software engineers” forgot to ask the right questions about the fundamental basics of the business they were “serving”. Like why the old language had a special data type for it. I thought C programmers were hyped by data typing. I stupidly thought that would include understanding how the data types worked. Silly me.

    But by all means get defensive and snippy and speak in sweeping generalisations. That’s what got the banks in the mess in the first place. “Your Cobol is bad. Let us replace it. Fire your DP department (including all your systems analysts). Hire cheaper one-size-fits-none consultants”.

    As for me, I’m used to hearing young “software engineers” and “sytems administrators” disrespect me because I cut my teeth on big iron. Only in the computer business could three decades of making computers work be seen as a disadvantage.

    Got to go. The thirty year old in charge of the E-mail REGEX cascade filter has (again) trapped a colleague’s mail and flagged it (incorrectly) as being oscene or racist. Of couse, I shall have to do his job for him and deconstruct the error the hard way to prove he has (again) deployed technology beyond his power to test properly. He won’t take my word for it (again) because I’m so old I can’t possibly understand regular expressions and believe in that out-moded idea of regression testing after mods are made. What a f***ing dinosaur.

    [Yes, shamus, you could say a nerve was touched. There's an old saying that applies in spades to the computer biz: A bad workman always blames his tools.]

    As for “can’t possibly test all contingencies”, Try reading Meyer’s Object Orientd Software Construction. Employ the principles detailed there in your designs and even if your software is taken outside its design envelope it will degrade cleanly.

    But what do I know. I read books with paper pages for crysakes.

    Steve.

  14. Roxysteve says:

    Thank you Mr Brain.

    That’s Object OrientEd Software Construction of course.

    Steve.

  15. Miako says:

    >I’ve worked for enough clients to see the truth Jay >mentioned: Many people have no concept of what goes into >software development. Most people can grasp the concept >that if you make the building higher or the bridge longer >it will cost more, but those same people have no >understanding of the tradeoffs involved in requesting new >features. It’s not just development and testing time, it’s >memory usage, interface clutter, security, maint. costs, >and performance.

    The best counterpoint to this is DarkRadiant. It’s a program designed for creating 3d video games. It contains buckets of improvements — and they don’t cause interface clutter. I think this is because the artists and techies lived together, and everyone knew that a few hours of techie work was worth it, to give the artists something quick and useful.

    Then again, Anyone who is using a computer and -not- using hotkeys isn’t using it at all efficently (except for drawing, which was what the mouse was originally designed for.)

  16. Shamus says:

    Roxy: I aspire to the day where I, as a programmer, have enough clout to screw up the project myself. Right now I have to screw it up by following the specs I’m given.

  17. Miako says:

    >Next up: C programmers explain why they didn’t simply look at >Cobol and figure out why it has a special currency type >designed over 50 years ago before they ran off and replaced >the bank’s software with “proper language” programs doing >real number arithmetic that has cost millions in binary point >droppoff.

    … how long was the COBOL type in bytes?

    …and you know that the banks just keep the roundoff error, so it was in the companies interest.

    Quito!
    … which was a far worse banking screwup.
    “and yes, kids, this is why we separate data from code”

  18. Miako says:

    So you’ve got programmers, and you’ve got software engineers… as a physicist, I can sympathize with not wanting to have code monkeys on the job.

    But there’s another category — the mad scientist programmer. They tend to live in places that require high optimization (read assembly languages), and tend to hate documenting their work. If you’ve ever seen assembly spaghetti, or self-modifying code, you know what i’m talking about. They are the sort of people who threaten to put in random error messages when you start to talk about code documentation.

  19. wumpus says:

    Howdy Steve,

    You said:
    But by all means get defensive and snippy and speak in sweeping generalisations.

    I say:
    Ya know, when you imply that software engineers aren’t real engineers and that C programmers are somehow all dysfunctional (as a result of using C?) and then proceed to talk about me as being ‘defensive and snippy and speaking in sweeping generalizations’, I’ve got to laugh. Or cry.

    I have no idea why the engineers or programmers in question made the decisions they did. Presumably there was some sort of obsolescence issue with the hardware and/or tools? My point was that it’s hard to gain any benefit from the wisdom of the ages if it’s all written in a dead language.

    Now, if they started from scratch (because it wasn’t deemed worth the time and effort to try to recover the wisdom of the ages) and fscked up the design because they didn’t understand the problem, well, they’re either badly managed or bad engineers.

    But bad engineers are, unfortunately, engineers too. And they exist in every discipline of engineering. Similarly, bad programmers are programmers, and they abuse every language out there. I don’t think good engineering practice is something that any generation, job title, or programming language has a patent on.

    (And no, I’m not even going to begin with bad management.)

    Alex

  20. ShadoStahker says:

    Shamus said:

    Roxy: I aspire to the day where I, as a programmer, have enough clout to screw up the project myself. Right now I have to screw it up by following the specs I’m given.

    I am a civil engineering student. All types of Engineers get the same sort of stupid requests for features and additions that can’t be done. But I think you (possibly inadvertently) just hit on the reason why it’s so much worse in the software industry.

    An engineer on a project isn’t just an employee, he’s the final signature before the project is approved, constructed, modified, etc.

    As such, we actually do have the ability to screw up the project ourselves. But as long as we’re aware of that, we also have the ability to avoid approving a screwed-up project. And when public safety is often at risk with an engineering project, we have both a moral and legal (yes, we can be held fully responsible if a bridge falls down, if there’s any possibility of faulty design) obligation to not approve those projects.

    A programmer doesn’t have that. Firstly, they often don’t have the leverage to reject things. Secondly, they usually don’t have the same moral and legal responsibility that comes with a project that can actually endanger lives. And thirdly, there will probably be another programmer who will implement the request without argument (while most engineers will refuse to approve a faulty design, due to the moral/legal obligations above), so the programmer can be disposable.

    So we do get the same requests.

    We just have more leverage to handle them.

    By the way, my fiancee’s brother is a Computer Engineer. Not a Software Engineer (or programmer), but a full-fledged engineering-degree iron-ring Professional Engineer, just like a Civil or Mechanical. He spends his days programming, but mostly for sensitive projects. Currently, he is working on sensor systems for bridges. Because of his degree, and the sensitive nature of the software he is working on, he is also able to “screw it up himself”.

    We engineers do dislike the name Software Engineer, but mostly for the reason that it causes confusion. What’s the difference between a Software Engineer and a Computer Engineer? In terms of the legal responsibility taken for projects alone, there’s a world of difference. But ask the general public, or the HR people doing the hiring…

    We have nothing against the work of programmers. Heck, without the general programmers, nothing would get done. Just like Engineering Technicians in Mech/Civil, we need you guys to do our jobs.

    But sometimes you need us, as well, to ensure that the job is doable. We have the ability to say no to the customer, and justify it with our personal legal responsibility. And, being the ones who create and finalise the design, we have the ability to change it to make it easier on you.

    It’s like I was taught in first-year engineering design. Listen to what the customer wants you to do (example was creating a machine with a robotic arm that would navigate an orchard, find apples, and pick them), decide what the problem actually is (easier way to pick apples), and then give them the best solution to that problem, even if it’s not their original idea. 9 times out of 10, it will work better than the customer’s original plan, and they will be just as happy (or happier), especially if it’s cheaper.

    It would be a lot easier if programmers had the ability to do this, as well, on projects what weren’t life-threatening (like bridges or airplanes).

    …Wow, that was 3 or 4 different topics.

    I apologise if anything was confusing. I’m slightly tired. I can clarify anything later, if something I said wasn’t clear or came across wrong.

    If you’re offended or angered by anything I said, please ask me to clarify first before condemning it. I may have worded it stupidly.

  21. Roxysteve says:

    Wumpus said:
    Ya know, when you imply that software engineers aren’t real engineers and that C ….

    I actually reported (rather than inplied) that other engineers, the ones that actually do, ya know, engineering, serve apprenticeships and vet each other by means of various engineering institutes you have to impress other engineers to get into do not appreciate computer programmers claiming the title, and this thread goes a long way to explaining why. How I could become defensive in the way you suggest about something I actually do myself for a living is a mystery. However, if you belong to an engineering institute, say so and I will appologise fully for any percieved insult.

    The “bank screw-up” programmers did what they did because they did not understand the ramifications of how a computer actually does fractional arithmetic and they (and anyone else who doesn’t yet get it) will continue to make the same basic mistake and cost you and me money as a result of their poor engineering. The problem is industry wide, by the way, not specific to a single bank.

    The irony is that a little less knee-jerking and a little more thought would have led any competent engineer (as opposed to programmer calling themselves that) to question the need for a specific data type in the legacy language that was neither the existing integer type nor the existing real type. Yes Susy, Cobol does have support for real numbers and has for over five decades, but the people who spec’d it out knew even back then then that you shouldn’t try and make up a paypacket of budget a new destroyer using them (Cobol was originally spec’d by Admiral Grace Hopper of the US Navy).

    It doesn’t matter whether these poor fools (not so poor, I’d bet) started from scratch. They were so busy telling everyone how good their tool was they didn’t actually take any time to understand how it worked. Worse, they didn’t understand the thing they were replacing (how could it possibly matter? It was Cobol).

    I don’t know if I can blame them really. Any documentation I’ve seen where real numbers are concerned in the last 15 years out of the likes of Borland, Microsoft, Oracle and the various sources of open software has contented itself with a vague warning about “aproximations” without explaining the specific technical danger. It’s not as though most of these software engineers have a batchelor of science degree in computer science and would therefore be expected to figure it out. Oh, hang on…..

    There is nothing wrong with the C language. It does not have an intrinsic data type that would clue in the clueless to the potential for disaster, but there is nothing stopping the bright young things from writing a class, function or even a structure to do the job. Of course, you’d expect such a beast to be well known and in the public domain by now….dang!

    The fault I was ascribing was entirely down to the people using the language to craft bad software, and doing so so cluelessly that the problem they created is now systemic. In a game, such a problem would be inconvenient and might cause a small company to go bankrupt. In the business world the disaster can unfold to enormous dimensions before the damage causes enough ripples for the effects to be reflected in the higher-level phenomena such as the stock price or an adverse bottom line in the general ledger. How much of this affects you? Well, how much interest are you paying on your house? Your car loan? Your student loan?

    Steve’s three rules of Improving Things You Had No Part In Making:

    1) There’s got to be a better way.
    2) There must have been a good reason for doing it the way it was done in the first place.
    3) Leave it the **** alone until you fully understand what rule 2 implies.

    Rule three was at the heart of the “bank” problem.

    Steve.

  22. Roxysteve says:

    Ah, I just put on Blue Oyster Cult’s “Don’t Fear The Reaper”.

    All together now: “I gotta have more cowbell!”.

    Have a mellow Father’s Day everyone, whether or not you understand the finer points of binary fractional arithmetic and the one’s complement rigmarole.

    Steve.

  23. wumpus says:

    Howdy Steve,

    > I actually reported (rather than inplied) that other engineers, the ones that actually do, ya know, engineering, serve apprenticeships and vet each other by means of various engineering institutes you have to impress other engineers to get into do not appreciate computer programmers claiming the title, and this thread goes a long way to explaining why.

    I guess that last bit is a dig at me? As in, you as a ‘real’ engineer, are totally unimpressed by me, a ‘fake’ engineer? Otherwise I don’t understand it.

    It seems to me that someone who has a degree in engineering and is paid to do engineering work is, by definition, an engineer, irrespective of his membership in, or even the existence of, a self-legitimizing society or institute in that particular field of engineering. (We can, again, quibble over whether or not that person is a good engineer.)

    > How I could become defensive in the way you suggest about something I actually do myself for a living is a mystery.

    It is impossible to be defensive about things that you have no investment in. Your very first post in this thread appeared highly defensive to me, and, as others pointed out, it looks like the article hit a nerve in you.

    (As to my investment, I am a software engineer with 10 years of experience in the field, almost all of it using C. I don’t like being blamed for all the ills of the software world by proxy.)

    > However, if you belong to an engineering institute, say so and I will appologise fully for any percieved insult.

    Would the IEEE count? I’m not a member, but I’ve come close to joining on several occasions. I’ve also been published in an ACM publication, but I’m not a member there either. Not that it matters – it seems clear that you simply don’t consider software engineering to be an engineering discipline, and I doubt I’m going to change your mind.

    Anyway, you asked for a C programmer to explain why he wouldn’t just read Cobol source, and I answered your question: source diving in a foreign language is a very bad idea. The rest of the whole bank programming thing isn’t really terribly germane.

    By the way, in what way was the Tacoma Narrows disaster caused by last-minute futzing? I was under the impression that the design simply failed to take into account resonance (from the beginning). (I saw the footage in my intro physics class.)

    Oh, and I do professional audio DSP coding (for embedded processors), so I’m very, very familiar with fixed and floating point arithmetic, precision issues, 1′s and 2′s complements, etc. (My impression of business software programmers’ understanding of the same is, uh, less good.) But bad requirements and specifications from managers and/or users who have no understanding whatsoever of the impact of their choices (and/or vagueness) and find it ‘funny’ how they just can’t get the technical aspects (I’m picturing a guy in a suit with a bored look on his face who is fiddling with his Blackberry while an engineer is at the whiteboard) is still the biggest cause of delays and dysfunction. Hey, look, back on topic!

    Alex

  24. Akatsukami says:

    Or to put it another way: The “software engineers” forgot to ask the right questions about the fundamental basics of the business they were “serving”. Like why the old language had a special data type for it. I thought C programmers were hyped by data typing. I stupidly thought that would include understanding how the data types worked. Silly me.

    [...]

    As for me, I’m used to hearing young “software engineers” and “sytems administrators” disrespect me because I cut my teeth on big iron. Only in the computer business could three decades of making computers work be seen as a disadvantage.

    Because, as you must know, having cut your teeth on big iron, 360-compatible machines have packed decimal instructions implemented in hardware; you can define and manipulate packed decimal numbers in C/370. Other architectures don’t; packed decimal arithmetic must be emulated in software (i.e., slowly and buggily).

  25. Coyote says:

    The modeler just didn’t get why it was such a big deal. He said, “it is funny how I cant understand how hard it is to get it up the y axis.”

    This is someone who should be fired. S/he was given specific instructions on how to create models and, apparently, decided to ignore them and consequently spent a bunch of time (==money) creating beautiful but unuseable sets. And sh/e considers this to be ‘funny’.

    Well, fired is pretty strong. When doing game development, the artists and programmers are supposed to team up and push each other to make a better game. Design documents are not (and should not be) iron-clad specifications. Sure, there are aspects of software development that are like that – creating software for medical systems or the space shuttle come to mind (and were mentioned earlier). But when doing web development, game development, or about 90% of other kinds of software development, change is part of the game.

    And that’s why “agile processes” have evolved to deal with the problem.

    But regardless, the breakdown is still in communication with the customer / client. Software engineering is a black box, and management / customers (or artists) too often have zero idea of the costs associated with their little changes in an attempt to “improve” the final product.

    Can THAT be improved?

  26. malfunction84 says:

    Steve, your “three rules of Improving Things You Had No Part In Making” are brilliant. It takes “if it ain’t broke” to a whole new level.

    The author of this arcticle is 100% correct. Engineering is just as applicable to software as it is to anything else. The only difference is that the Silicon Curtain is a lot thicker. There’s no “magic” that holds your house together, but many clients view computers as these arcane devices which “computer wizards” can mysteriously command. Some have a better understanding than that, but all of these clever analogies are lost on them.

    I’ve met educators with the distinct goal to “take the magic out of the box.” Of course, this gentleman taught a CS class designed for non-CS engineering majors, so his audience was very appropriate, albeit rather small. Is there any way to speed up the process Shamus mentioned in post #10?

    When it comes to poor programming practices, I believe the Internet is fanning the flame. Open source, free IDEs, online tutorials… all of these things are good for spreading interest in the field, but not for spreading the skill or experience. These days, what we have on our hands is the programming equivalent of a bunch of 10-year-olds making treehouses and calling themselves architects. Then they gripe when they have to spend $250K on a house they think they could’ve built themselves (or had their kid build) for $50.

    My colleagues and I often remark: say what you want about Java itself, but if every language/library had the equivalent of the Java API, the programming world would be a much happier place.

  27. Deacon Blues says:

    Steve, there is only one thing I have to take exception to: the example, waaaay back in your first post, of a structural-engineering job ruined by last-minute spec changes.

    The first Tacoma Narrows Bridge, Galloping Gertie, wasn’t brought low by changes in design specs, but rather by a failure of those who wrote the specs to stay up to date on advances in physics. Otherwise, perhaps they would have been aware of Tesla’s work, some forty years prior, in the field of resonation, and realized that the winds whistling up the Narrows would occasionally hit the resonant frequency for the bridge (a problem overcome in the redesign by building each section with a different resonant frequency, causing resonations to damp out over its length).

    Now, there were some problems with the recently-built second span, but those were more political and legal in nature (the design seems sound, but no one thought to allow for the state’s limitation on what masses can be transported by road, causing an expansion joint to get hung up at the state line for almost a week).

    On the other hand, I was exposed to enough undocumented crap when I was a programmer in the Air Force (changing some programs, dating back to the Sixties, over from FORTRAN, for weeping aloud!). Some programmers – pardon me, software engineers – seem to actually live by the slogan, “Documentation? Why do you think they call it ‘code’?”

  28. Zaghadka says:

    An analogy:

    We’ve just invented “the wheel.”

    The programmer is being asked to code “the chariot,” and halfway through the product development some dork then comes and asks, “Oh by the way it would be really nice if we could have archery off that chariot, because the Egyptians are whipping our butts.”

    Which means that he has to invent the “suspension system” and when the King breaks his leg practicing, that programmer will get criticized for not having invented “the road.”

    It’s all about the specifications folks. No one takes nearly enough time *designing* software, because at this point you have to code it to see how it will work in the first place. Ideally, design would be finished *before* you code, not concomitant.

    After we’ve had a few more “inventions,” things will settle down a bit and we’ll be able to have stable specs.

    Until then: Go with God… and CYA.

  29. What’s Special About Programming.

    (I wrote that; it’s that having written it once, I don’t care to try to fit it into a comment. :) )

  30. Barron says:

    Personally, as a CS grad, I would prefer _not_ to be referred to as a software “engineer”, because all the engineers _I_ knew at school were both arrogant and mediocre programmers. Which is not to say there aren’t plenty of bad programers who are not engineers, but…

    And Shamus has a real point. Maybe you are fortunate enough to work at a company where the spec never changes, where you have a large budget, and lax deadlines. But where I work, the requiremnts are in a constant state of change long after the product has been deployed, proper documentation is far less important than expediency, and the end users are considered good beta-testers.

    I think the problem is that much as people hate buggy code, there is still more software out there to be written than there are people to write it. Once quality code is worth more than a quality programmer, we’ll see a lot more of it. Until then, it will be enough to slap together some half *ss implementation and call it a day. Either that, or someone decides what the code needs to DO before they tell us to write it

  31. Dave says:

    Roxy: I aspire to the day where I, as a programmer, have enough clout to screw up the project myself. Right now I have to screw it up by following the specs I’m given.

    …Shamus

    Well.. I think that’s the core of this. People are people. “I’m just doin’ my job here.” We as animals are pretty lazy. It takes a lot of energy to go up the stream to correct what came down the stream… of course the problem coming down stream was passed along .. which is why the guy at the headwaters has a sign that says “the buck stops here.”

    But.. we really should be questioning what comes flowing downstream… Like perhaps if some guards at a prison in Iraq had questioned off-color behavior.. or if a guard in Germany in the 1940s had said, “do ya’ think we should be doing this?”… but for some reason we just laugh and say, “boy those guys upstream sure are idiots.”

    Like they say.. “IT rolls down hill.” .. and ya know what.. we ALL live at the bottom. Now we ALL have to live with the extra costs that banks will have.. They ain’t eatin’ that.. we do.

    In a society we all have a responsibility to the rest of the group. So if you pass the buck without making an attempt to fix it, well.. we ALL have to live with it.. Thanks.

    If one person stands up.. they say he’s crazy.. If two stand up they say they’re both crazy.. but if three stand up it’s a movement.. people listen to movements…

    Now.. let’s sing it in three part harmony… (I’ll duck all the stuff that sure to be slung at me for this one… I have a +3 mod to my Dex.. and I’m dropping my pack to be unencumbered.)

  32. Lots of aerospace projects get the same ever-expanding cloud of requirements that software ones do, with the resulting lack of anything useful produced for all the time and money used. So I came up with this requirements vs. success curve to beat people with when they come up with new reqs.

  33. Brian says:

    The problem with software isn’t that we don’t know what it’s supposed to do. The problem – particularly today – is that you *have* to assume that the environment is unknown and will change. To use the bridge analogy, you know you have to design a piece of software that gets cars across the chasm.

    But you can’t assume what the land will be like on each end. You can’t assume what materials will be used to build the bridge. You can’t assume anything about the bridge. As a matter of fact, you have to assume that even if you *do* know these factors at the start they will change during the software’s life cycle and therefore cannot be relied upon.

    Users complicate this by saying, “No, trucks never cross the bridge. It’s always cars. We never allow big trucks on the bridge. Well, except on Thursdays when it’s raining and the temperature is 68 degrees. But that almost never happens. Don’t worry about it.”

    The bottom line is that software design today is a series of black boxes. Some are enforced, some have to be assumed. But doing anything against a black box is going to be fraught with problems.

  34. wumpus says:

    Howdy,

    Barron said:

    Personally, as a CS grad, I would prefer _not_ to be referred to as a software “engineer”, because all the engineers _I_ knew at school were both arrogant and mediocre programmers. Which is not to say there aren’t plenty of bad programers who are not engineers, but…

    I say:

    Exactly! This is what I was trying to say before, in my first posting: there are two different classes of people who get called ‘software engineers’ or ‘programmers’. The arrogant engineers who write software as a part of their jobs (me), and the effete (heh – joke!) CS types who write software to satisfy whatever requirements. The former specialize in a particular discipline of engineering and have a limited ability to write code, understand CS concepts, etc. The latter have a very good understanding of coding, but don’t usually have specialized engineering knowledge.

    Two different things which are, unfortunately, often lumped into the same title. (And, of course, there’s a whole spectrum of grey between the two.)

    Alex

  35. Heather says:

    Wow. As a hack web designer and artist married to Shamus (a programmer as we like to call him), daughter to a computer science teacher who was trained in the 70′s and now only teaches the stuff the software companies send him for free, and sister to an electrical engineer (who almost became a computer engineer but the university program was too hard) and who now works at NASA (yes, runon, bear with me) I am fascinated by this particular conversation. If the sentence hadn’t been so long I would also mention that I have dabbled in Open Source and ran Linux for a year, until I got sick of messing with it.

    May I say that anyone who is being paid by a client to do something that the client does not himself understand or know how to do will find that the client is very likely to come up with a list of specification that are not possible or are not optimal for said job. As an artist I run into this but more often than not the client realizes that he is clueless in this and is willing for me to take my own pictures of the subject prior to painting it. On the other hand, when it comes to computers, people seem to think that anything is possible and will ask for the moon expecting to get it. Unlike mechanical processes which obviously have certain natural laws governing them, computer software processes seem to have infinite capabilities. Non-programmer people can not comprehend that just as perpetual motion machines do not exist you cannot have a single software that does everything without losing something in the mix.

    It is an ever changing field and unlike education (which is where I hold my degree) is constantly changing in what it is capable of, but like education–where the names and styles are in constant flux, some changes are good and some are just plain bad–and sometimes the bad ones are the hardest to get rid of.

    In 30 years computers have come a long way, and programmers have been hard put to keep up. A friend of mine was a programmer in the late 70′s. She quit work to have her son and found that, when she was ready to go back, she had been completely left behind. I, on the other hand, will likely be able to return to the education field when my children are grown with just a quick refresher on the new buzz words.

    Steve, I admire programmers who have been around so long and can comprehend all the backend stuff and can “read” old, undocumented code. (I have probably put my foot in it somehow there but remember I am a clueless non-coder.) It must be frustrating to constantly have to fix something caused by someone else’s mistake–which is kind of what Shamus was originally talking about–creating a good thing only to have to break it to do things that only make sense to a non-programmer or have it handed off to a different programmer and then have it handed back so he can “fix it” –all of which he will spend the next 5 years dealing with the extra broken bits. And sometimes–as I have found in my own experience, it is really hard to “educate” the person who pays you.

  36. Dave says:

    Yup… extremely hard to educate the person who pays you.. in any business… I’m a stay-at-home-Dad… and I have a hard time “educating” my wife and kids all the time…

    But.. I think Brian kinda said a few important words..

    Users complicate this ..

    Well.. yes.. users complicate things.. but .. well.. that’s what this stuff is for.. otherwise the box is just crunching numbers for crunching’s sake.. the old GIGO.. garbage in garbage out… while glorious code that makes angels sing when it runs (excuse my archaic terms.. I quit officially programming in the 80s).. if it doesn’t help the user.. well.. crappy programming that does help is more useful.

    I once wrote a glorious program for some FORTRAN class.. the proffessor showed everyone how great it was.. a gem.. unfortunately the result was that I figured out how much paint it would take to fill up a house without getting any on the windows… which may have been useful to someone.. but I guess they wanted to know how much paint to buy for the walls.. which was much less paint.

    Pesky users… a waste of perfectly good programming.

    And speaking of fixing things over and over.. and educating the person whose paying for all that fixing.. well.. thank goodness for that.. imagine the world if we did everything right and only had to do it once.. we’d be taking vacations, smiling at co-workers.. then pretty soon we’d realize we don’t need these fancy boxes.. then where would we be?!!

  37. Brian says:

    I respect the point about users, but I wasn’t necessarily knocking them for not appreciating my code. The point was rather that they tend to give – even to direct questions – bad requirements.

    I’ll admit it – I do accounting systems. There. I said it. Just a couple of months ago I was doing a modification for a client. Their requirements – documented and all, hardcopy no less – stated that they always processed documents in a certain manner. I sent the code to the client for testing and the very first test scenario resulted in “It doesn’t work.” The situation their requirements had said they never did was the very first test scenario they tried.

    It’s just another reason why we can assume so little about the bridge. I generally throw out words like “never” and “always” when I’m reading requirements.

  38. Yahzi says:

    I think ShadoStahker hit it on the head.

    Programmers don’t have the power to stop a project from imploding. They can’t refuse design requests.

    An civil engineer can say, “Look, metal only holds so much weight, so if you add apartment buildings on your bridge it will fall down.”

    But everybody knows that software can do _anything_, so if a programmer says, “If you add that X, it will break,” management just says, “We need a new programmer.”

    After 20 years in scientific programming, I’m now joining the aerospace industry. It promises to be… interesting.

    :D

  39. The Panackes says:

    Pendantic *and* Nitpicky!

    Awesome topic, Shamus. I think about the only thing guaranteed to generate more contentious thread traffic than, “Just what the heck is wrong with sofware these days?” would involve politics or religion. Maybe both of them in a single thread like, “Jerry Falwell’s son announces candidacy for Senate; vows to legislate mandatory church attendance”

    Love the comic!

  40. Dave says:

    But everybody knows that software can do _anything_, so if a programmer says, “If you add that X, it will break,” management just says, “We need a new programmer.”

    Which has proven over and over to work.. look at Shamus’ other posts… the photosyth demo and the sticks and stones post.. there is always someone that can take the same technology, look at it a different way and make the magic box spit out what the user wants.

    remember the magic you could make the Apple II do .. it shouldn’t have done lots of that stuff… but for outta the box thinkin’

  41. Deoxy says:

    “Which has proven over and over to work.”

    No, which has been proven to mostly, most of the time, fail spectacularly, as we have been discussing.

    OCCASIONALLY, it works, as you said. Most of the time, you get the usual crap.

    Result: when a programmer is given bad specs, and says they’re bad specs, and is told to do them anyway… he does them anyway, and we get crap.

  42. ShadoStahker says:

    Whereas when an engineer is given bad specs, we often have the laws of physics to back them up. :-P

  43. Dave says:

    OCCASIONALLY, it works, as you said. Most of the time, you get the usual crap.

    Fair enough… but each time is taken singularly.. and each person with a project thinks their project is the one that is special.. just like each time the programmer thinks their way is the only way.

    As Miss Frizzle of the Magic School Bus says.. ya gotta take chances and make mistakes..

    of course .. all that works to be “visionary”.. to just get something that works for a useful job they should listen to the programmer.. but that leads us back to the idea of their project being so ground-breaking… and the loop starts over again….. fatally.

  44. Julia says:

    Coming late to the party –

    The user who orders the software assuming that the programmer has knowledge that not everyone has can lead to problems.

    In Austin a number of years ago, a system was set up to feed data from weather stations in the area to a TV stations computer system, and the results would be displayable on the web to anyone who came along interested in it.

    To save on data transfer, the coder decided to round off barometric pressure to the nearest inch. Barometric pressure is usually given to the 0.01″. I went to check it out shortly after it was deployed and found it very odd that different stations, ten miles apart and at different altitudes, all reported 30.00″ of mercury. An e-mail exchange with one of the meteorolgists brought this to their attention, and the coder fixed whatever needed fixing to report barometric pressure to the accuracy that those in the know expect. (Once the problem was tracked down at their end, the meteorologist I was corresponding with told me just what had happened, and that’s how I know how it all played out and where the error was made.)

    But all this could have been avoided if the weather people writing the specs hadn’t assumed that everyone knew that barometric pressure should be reported to the nearest 0.01″.

    When writing specs for software, spell out any assumptions, don’t assume the programmers know everything you do.

Leave a Reply

Comments are moderated and may not be posted immediately. Required fields are marked *

*
*

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun.

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!