Survey Results Part 2 - What Fusioneers Want (in a framework)

Okay so this isn't the best title ever, but I wanted to spoof the title of the movie What Women Want and this is what I got. :) I think it's appropriate, given the content of this article.

In the film, the main character Nick is a philandering chauvinist who's job as an ad agency executive suddenly changes when his boss, the company's CEO realizes that the company is losing market share to ad agencies who are marketing to a new generation of liberated women. The CEO does what any smart CEO does in that kind of situation - he hires a tough, smart woman to head up the creative department in an effort to save their sinking ship by reinvigorating it with the kind of know-how that will keep them going in a changing market. Now that tough, smart woman, Darcy McGuire is his boss! So Nick does what any smart philandering chauvinist would do: he has a nervous breakdown, gets drunk and accidentally electrocutes himself in the bathtub. When he wakes up, he's able to hear the innermost thoughts of every woman he meets. The horror!... After a brief visit with a psychiatrist however, the panic subsides and he's able to see the advantage in his amazing stroke of luck. Now Nick, who's personal world-view is obviously far from flattering to the average woman, is suddenly able to get a very intimate crash course in all the inns and outs of a woman's world-view. With this new found insight he's able to save his floundering job! Although along the way he chooses to save his soul instead.

My situation may be a bit different than Nick's, but my basic challenge is rather similar. Often in business owners or product managers make the mistake of assuming that they know what everyone wants. Sometimes they do and things work out smashingly, sometimes they don't and the product flops. The smart money is on finding out what people want, so I want to avoid this mistake. But barring a sudden electrical accident in my bathtub and subsequent extrasensory perception, the only way for me to discover what people want is to ask them. I'm not one of these big wall-street businesses, so I can't spend tens of thousands of dollars on focus groups and then bill the American tax payer for the largesse. ;) Instead I have to be more frugal and rely on simpler solutions like direct feedback and free surveys. It's not like I'm in bad company though; Adobe uses frugal surveys to get customer feedback also. So that's what I did. And now I'm sharing those results with the community so everyone can benefit from them.

As I mentioned in the previous survey article, it's important that we not overgeneralize the results of our survey. In other words, these results really can't be seen as the opinions of all programmers because different groups or subcultures have different trends and there's bound to be a certain amount of self-selection. So what we need to do is to determine what kinds of people are answering the survey before we draw any conclusions. So we asked first about the kind of experience represented by the people answering the survey. From this we've determined that most of the 46 people who answered the survey have certain common experiences. In terms of general programming experience most of them are neither novices nor are they software polymaths. Most of them report either that they work predominantly with ColdFusion or that they work predominantly on software for the web.

Following the general programming experience question we asked about people's experiences with frameworks and divided them into categories of virgins (never having used a framework), dating but not committed (using several public frameworks for different projects), hapilly married (preferring a single framework) or do-it-yourselfers (who build their own frameworks - sorry for the mixed metaphor). In this respect we found that although a handful of people responded to say that they're happily married or do-it-yourselfers that these were in the minority, with the majority. Instead the majority of respondents say that either they're in the dating scene or they're new to frameworks. This is by no means any kind of disrespect to the happily married or the do-it-yourselfers, it merely indicates that they're not as likely to participate in these kinds of surveys because they're not actively looking for framework solutions (the aforementioned self-selection).

Because most of the people who responded to our survey indicated a total unfamiliarity with the onTap framework (huh? what's that?), this means that these results can be fairly well generalized across those ColdFusion or more generally web programmers who work with ColdFusion who are looking for framework solutions. Of them I personally was curious to know what their opinions of Java are as I suspected that general opinion is somewhat different from the kinds of opinions frequently held by industry gurus. In the past several years and for likely obvious reasons there's been a lot of "java love" amongst the variety of ColdFusion programmers who are active in the community giving conference presentations and the like. Knowing that the DataFaucet ORM I developed uses JDBC metadata under the hood more than any other popular ORM for ColdFusion, I'd like to think I could be considered knowledgeable about Java. ;) Although I personally find it "useful but overcomplicated", a sentiment that seemed to be shared by most of the folks who participated in our survey. So while there is certainly more java love than there is java hate in our community, I don't think this phenomenon of writing the entire business model in Java is really going to grow much beyond the small handful of people who are already doing that.

So if it's not Java, what do Fusioneers want? No survey can be perfect... and honestly after the survey had started collecting data, I found myself wishing that I had included certain other things in the list of offerings that might be found in a framework. But of course like everything else this is a work in progress and we'll have to come back to those questions later. :) People in general (and not just programmers) tend to hold opinions that fall in the middle, rather than holding extreme opinions about things. This is not to say that people don't hold extreme opinions but merely that extreme opinions aren't particularly common. As a result, when you give people a list of items and you ask them to rate their desirability on this sort of 1-5 scale with 1 being "undesirable" and 5 being "required", most of the questions will receive a resounding "nice to have" response. In the end what you're left with is a small handful of things that the majority of people describe as "important" when looking at a new framework. In our survey these were:

• Easy to Learn / Use 66%
• Thorough Documentation 66%
• Scalability 55%
• Extensibility / Plugins 42%
• Object Oriented Architecture (OO) 36%

Although I was certainly not surprised to see that object orientation made it into this list of framework features people generally agree are "important", I was rather surprised at how it sort of snuck in as the last item on the list rather than being near the top. While the number of participants who described it as a requirement was also significantly high at 33%, this only mirrored roughly the same percentage of people who described OO architecture as merely "nice to have" or even simply neutral (8%). Although no one described OO architecture as undesirable, which can't be said of either XML configuration files or of "convention over configuration".

That being said it seems that the number one concern at least amongst this group of programmers is what folks in the Ruby crowd call "beautiful code", meaning that the framework is easy to work with and well documented. Mechanical performance and logistical flexibility then come in close second and third with OO architecture trailing but still "important".

But we can also glean some useful information not just from the items that people report as "important", but also from the items for which people have "strong feelings". By this I mean that, while the average programmer may have responded only "nice to have" on a given feature, it may still be important to see that a notable minority of them describe that feature as either "required" or "undesirable" at either end of the scale. This answers the question of whether there is a broad but general consensus about these feature being desirable or if there is a more narrow group of users who feel more strongly about them.

Out of twenty one features listed, six of them received more than 10% of their votes in the "required" column. They are:

• Scalability 40%
• OO Architecture 33%
• Thorough Documentation 32%
• Dependency Injection (IoC / DI) 18%
• Easy to Learn / Use 16%
• Extensibility (plugins / filters) 11%

Most of these are the same things that showed up as having a broad consensus, merely in a different order. We notice here that although scalability isn't the feature most widely desired, it is the most strongly desired. In this case OO architecture and documentation are the close second and third... The real surprise for me here is the small spike in desirability of dependency injection features (which I'm glad was a recent enhancement to the latest versions of the onTap framework). Although I must say that having been "late to the party" on IoC I'm a convert and very glad to see that people have strong feelings about using it in spite of the fact that it's an optional feature with the onTap framework (as opposed to Model-Glue for example that doesn't work without ColdSpring).

So that covers what people want... What do people NOT want? There were a lot of features that got a couple of votes in the "undesirable" category, but only two that managed to break the 10% mark. Those two were XML configuration files (16%) and AJAX integration (11%). I must admit to feeling rather self-congratulatory about the results regarding XML configuration files, because while most of the other ColdFusion frameworks depend on them, DataFaucet and the onTap framework are a couple of rare exceptions, having no XML requirements (though some features use XML). When you consider that most of the people who participated indicated that they had no idea what the onTap framework is and that even fewer of them described XML configuration files as "important" (8%) or "required" (5%), it's nice to know that I designed an architecture that predicted the trend. Unfortunately I have to stop patting myself on the back when I see that the next item is AJAX integration, since there's a fair amount of optional AJAX integration in the onTap framework. D'oh!

Lastly here are some results you might find interesting... What features are programmers most ambivalent about? What features are most likely to inspire a yawn? Overwhelmingly those features are:

• Commercial Support Contracts 58%
• Aspect Oriented Programming (AOP) 49%
• XML Configuration Files 48%
• Convention Over Configuration 33%

That honestly surprised me... I know that ColdSpring provides AOP and I've seen several folks talking about wanting to get into AOP even before CS came along... and I know too that Mark Mandel has sold a few support contracts for Transfer ORM, that bastard! That damned lucky bastard... ;) Anyway playful jabs at "competitors" aside, I was fairly surprised by these results. Although there's a fair amount of ambivalence about printed books, demand for them still outweighed the demand for commercial support. I was also surprised by general ambivalence toward XML configuration files, I honestly thought there would be more votes in the "important" and "required" columns for that because I've seen so many bloggers or framework authors talk about how they feel the config files for Mach-II or ColdSpring help them see their application "at a glance" and make changes quickly. Convention over configuration surprised me less, but it did still surprise me, given that I thought the CF community had become pretty familiar with the concept by way of osmosis from the Ruby on Rails community, but I guess it's not as widespread as I thought.

Anyway, that's all I've got for now. I may post another article or two about these survey results later after another look at them. I'm looking forward to hearing people's thoughts. :)