For the past month or so I've kept all of my open source projects on hold. It's not because I haven't wanted to work on them... I absolutely have. There are several specific tasks that I'm looking forward to completing. I've kept them on hold because Tiffany and I are preparing for a move back to Texas so I can be near my children again and because at the same time I've been dealing with getting some neurological testing (which went exceptionally well), and because I've had some paying client projects (using DataFaucet and the onTap framework) and I wanted to make sure that with the extra things going on at home I would have plenty of time to ensure that these client projects received all the attention they needed.
I find myself in a short lull here and thought this would be a good opportunity to start sharing the information that I've gleaned from the survey I posted in September. Several other framework authors or communities have done the same thing recently, posting surveys to help inform them about how people are using the tools and give them a better handle on fine-tuning their development roadmaps. Adobe has been doing this recently too with an RIA survey they posted to get similar feedback about how people are using Flex and related technologies. This is the first installment in a series of three or maybe five articles sharing my thoughts on some interesting information gathered in our survey.
When I posted my survey I decided that I wanted to try and take a slightly broader view and include information that might be helpful not only to the onTap framework community but to framework communities in general. That's difficult to accomplish when the chair you're sitting in is firmly rooted behind a specific framework. Even assuming that I can overcome my own inherent biases, there remains the question of other people in the community opting out of the survey because they expect a particular bias. On the other hand, the old addage springs to mind: if you don't gosub a program loop, you'll never get a subroutine. Of course, some folks say "nothing ventured, nothing gained", but I think the other one is catchier. ;)
With this in mind I felt it was important to start by identifying just who had opted-in to my survey. I didn't collect any personally identifying information, but rather I felt it would be important to know in a more general sense what kinds of experiences and attitudes the survey results represent. As an example, most frameworks in today's ColdFusion community use XML as a configuration language and rather than simply assuming that this is a preference for most of us, I wanted to find out how people in the community feel about this use of XML. To determine this I included a question about XML configuration files that asked you to tell me how you felt about them, ranging from undesirable or neutral up to "nice to have" or even required (i.e. you won't use a framework that doesn't use XML for configuration). The results for this question may indicate a gentle bell curve with most people in the middle, or alternatively it could be strongly weighted on one side or the other in a long-tail distribution with people mostly preferring either to use or to stay away from XML. What does this really mean? Does it mean that most programmers have this preference? Maybe it means that most CF programmers have this preference? Or that most novice programmers have this preference? Or that most advanced programmers have this preference? We need some basic context in which to understand the result as being the opinions of a particular group of people with certain kinds of past experiences. So it's important to see for example how much general programming experience the respondents have as well as how much experience they have with a variety of frameworks, etc.
Before I get into the specifics of the results, I will say that some of these results surprised me. :) Also of the results that surprised me, some of them also made much more sense to me after I'd thought about the results for a while.
As I start writing this article, there have been 46 participants in the survey. It seems generally rather difficult to get people to participate in these surveys, so although I would have liked to see more participation, I am quite happy with the number of responses we received. And I will note that in the early part of the survey, the results jittered or "jumped around" quite a bit, swinging violently from one end of the spectrum to another. This is a normal and well known phenomenon in statistics and it's the reason why larger sample sizes are preferred. As the number of respondends approached 46 however, the results evened out quite a bit and with each individual answer representing only a little over 2% of the result, I feel confident in saying that I think we can get some tangible value out of these results.
To better understand the experience of our participants (thank you all for participating, by the way), I started with two very general questions about past programming experience. These were "programming experience" and "frameworks experience".
In the question of programming experience I offered several generalized options for you to provide a "best fit" to represent the kind of work you've done. The options I provided broke down into these general categories:
Non-programming background - 2.2% ColdFusion / CFML Specialist - 43.5% Web Specialist - 37% Java Background - 4.3% Resume Bingo - 13%
So based on the self-report, the majority of the people responding to the survey fall into two main categories of ColdFusion specialists who dable in other languages, shortly followed by web specialists who do a lot of work with several web languages like CF, ASP or PHP. People with exceptionally varried experience with lots of different programming languages contributed a significant minority of the results, while those with little programming background or who identify as having a primarily Java background received almost no representation. (Or I may have phrased the answer poorly resulting in fewer people reporting a Java background.) ;) It's possible that this may be a fairly accurate cross-section of people who work with ColdFusiuon at large, although, I would be very hesitant to claim that it is. There's just too much potential for self-selection for me to declare that "this is representative of the ColdFusion community". Rather I will say that this is simply an indication of the people in our community who chose to participate. :)
I used a similar approacuh to the question of frameworks experience with several generalized options to offer a "best fit". Responses to this question broke down into these categories:
Virgins - 26.1% (never used a framework before) Hapily Married - 15.2% (this is not a dig, I'm kind of in this category myself) Dating but not Serious - 39.1% (used several public frameworks for different projects) Do It Yourselfers - 19.6% (sorry for the mixed metaphor) ;)
So from this question we gather that most of the people who responded to the survey have been comfortable with the "dating scene" where they've tried out several different frameworks on different projects and may have seen some things they like but haven't felt compelled to stick with any specific framework. Over half of the respondents fall either into this dating category or into the virginal category in which they've not used any frameworks before. At first I was a bit surprised that more people hadn't replied that they are in the hapily married category, however after thinking about this for a while it occurred to me that people who have settled with a particular framework they like are less likely to participate in a survey about what people look for in frameworks. Once you've found what you're looking for, you tend to stop looking. ;) This also helps to explain why even the do-it-yourselfers outnumbered the hapily married respondents (albeit by a shallow margin) because I suspect the DIY crowd are more likely to be looking for new ideas.
It's also important at this point for me to note something else about the background of the people who responded to this survey. One of the things that I'd been concerned about in the past is the possibility that the onTap framework may have developed a reputation for being a "procedural" framework. This is in part because a few years ago there were a couple of different ColdFusion luminaries like Jeff Peters who had given a presentation or written an article in which they had looked at the onTap framework and said basically "ahh, well, there aren't any XML configuration files, so it seems to be a procedural framework". A non sequitur if I've ever heard one. Based on these survey results, at least on the surface that fear seems to be unfounded. Although it's possible that some people may have chosen not to participate in the survey because they had this misconception, it doesn't seem particularly likely. Overwhelmingly the people who responded to the survey indicated that they were basically unaware that the onTap framework existed to begin with, much less having any specific notions about what it is or how it works.
The bad news is that this means I got very little data for my last two questions which were designed to help me assess the progress I've made in terms of improving the framework in the latest 3.2 release over previous releases. So although that data does indicate improvement, it's not the large sample size I would like. The good news is that this means my data regarding what people are looking for in frameworks in general should not be tainted in any real way by the biases of people who prefer the onTap framework. Since these results represent a more general audience of people who are interested in frameworks, I feel this means I achieved my original stated goal of providing broader value to the community as a whole. ;)
I actually know why very few people are familiar with the framework. It's relatively unknown primarily because I haven't been very good at marketing it in the past. This is a side-effect of my having a condition known as Asperger Syndrome (a mild form of autism). So now that I know more about my condition, I have chosen to focus more effort toward marketing the framework as well as finding someone to help out in that department by taking over as an official evangelist. This is also part of the way in which this survey came about as a result of my increased effort to grow involvement in the framework community. I could of course guess all day long about the things people want in a framework, but ultimately I think it's better to ask. :)
This leads me to two specific questions I'd like to focus on in this article.
These questions are: How does the average ColdFusion programmer really feel about 1) Java and 2) XML config files.
Some of you may know right away why I was interested to answer these questions. Others of you may be scratching your heads and wondering about my motivation. So for those of you in the head-scratching category, I'll explain. I have for a long time had fairly strong feelings about the process of programming, by which I mean, what techniques really help me to produce better quality code faster. This means I'm biased. But I've also known for a long time that there are a good number of people who are similarly biased, but polarized, holding precisely the opposite of my opinions on the subject.
So while I prefer to limit the amount of XML I write and the amount of Java I use (in spite of the fact that the onTap framework contains more Java reflection than most ColdFusion frameworks) because I feel these things slow me down, I've seen blogs and articles from people claiming exactlyu the opposite. I know I've seen more than one article where a person said "boy, y'know, it's nice having this big XML config file, 'cause I can look at it and see how my whole app works at a glance, and I can rearrange and refactor things at a moment's notice".
Obviously they're having a different experience than I've had, since I've never really felt personally that an XML file either gave me a good "at a glance" feel for the application or that it gave me the ability to change things quickly. In fact I've pretty much always felt that XML configuration files, at least as they're used in the popular frameworks like Model-Glue, Mach-II and ColdSpring, have only slowed me down. (Although they have become much less verbose since I first formed that opinion, so I'm happy about that.)
But regardless of which experience is being expressed in these examples, the expression is still just an anecdote. Sure, it's a viable expression of our experience and formed opinions based on those experiences, but it realy tells us nothing about how the average person relates to these things. Why is this important? Well a little while ago someone sent me an email with a link to an article written by Andrew Powell in which he made some claims that just seemed silly to me.
To summarize, his claim is that ColdFusion does not do x, y and z as well as Java, notably that java is "a true OO language". And implying that therefore because the "vocal minority at the cf.Objective CF9 BOF" were clamoroing for moving the entire business model to Java (Spring, Hibernate, etc), this would be the wave of the future for ColdFusion to be relegated to a basic "controller" and that the CF community as a whole would all gradually become primarily Java programmers. Frankly, I'm tired of the "OO==Java" elitists. And although I readily admit that I don't prefer Java, I think you could hardly call me a slouch in that department. Under the hood, DataFaucet uses JDBC more extensively than any other ColdFusion ORM to my knowledge, so it's not like my opinions about Java are formed out of ignorance. Saying that "java does strong-typing well" to me is like saying that "extreme sports are a really easy way to injure myself!" However I bit my tongue at the time rather than fire off a nasty response on my blog. I've been in too many religious debates already. ;)
But here's the real question. If Adam was right, that trend would have a significant impact on a number of frameworks including DataFaucet, ColdSpring and especially any MVC++ framework like the onTap framework, CF on Wheels or for that matter potentially even Ruby on Rails. So why don't we test the theory? Lets see what people in the community are really looking for.
Ultimately, at least amongst the participants in this survey, Adam's prediction seems to have been unfounded. I asked you to identify various feelings you have about Java ranging from "I hate it" to "I love it" and a few related but off-center questions like "how do you feel about duck typing?"
The community's resounding response with regard to Java is that they feel very pragmatic about it... and specifically that while they feel it's useful, they also feel it's overcomplicated (50%). Could this be a byproduct of the fact that ColdFusion is a very simple language and that CF programmers are "spoiled" by it? Perhaps. But there's also been a lot more interest lately in Ruby for similar reasons in spite of the fact that according to what I've read, Ruby seems to be less mechanically efficient than ColdFusion (due to a lack of or underdeveloped JIT compiling). When you consider that Groovy was created by Java developers essentially in answer to dynamically typed languages like Ruby and ColdFusion and that the CF community is not only going strong but growing, my inclination is to say that if there is a trend in movement, it's toward more CFML rather than less.
Those who felt strongly in favor of using more Java in their CF applications tied evenly with those who reported being totally ambivalent about Java (23.9% in both cases). Which means there appeared to be only as much support for the idea of more Java as there was support for the idea of "whatever man, I just work here". And this is amongst people who are likely to have strong feelings about their work, since they took the time to respond to a survey about it. ;)
Interestingly in spite of the fact that strong feelings about Java were three-to-one in favor of java ("I love it" 23.9% vs. "I hate it" 8.7%), feelings about typing were reversed, two-to-one in favor of duck-typing over the explicit typing of Java (duck typing 28.3% vs. explicit typing 13%). So while the vocal minority Adam spoke of at cf.Objective may have been represented by the 11 people who said they love java and would like to see more of it in ColdFusion, it seems likely that even among those respondents, "strong typing" is not generally seen as one of Java's strengths.
One of the primary arguments for this notion of trending the persistence and domain model awayu from ColdFusion and instead to Java is this idea of mechanical performance. While it's true that a CFC is not as mechanically efficient as a plain old java object (POJO), arguments about the performance of CF in comparison to Java are losing their appeal because the underlying technologies continue to change in ways that reduce the comparative difference in performance between one technology and the next. ColdFusion has always been a "middling" performer and there have always been detractors who complained about its speed, claiming that it just wasn't usable as a result... and they've always been proven wrong. Look at what happened with Twitter. There was this big cat-fight over the fact that Twitter wasn't scaling because they chose to use features of Ruby on Rails that were inefficient, and what happened? Somebody made it more efficient. ;) From what I've read, Python seems to perform even worse than Ruby in terms of mechanical efficiency and its community is also growing. (Don't shoot me -- I read a fairly thorough benchmark article about all these languages, but just can't find it on Google at the moment.)
And that's not even getting into the issue that although some of us do work on "enterprise software" that receives bazillions of hits per nano-second, these are rare exceptions rather than the rule. The rule of thumb is actually that the majority of applications in general are not eBay or MySpace or Amazon. The majority of applications have much more modest requirements.
These notions about mechanical performance not being the number one concern have actually been upheld as well by the results of the survey. One of the additional questions asked in this survey regarded the importance of scalability in a framework. This was actually another result that surprised me. I fully expected this to be a long-tail distribution with near everyone describing scalability as "required" and maybe one or two people describing it as "important". It turns out that's not quite how the results panned out.
So how important is scalability in a framework?
Undesirable - 0% (no surprise there) Neutral - 0% (again, no real surprise) Nice to have - 5.3% (interesting) Important - 55.3% (even more interesting) Required - 39.5% (not particularly surprising except that it's lower than expected)
It would seem that a typical application just doesn't need to jog the track with the six-million-dollar man. There are a lot more corporate intranets and moderately trafficked applications where performance concerns can (and imo probably should) be dealt with on a case by case basis. I'm not saying I've given up on performance tuning - in version 3.2 of the onTap framework I managed to eliminate a fairly large start-up cost by finally creating the lazy-loading library I had originally envisioned. What I am saying is that performance is only a priority for me when I have a particular use case where something the framework does frequently is problematic (like starting up). But even assuming that the framework becomes a bottleneck, there's the example of Twitter, ta-da! Come, see, conquer!
So I've talked at some length now about my first question. ColdFusion programmer's general feelings about Java. Short answer, we seem to be mostly divided between pragmatism and general ambivalence, although we are generally more in favor of Java than we are opposed to it. Feelings about explicit typing seem to be reversed, so we seem to view explicit typing as a liability rather than a benefit of Java and of course that means we view duck typing as a strength in ColdFusion.
My second question regarding how ColdFusion programmers feel about XML configuration files is somewhat similar. I've made no secret about the fact that I've always felt that XML is over-used and particularly not very useful as a framework configuration language. Moreover when Mach-II and Fusebox 4 were first released my initial reaction was "this XML is WAY too verbose". Since then the XML for both frameworks has become much less verbose and for that matter the new XML-free configuration option (implicit circuits) in Fusebox 5 is nearly identical to a suggestion I made shortly after the release of Fusebox 4 and was at the time gloriously shot down! ;)
But again even in spite of the recent popularity of ColdBox, these are merely anecdotes. I wanted to find out how programmers in our community feel about XML confuiguration files in a much more general sense. If it had turned out that people really felt strongly about XML config files then I might have been persuaded to add at least an option to configure the onTap framework with XML. It turns out that's not the case. I asked a number of questions about what people look for in a framework and I'll talk about some of the others later, but for the moment, I'm going to focus on just two. Do you look for XML configuration files? And similarly do you look for convention over configuration?
Both of these questions were asked on a scale from being undesirable to neutral up through nice to have and "required". As a rule, it's challenging to find much of anything in terms of architectural decisions or framework features that programmers (at least this cross-section) will generally agree to being at either end of this scale. So basically there were no long-tail distributions in this section of the survey, they were pretty well all bell curves of some kind, although some were sharper than others. Having said that, of the things that turn people off of a framework, nothing appears to turn people off more than XML configuration files, having received six votes in the "undesirable" column, which is two more than the next runner-up (AJAX integration).
Both of these responses honestly surprised me, in spite of the fact that I dislike XML configuration files myself. There were a number of people who had strong feelings in favor of XML configuration files, however, they were still outnumbered by those who dislike XML configuration, while the overwhelming consensus about XML is that people are ambivalent about it. "Whatever man, I just work here." What I take away from this is that the authors of popular frameworks like Mach-II and Model-Glue like XML because it seems to be convenient for them (or at least familiar), while those using these frameworks don't really have strong feelings about configuration one way or the other.
The total distribution of answers regarding XML looked like this:
XML Configuration Files
Undesirable - 15.8% Neutral - 47.4% Nice to have - 23.7% Important - 7.9% Required - 5.3%
This is good news for DataFaucet and the onTap framework however because both of these frameworks combined have fewer XML configuration files than any single other popular framework. That number happens to be ZERO. Have I mentioned that I don't like XML as a configuration language for frameworks? ;) Obviously based on the results, this isn't going to be a make or break issue for any framework, however, it's nice to know that these frameworks lean in the same direction as the minority of people who do have strong feelings on the subject.
The distribution for the convention over configuration (CoC) answer is more even, with responses spread fairly evenly across the neutral-to-important range. Feelings about CoC are also not as strong in general, with only three people responding in the "undesirable" category (likely the same people who described XML as important or required), and no-one describing it as a requirement. What's interesting to note here is that although fewer people reported CoC as a requirement compared to XML configuration (0 to 3), the number of people who reported CoC as "important" in their decision making was over three times as high as those who reported XML configuration files in the same category (10 to 3). So while positive feelings toward CoC appear not to be as strong, they do appear to be more wide-spread than positive feelings toward XML configuration.
Convention Over Configuration (CoC):
Undesirable - 8.1% Neutral - 32.4% Nice to have - 32.4% Important - 27% Required - 0%
So on the whole, it seems that the average ColdFusion programmer (at least, those who specialize in web development or in ColdFusion specifically and who aren't hapilly married to a framework), tend to have stronger feelings in favor of conventions-based architecture than they do in favor of XML. This in itself is rather interesting when you consider that the majority of popular frameworks for ColdFusion today are dominated by lots of XML configuration files. So I think that what we're liable to see in the ColdFusion community in the next few years is going to be more of the trend that I suggested several years ago when Fusebox 4 was released, and that actually started with the recent popularity of ColdBox and now Fusebox 5.5 moving away from XML and more toward implied or conventional behavior. So I feel very confident in having chosen this architecture from the very beginning, when it was still unpopular and the other framework authors were saying "you have to configure with XML to do things the OO way".
As I said before however, this survey was intended to benefit the community as a whole. I plan to write several more articles (probably shorter) about some of the other information I've collected and eventually I will make the data itself generally available. (I'll have to copy it to a spreadsheet by hand or pony up the cash for the SurveyMonkey subscription.) For the moment I have to wonder how the authors of other frameworks will view these results? Is it likely that Mach-II or Model-Glue or ColdSpring will aquire more XML-free configuration options the same way FuseBox has? Or do you suspect that most of the people using these frameworks are "hapilly married" and so their opinions simply weren't represented in the survey results?
What are your thoughts?