Opinion Driven Development (ODD)

Sean Corfield likes ColdFusion. He also likes regular expressions. Why is this important? Well, because Sean has said on various occasions that his first impressions of both of these technologies was rather different. Of regular expressions he said, "I always used to think regex was a bit of a hammer and that regex fans thought all problems were nails but as I've become more fluent with it, I've seen the light." Of ColdFusion he said, "I thought oh my god, what a horrible language, it's all full of tags." Yet today, Sean is an Adobe Community Expert for ColdFusion. Of course Sean's not the only one. We all learn from our experiences and our opinions about things do change over time. I was late to the party on dependency injection or inversion of control (IoC). So was Ray Camden. On the other hand I was very early on the scene with database abstraction, having started work on DataFaucet on CF5 before there were even CFCs available. At the time nobody really understood or liked the idea, but these days database abstractions are all over the place in ColdFusion, so obviously as a community we've grown to understand and like them.

Hal Helms made a great point recently. In his article titled "Why Ambiguity Is Vital to Success", he said:

Hal Helms: Humans dislike ambiguity and uncertainty. It causes unpleasant physiological reactions: everything from elevated heart rates to shortness of breath and sweating. No wonder we don't like it.

And the groups we participate in confirm the tendency to avoid uncertainty by ignoring information that doesn't fit the group belief system. To take a particularly tragic example, the lower-level information-gatherers had warnings about 9/11 and raised them to their superiors. But these superiors were operating under the belief that nation-states were threats, not mere individuals, and the warnings were ignored.

There's quite a bit of good science on the subjects of neuroplasticity, cognitive dissonance, attitude polarization and the endowment effect which shows that people often make decisions not on the basis of rational criteria, but instead on the basis of previous decisions they've made. What this means is not only that accepting new information is often difficult, but that in many cases, being an expert in a particular field makes it much harder to accept new information, even when the new information is fantastically helpful! I've said before and I'll say again that innovation must be blasphemous. Innovation can't be readily accepted because if what you're doing already makes sense to your colleagues, the odds are you're not the first one doing it! :) As Hal Helms said, "To gain a deep understanding of a subject, you must approach it with what is popularly referred to as a 'beginner's mind'. You must take it on its own terms and learn, as much as possible, without preconception, without trying to shoehorn it into existing categories."

What am I getting at? In order to continue to learn, grow and adapt to a continually changing industry like the IT/Software industry, it is absolutely imperative that we stretch ourselves and do and try new things that are at first uncomfortable. Given a fair shake, these at first uncomfortable ideas can become quite valuable. Hal says:

Hal Helms: As a teacher, I see this very often. Developers who are very comfortable with the relational database model struggle with OO concepts; they tend to want to force-fit these new concepts into comfortable, but incorrect, models.

All learning involves ambiguity. It places the student in a mode of uncertainty, which breeds insecurity. Seeking to escape this unpleasantness, students look to clear guidance on steps to take and often fail to gain a deep understanding of the new technology.

...

Consider this post a call to embrace ambiguity. If web development is fundamentally changing, as I believe it is, the smart move is to put our desire for certainty and comfort on hold. Yes, JavaScript is a strange language to those of us who came to age in ServerLand. But it is such a game-changing technology that, I believe, we ignore it to our great peril.

Something I find particularly interesting about these comments from Hal is that, of all people Hal in particular knows he's not immune to the effects of becoming an expert.

Hal Helms: I've spent a lot of time messing around with CSS and I still don't get it. ...

That is to say, I've never really taken the time to just deeply learn the philosophy behind CSS and approach it not to make it "do" anything, but to just absorb, learn, and internalize it.

It's funny, because years ago, I used to feel the way about JavaScript that I do about CSS. Then, I decided to stop fighting and really learn it. Now, I greatly enjoy JavaScript and have a deep respect for the language.

Could the same thing happen with CSS? I'm going to try again, approaching it with an open mind. I'll let you know how I progress. In the meantime, how about you? Do you have a good understanding of it?

You might have seen something similar happen with Sean Corfield as upon coming from a background in C++ and Java he at first was a major proponent of interfaces in ColdFusion. Later he expressed that he learned to embrace ColdFusion for its own merits instead of trying to (as Hal described) shoehorn it into the model of Java development. Not only did he decide that interfaces weren't really necessary, he started giving a presentation titled "Heresy! Embracing Duck Typing in ColdFusion." (Which interestingly enough had been one of Hal's presentations originally.)

One of the problems that we've faced for a long time in the ColdFusion community is that there have been a lot of myths about the technology. We've had to be advocates and explain to people that yes, ColdFusion IS object-oriented. We've had to content with uninformed people spreading the myths that ColdFusion is insecure, that it's expensive and that there aren't any open-source apps for CF. Heck, over a decade after Allaire changed the face of the web when they introduced it as the first real web-to-database platform, we still contend today with claims from uneducated outsiders that "ColdFusion is dying!".

What's the common theme here? All the people who make these comments about ColdFusion, speak from a place of ignorance. None of them have a deep understanding of the platform. (Especially now that there are not one but two open-source CFML engines and a language advisory comittee that Sean heads up.) And although we like to think we're immune to that sort of thing, of course the same thing happens within the ColdFusion community. I know there have been several times that I've inadvertently commented from a place of ignorance about a project or a framework someone else developed, and on a few occasions it's come back to haunt me. ;)

This is just part of human nature, so rather than getting into flame wars, what we really ought to do is simply learn to identify it and work around it (much like any other engineering task).

So I'm throwing down the gauntlet. ;)

I feel lucky to consider Sean Corfield a personal friend, not just a professional acquaintance. Sean on at least one occasion has said to me personally, "I really don't know why there aren't more people involved in the onTap framework. It's [conventions based] not the way I prefer to work, but there's nothing wrong with that. There ought to be plenty of people interested in that style of development." In particular what Sean still doesn't realize is that he himself is a significant deterent to the growth of the onTap framework community! In the ColdFusion community, Sean is what is called an "opinion leader". Wouldn't it be nice if those of us in the software world couldn't all simply make decisions on the basis of solid, rational evidence? Wouldn't it be nice if we could avoid all the politics and other non-technical impediments to progress? But the reality is we can't. Our communities are still heavily political and very significantly driven by opinions -- specifically by the opinions of "opinion leaders" like Sean Corfield, Joe Rinehart and Hal Helms. So to be realistic, we need to work with those opinions to move forward.

At a job in Boston in 2008 after several weeks of trying to convince folks to try the onTap framework in a non-critical location, the lead developer who should absolutely know better (but didn't) and had spent weeks doing everything he could to avoid receiving any input, asked me the archetypal question: "who's is using it?" It was the only question that interested him. At the time, I shuffled my feet and said something not particularly profound about growing interest. Today I can tell you that, in addition to some previous implementations, it's the foundation of a mission critical application for one of Canada's largest agricultural companies. This is rather missing the point that instead of hemming and hawing over the subject, I should have pointed out that the question is garbage. The answer tells you nothing of value about the platform. It doesn't tell you how stable it is. It doesn't tell you how suitable it might be to address any of your technical challenges. It doesn't tell you how easily you can learn it, how thoroughly it's documented or how easy it is to maintain. Even when people are using it, it doesn't tell you if they're using it well or efficiently. What does it tell you? It tells you how popular it is. How popular was JavaScript when Netscape first introduced it? Not at all. CSS? ColdFusion? SQL? Every groundbreaking technology is unpopular before the opinion leaders like Sean and Joe gain what Hal Helms describes as a deep understanding. It's only after the opinion leaders start to gain a deep understanding and share their knowledge with others that the technology gains widespread acceptance.

The problem I have with Sean right now is that he's inadvertently spreading disinformation about the framework and closing doors for many developers who come to him for his (usually very good) advice. But Sean isn't really aware that he's preventing people from getting involved. Cognitive scientists will tell you six-ways-from-sunday that people aren't aware when things like this are happening. What does happen is that a CF community member, lets call him Bob, happens upon the framework amidst their daily blog reading. Bob sees some information about the onTap framework, it looks interesting and he wants to find out more. But Bob's not prepared to spend the next few hours sifting through my blogs or documentation, or maybe he just wants a second opinion. So like most of us, Bob turns to someone he trusts for information. Who does Bob trust? Obviously, Bob trusts the opinion leaders in the community: people like Sean Corfield, Hal Helms and Joe Rinehart. What I can tell you right now is that none of these guys have a deep understanding of the onTap framework today. All of them would give what advice they have about it from a position of ignorance. And this leads to some myths about the framework. The instant Sean says "it's procedural", he's closed the door on that person trying out the framework even on a trivial application.

The onTap framework is Object Oriented. Sean describes the onTap framework as "procedural". Hmmm... Sean was never a fan of the Fusebox framework actually... right up until he became the lead developer on the project. Version 5.5 of the Fusebox framework is finally backward compatible to Fusebox 4, thanks to some excellent work that Sean did while he headed up the project. And although it was on his list of things to achieve, he didn't get around to backward support for Fusebox 3. Instead I published an upgrade path recently for companies who still have large applications in FB3 (I know of several). It's true that just like Fusebox, the onTap framework has been around since the days of ColdFusion 5, when it was still a procedural platform. Today however, the onTap framework has the same advantage that Fusebox has in this regard, that it's agnostic about your development style. Want to develop OO? Great! I love OO and use it in every application. Want to develop procedural? Maybe you've just got a report you need to hammer out and it doesn't have any complicated behavior. Awesome, bang out that procedural report code!

I say that the onTap framework is not only Object Oriented but also Service Oriented (SOA). Likely part of the issue with Sean is that his information about the framework is a little outdated. He probably doesn't read up on its newer developments very often because it's not his thing. It's only in the latest version, 3.2, that the onTap framework received an IoC Manager feature for managing multiple IoC containers for peer and sub-applications (services). Likely Sean doesn't know anything about this yet. Further even before this latest version and for a good long while now, the only way to create self-installing plugin applications in the onTap framework has been to use an OO (CFC-based) installer API. The plugin architecture is one of the primary selling points of the onTap framework, it has been for a very long time, so how is it possible to say that to use one of the main features of a "procedural" framework requires the use of OO techniques? Sean's a great guy and on many subjects he's very knowledgeable. He's just not up to speed on this one.

I'm going to cover one more item here, and I'm going to do that with a code sample. Sean also describes the onTap framework as "explicit". What I'd like to show is that aside from the fact that I feel "implicit invocation" is a very poorly chosen name for the design pattern, the onTap framework is in some ways more implicit than any of the implicit invocation architectures I've seen in ColdFusion.

First of all, what are we talking about when we say "implicit invocation"? Here's a passage from the Wikipedia article for Mach-II.

Implicit Invocation

In a Mach-II application, procedures and methods are not invoked directly, but rather are invoked implicitly as the result of an event announcement and any related notifications to Mach-II listeners. This concept is also referred to as the Hollywood Principle, which is a reference to the cliché Hollywood phrase, "don't call me, I'll call you." This concept leads to application components that are highly cohesive and loosely coupled, which makes applications easier to test, debug, and maintain.

Keep this phrase in mind, "don't call us, we'll call you".

I decided to use a sample from a real-world application, so I'm going to use some code from an open-source project called AppBooster. This is a Mach-II project from Kurt Wiersma (one of the Team Mach-II members), and it does for Mach-II what the Members onTap plugin does for the onTap framework. It creates a baseline application with user management and security, that other Mach-II teams can use as a boilerplate for new applications. Fortunately it's released under the apache license, so there should be no problems with me using it here for demonstration. :) I asked Matt Woodward who's also on Team Mach-II (and a personal friend), for some feedback on this article before I posted it. He highlighted that although the AppBooster may not be the team's current recommended "best practice" due to new features in recent versions, the fundamentals are still the same.

AppBooster is a Mach-II application, so lets look at how the "implicit invocation" works for Mach-II in this case. There's an XML config file for Mach-II that declares the default set of events, event handlers and views for the application. Kurt also set up ColdSpring for IoC and Transfer ORM for talking to the database although neither of these are part of the implicit invocation pattern. The XML config file looks like this (I'm going to remove some stuff I don't need for this example):

I realize that even after having removed a sizeable portion of this XML file it may still be somewhat daunting to folks not familiar with Mach-II. Don't worry about the specifics for now. I'll explain what this is doing in very simple terms. :) (Matt also mentioned that version 1.8 of Mach-II is going to include some optional auto-wiring features that can be used to reduce the size of the XML by eliminating the need to specify many page-view, listener, filter and plugin nodes for example.)

So we have a site set up using AppBooster and a user, we'll call him Jim, comes to our site. The first thing that happens is the Mach-II framework reads the XML config file and saves the information, then it loads all the listener CFCs into memory when the application loads. Once the framework is loaded, the request starts and the framework looks for an event variable in the URL to indicate wich event to execute. In days gone by, this would have been the server's job, to look for a file name in the URL and if not found to then redirect the user to index.html for example. In the world of ColdFusion frameworks however the majority of requests are piped through only one file. We call this file a "front controller" and usually it's index.cfm, which means that we now need to find some other way of determining what code will execute when Jim reaches our site and that's why we use the event variable in the URL (i.e. index.cfm?event=main.login). When the framework doesn't find an event variable in the URL it then serves up the default event, in this case "admin.home" which you see declared at the top of the properties section of the Mach-II config XML file.

Once the framework knows which event Jim requested, it then executes that event. If you're familiar with ColdFusion's Application CFC, this idea of executing events should be old-hat to you. For example you've got an onRequestStart and an onRequestEnd event, which execute automatically at the beginning and end of ColdFusion page requests. The same thing is happening here, except that, instead of using some ColdFusion code like the onApplicationStart method in an Application CFC, Mach-II is using some XML to declare what will happen when the event occurs. So for example, what might be a CFC method like this:

You have some similar XML like this:

I don't think you would actually have this as an event in Mach-II (especially because the framework has built-in logging features), I just want you to see how an event like onRequestStart might be written if it were a Mach-II event. They're basically the same thing, the only thing that's changed really is some of the syntax and the context. One of the hallmarks of OO development like this is that each piece of the application is encapsulated, so they don't know about the other pieces. In this case the onRequestStart method in Application CFC knows that it should call the logRequest method of the application.logger object, but, it doesn't know anything at all about what the application logger is, what it does with its input or more importantly how it works. That's more or less equally true of the Mach-II application with the small exception that the Mach-II config requires the "logger" listener to be defined elsewhere in the XML packet in the "listeners" section. And unlike how the Application CFC calls the logger method directly on the business object, in the Mach-II sample the "logger" listener is not a business object, but rather an extra CFC that bridges a gap between the XML config file and the business object. So for the Mach-II sample here to really be complete, it would also need to include a listener CFC that looks like this:

Don't worry about the details here. This is simply showing how business objects are fetched and used in a typical Mach-II application.

If you're not familiar with Mach-II you might have noticed that this example has added quite a bit of code to what was previously rather simple in the Application.cfc example. We went from a straightforward "application.logger.logRequest()" call in the Application.cfc to now having both an XML file (with data in two places), and an additional CFC with not one but two separate methods, just to log the request. That's about 10-12 extra lines of code or roughly 200% larger. Does this make Mach-II a bad framework? No absolutely not. Mach-II is a good framework, but the benefits of using it aren't immediately obvious from looking at the code.

After adding this new code, most new programmers who aren't already familiar with this style of development won't immediately see any advantage to this Mach-II code. That's part of the reason why Mach-II took a while to grow an active community, however the Mach-II community had an ace in the hole that the onTap framework doesn't yet have. The Mach-II community had opinion leaders (Sean Corfield is a poignant example), who showed people how using this listener made swapping out business objects much easier because it prevented the views from driving the business objects. What if you started working on this application and then several months later you decided that the business model needed a major overhaul? If your views contained lots of references to the business objects that would be a real hassle. If all your business objects are handled by these listener CFCs on the other hand, what might otherwise be a rather challenging upgrade becomes rather easy, because none of your views would change.

This is the essence of what the other framework authors mean when they talk about "implicit invocation" architectures - "don't call us, we'll call you." The view isn't driving the business objects and neither is it the other way around. Instead the framework drives the business objects and the views and separates them from each other. This way at any point in the chain of objects it becomes very easy to swap out new components as needed.

Interestingly enough, part of the plan for the upcoming Mach-II release version 1.8 is to include a new call-method node in their XML dialect which allows you to omit the listener CFCs. What they've discovered is that a large number of Mach-II event-listeners ultimately boil down to no more than returning myService.getSomeData(). So the upcoming release will allow you to references the business objects directly within the XML instead of needing an event-listener. This still maintains the "don't call us, we'll call you" philosophy because it's still the framework controller (the XML in this case) driving the business objects, rather than the views driving the business objects.

The onTap framework can do much the same thing with a bit less code. Although I say this with a bit of pride in my work, I don't say this to detract from Mach-II. Some people actually prefer to have the lines of code I omit - they feel it helps them to understand what the application is doing. I know Matt Woodward is one of those guys who really likes having an XML config file because he feels it helps him get a good feel for the application. Matt says, "I call it a roadmap." :) I personally don't find myself in that group, that's why I design my framework with an emphasis on reducing the code-footprint of common tasks instead of the Mach-II approach in which a primary concern is the expressiveness of their XML config dialect, i.e. how much information can it convey and how clearly can it be conveyed.

With this in mind, lets look at a few of those Mach-II events from the AppBooster application I chose for my sample.

If you're familiar with frameworks, you might notice the contentArg attribute in the view-page element in each of these events. This is a pretty traditional setup for these kinds of applications. What's happening here is that each event needs two things. First each event needs some data, which it gets by invoking the lisenter methods you can see in the notify tags. Then once it has the data it needs, it passes that data off to a view template which simply uses that data to generate content (usually HTML) just like any traditional ColdFusion page. Once the event finishes executing the framework then returns any content generated by those views to the browser.

So Jim is using our application and he reaches the page index.cfm?event=admin.users.list. The framework then executes the admin.users.list event, which involves

1. notifying the userListener object (remember this is not a business object) to fetch the necessary data for the event (you can skip this and go straight to the business objects in 1.8)
• within which business objects are invoked
2. generating content using
   • the admin.user.search view and
   • the admin.user.list view
3. executing the admin.users.layout subroutine which wraps the previously generated content in the appropriate admin layout (in this case primarily adding navigation).
4. The generated content is then returned to Jim, who obviously has no idea how any of this works! ;)

The onTap framework would do the same thing, however, we've merely turned the notion of the listeners on their side. Instead of having an XML config file that maps a collection of intermediary listener CFCs to then call business objects, we do away with both the XML and the listener CFCs by mapping the name of the event to a convention, in our case a directory structure, and then use that mapped convention as our controller (XML config + listeners).

So Jim comes to our application at /admin/users/list.cfm and the framework then executes the "admin/users/list" event (this is exactly the same as the Mach-II example, only the syntax has changed here). Unlike Mach-II, the framework itself actually has no idea what's going to happen at this point, it's merely implied that some business objects will be called and content generated. This is where we've deviated from the implicit model in Mach-II, which has several specific advantages I'll explain in a moment.

The framework looks for it's "listener" code in these directories in order (and executes the instructions found):

1. admin/
2. admin/users/
3. admin/users/list/

One advantage is that it's not necessary to declare the "admin.users.layout" subroutine in each individual event as you see this is declared three times in the Mach-II XML config files, once at the end of each event. In the onTap framework the code for this layout only needs to be declared in the admin/users/ directory, and it's then inherited by all the subdirectories, so for all three events, admin/users/list/, admin/users/list2/ and admin/users/edit/ the declaration of the layout occurs only once in admin/users/. This kind of implied layout behavior is currently not available with the latest version of Mach-II (as far as I know, although that may be changing, I'm not entirely sure what their roadmap looks like).

What's different in this case in particular is that because our listener is a collection of directories instead of a CFC, our listener can be (and often is) an aggregate of functionality from multiple sub/peer applications. You're probably thinking to yourslef "huh?!" ;) This is one of the key selling points for the onTap framework, the place where integration magic happens.

We all like CFCs, right? And one of the great things about CFCs is that they can be extended. In the design pattern world of course, it's said that you should always prefer composition over inheritance, but that doesn't mean avoid inheritance. It simply means use inheritance wisely. The advantage of inheritance is that it allows us to add to or change the functionality of an object without editing the original object code (the original CFC). So for example Mach-II has a listener CFC and all your specific listeners have to extend the Mach-II listener CFC, which is where they get their basic functionality. The advantage here is that you can add code for your specific needs to the listener like the userListener.cfc we saw before, without editing the original Mach-II listener.cfc. You get the basic listener functionality "for free", without having to edit the basic listener code, and more importantly the basic listener doesn't have to know anything about your listeners (and really it should never know anything about your listeners).

By turning the listeners into a collection of directories, the onTap framework "listeners" can be aggregates of the functionality from multiple sub/peer applications. So lets say our user Jim decides he needs to edit one of those users. In the Mach-II application he clicks the link to index.cfm?event=admin.users.edit&userid=x and is whisked away to the admin.users.edit event. There the framework locates the event and sees that it needs to execute two listener methods, include one view to generate content and then execute the layout subroutine. In the equivalent onTap framework application as I mentioned before, there is no XML config file and we don't need to re-declare the layout because it's already been handled in the admin/users/ parent-event. So the only thing the admin/users/edit event "listener" needs to do then is fetch the data from the user and role services and include a display template to generate content the same way the Mach-II event would.

Here's where the magic happens. What if the AppBooster application didn't include all the things you need for users in your application? What if it didn't include the ability to upload an avatar picture for each user? You could modify the application, right? That's easy isn't it? But then if Kurt publishes a newer version later on, upgrading is going to be a hassle... You could send your modifications over to Kurt for inclusion in that new version... but what if he's already included a similar but incompatible modification from someone else? Well what if you didn't have to worry about any of those questions? What if you could just ADD a file or two without modifying any of the existing code? Wouldn't that be better? Wouldn't that be much the same way you extend a CFC to add functionality without editing the original component?

That's precisely what directory-based listeners in the onTap framework do. And not only can you add your features without modifying the original code like you would extend a CFC, you can even bundle up your additions as a self-installing plugin and pass them out to other people. And they can just click a button and have your addition magically included in their application, tada! And suddenly everyone who wants it has avatar images with their AppBooster. As far as I know, you can't do that with Mach-II, Model-Glue, ColdBox, Fusebox, CFWheels or Edmund... in any of the currently available versions. It's possible that any of these frameworks might in the future develop architectures that could allow you to do this, although as far as I know, none of them have shown any interest yet. As far as I've seen, the closest any of these frameworks come to this is with something like Mach-II's new modules feature, which would allow you to drop AppBooster or another Mach-II application into an existing application as a sub-application. Modules like this allow some integration primarily in the form of navigation, however, they only allow integration at the application-level, where each individual application is kind of an island of its own. Modules don't allow you to alter the behavior of events or the content of views that already exist the way onTap framework events can be modified.

This works because there's very little direct knowledge between components. The framework doesn't know what's in its listeners and the views don't drive the model objects (at least not the way I work). Don't call us, we'll call you. The framework calls your listening code by convention, which then fetches data from business objects, then it generates content by including a view. The view doesn't know what the business objects are, it just assumes or implies they've provided the data it needs. Once views are generated they can be further modified prior to display if necessary.

So to recap, you can't always trust an expert to give educated advice (including me). ;) Often they don't have a deep understanding of the subject matter, so it's important to consider their background when asking their opinion. Sean Corfield as an example came from a Java/C++ background and it took him a while to embrace ColdFusion and duck typing. Sean doesn't have a deep understanding of the onTap framework, and when he gives his opinion about the framework it's still heavily colored by the same prior experiences that originally made him a strong advocate for interfaces before he embraced duck typing. His opinion of the onTap framework rather similar to the opinions of outsiders to ColdFusion who talk about it not being Object Oriented, not being secure, etc. So despite what you may have heard and in spite of the fact that I didn't originally describe it as an "implicit invocation" framework, the onTap framework is an OO, service oriented and even implicit platform.

And if you find me giving out information or advice you think is inaccurate, please correct me as well. Ask Mark Mandel or Matt Woodward about it, they've both had to correct me a few times. :)