I'm reminded that when ColdFusion 7 was released, it shipped with a small bug in the XForms implementation which caused multiple-select input elements to display with none selected if a user had previously selected multiple options. It had turned out to be something fairly simple, an oversight in their default XSL sheet and at the time I'd provided the fix for Jeff Small (although at the moment I can't find his blog entry where he posted it for everyone else -- send me the url if you have it and I'll post it here). I'm sure it was something the folks on the CF development team just hadn't antiscipated, after all how often do you use a multiple-select? (Of course, I say this expecting a quick response from the couple of folks who use them frequently.) :)

It's sort of come back to haunt me... In the previous versions of the onTap framework, I'd built in query-driven select input elements. The first iteration of these were unfortunately painfully slow. The problem at the time was that I was making it create a whole new html element structure for each option, instead of simply holding on to the query until display time. It turned out to be far too much overhead, and just as well most of the data in those structures was going unused anyway. So I then moved to the more logical approach of storing the query in the element structure for the select input and looping over them to create the options at time of display. This worked pretty well and even allowed me an easier way of fetching the current value of the select input by using the native ValueList() function. However, when setting the current value, I was still looping over each record in the query, like this:

You can see from this code how I can then use ValueList(input.query.inputvalue) to fetch the list of currently selected options. Well this was great, and then shortly after I started working on version 3.0 of the framework I realized that there may be an even more efficient way. Instead of looping over the query one record at a time, I could set the entire query all at once and it could potentially be much more efficient. Something like this:

So in this case the loop only processes one iteration for each selected option, instead of for each record in the query. In some cases where there are a lot of options in the list this can be much more efficient. I tested this after I implemented it and it is much more efficient when the select list is excessively large: hundreds of options or more, which may not be the best idea in the first place, but that's another blog. For most uses though, the efficiency probably won't change much from the last version.

The problem lies in that getSelectedIndexes() function (which is a simplification for the sake of the blog -- suffice to say that it creates a list of query indexes that should be selected without looping over the query). The algorithm I had for determining which query indexes were selected turned out to have a minor flaw in it and so where I had a query containing the values "en,en_CA,en_US" (and a few others), it was selecting all options beginning with "en", so the options for CA and US were being additionally selected when only "en" should have been. Of course I fixed this yesterday and uploaded a new build of the framework core code a few minutes ago.

I will give one caveat and that's the form tools do assume that you won't have any linefeed characters (ASCII char(10)) in your option values. So if for some reason you need that particular character in your data, you'll have to replace them with something else before displaying the form and after submitting it. Probably most people won't have any issues with that, since the inputvalue column will usually contain either integers, UUID strings or some other equivalent. Speaking of which, this is probably why I didn't catch this sooner, because I tend not to use autonumber / identity columns (though they're better supported now), and since most of my select elements contain a UUID equivalent, the values generally don't offer that kind of overlap.

I know that verity includes a lot of advanced search features, but I honestly just haven't worked with it much. So I'm not sure if verity offers an "and/or" keyword feature, though I wouldn't be surprised. Then again verity requires you to create and maintain collections and that means double-maintenance if the data is stored in your database and you don't always need the other features verity has. So a while back I decided I wanted the ability to include and/or keywords in searches, (similar to google I suppose) and rather than copying and pasting that code every time, the framework's SQL abstraction layer gave me the ability to simplify the process of creating those and/or filters. The framework code looks something like this:

This code then takes whatever you've entered in the search form, breaks it apart into phrases separated by the words "and" and "or"1 and spreads those phrases across the title, keywords and contentText columns in the tbl_content table. Being able to do this with just one line of code is awfully convenient. The resultant query is something like this:

searched for "fish and chicken"

So with this sql output, you can see that the query will return any record that contains both words, but it also returns results where the words are in different portions of the content, for example the word "fish" in the title and "chicken" in the contentText, so if there's an article titled "let's go shark fishing" with the phrase "are you chicken?" in the contentText, this search will return that record. What makes this really nice however is that as the number of columns increase and/or the complexity of the search string increases, the SQL statement becomes very large and very complicated rather quickly, however, the andOrFilter facade which handles the thankless task of creating the SQL syntax never becomes any more complicated than the one simple line above2.

Today I realized that the facade was limited to only accessing columns in a single table, so although it worked it was still not as flexible as I wanted it to be. I was working on a contact manager and needing to include two columns from joined tables in the and/or column list. So I updated the facade to allow that and now you can also do this:

It was a really good thing for me, because the query I was working on was massive... Check this out: 3

SELECT DISTINCT [ONTAP].[DBO].[TAP_CONTACT].[contactid],[ONTAP].[DBO].[TAP_CONTACT].[authorityid],[ONTAP].[DBO].[TAP_CONTACT].[contactauthorid],[ONTAP].[DBO].[TAP_CONTACT].[contactmemberid],[ONTAP].[DBO].[TAP_CONTACT].[namelast],[ONTAP].[DBO].[TAP_CONTACT].[namefirst],[ONTAP].[DBO].[TAP_CONTACT].[namemiddle],[ONTAP].[DBO].[TAP_CONTACT].[namepreferred],[ONTAP].[DBO].[TAP_CONTACT].[nametitle],[ONTAP].[DBO].[TAP_CONTACT].[namesuffix],[ONTAP].[DBO].[TAP_CONTACT].[contactlocale],[ONTAP].[DBO].[TAP_CONTACT].[contactregionid],[ONTAP].[DBO].[TAP_CONTACT].[contacttimezoneid],[ONTAP].[DBO].[TAP_CONTACT].[contactcreated],[ONTAP].[DBO].[TAP_CONTACT].[contactmodified],[ONTAP].[DBO].[TAP_CONTACT].[contactbirthdate],[ONTAP].[DBO].[TAP_CONTACT].[contactgender],[ONTAP].[DBO].[TAP_CONTACT].[contactphotouri],[ONTAP].[DBO].[TAP_CONTACT].[contactlogouri],[CR].[contactrelevance],[PHONE].[methoddata] AS [phone],[EMAIL].[methoddata] AS [email],[NOTE].[notetext] FROM [ONTAP].[DBO].[TAP_CONTACT] INNER JOIN [ONTAP].[DBO].[TAP_CONTACTLANGUAGE] ON ([ONTAP].[DBO].[TAP_CONTACTLANGUAGE].[contactid] = [ONTAP].[DBO].[TAP_CONTACT].[contactid]) LEFT JOIN [ONTAP].[DBO].[TAP_CONTACTRELEVANCE] [CR] ON ([CR].[contactid] = [ONTAP].[DBO].[TAP_CONTACT].[contactid] AND UPPER([CR].[memberid]) = 'MBRADMIN') LEFT JOIN [ONTAP].[DBO].[TAP_CONTACTMETHODXREF] [PHONEXREF] ON ([PHONEXREF].[contactid] = [ONTAP].[DBO].[TAP_CONTACT].[contactid] AND [PHONEXREF].[methodisprimary] = 1 AND UPPER([PHONEXREF].[methodmedium]) = 'PHONE') LEFT JOIN [ONTAP].[DBO].[TAP_CONTACTMETHOD] [PHONE] ON ([PHONE].[methodid] = [PHONEXREF].[methodid]) LEFT JOIN [ONTAP].[DBO].[TAP_CONTACTMETHODXREF] [EMAILXREF] ON ([EMAILXREF].[contactid] = [ONTAP].[DBO].[TAP_CONTACT].[contactid] AND [EMAILXREF].[methodisprimary] = 1 AND UPPER([EMAILXREF].[methodmedium]) = 'EMAIL') LEFT JOIN [ONTAP].[DBO].[TAP_CONTACTMETHOD] [EMAIL] ON ([EMAIL].[methodid] = [EMAILXREF].[methodid]) LEFT JOIN [ONTAP].[DBO].[TAP_CONTACTORGMEMBER] ON ([ONTAP].[DBO].[TAP_CONTACTORGMEMBER].[contactid] = [ONTAP].[DBO].[TAP_CONTACT].[contactid]) LEFT JOIN [ONTAP].[DBO].[TAP_CONTACTORGTREE] ON ([ONTAP].[DBO].[TAP_CONTACTORGTREE].[orgid] = [ONTAP].[DBO].[TAP_CONTACTORGMEMBER].[orgid]) LEFT JOIN [ONTAP].[DBO].[TAP_CONTACTORG] [ORG] ON ([ORG].[orgid] = [ONTAP].[DBO].[TAP_CONTACTORGTREE].[orgparentid]) LEFT JOIN [ONTAP].[DBO].[TAP_CONTACTNOTE] [NOTE] ON ([NOTE].[contactid] = [ONTAP].[DBO].[TAP_CONTACT].[contactid] AND UPPER([NOTE].[memberid]) = 'MBRADMIN') WHERE ([ONTAP].[DBO].[TAP_CONTACT].[authorityid] IS NOT NULL OR (UPPER([ONTAP].[DBO].[TAP_CONTACT].[contactauthorid]) = 'MBRADMIN' AND [ONTAP].[DBO].[TAP_CONTACT].[authorityid] IS NULL)) AND (((UPPER([ONTAP].[DBO].[TAP_CONTACT].[namelast]) LIKE '%NYGMA%' OR UPPER([ONTAP].[DBO].[TAP_CONTACT].[namefirst]) LIKE '%NYGMA%' OR UPPER([ONTAP].[DBO].[TAP_CONTACT].[namemiddle]) LIKE '%NYGMA%' OR UPPER([ONTAP].[DBO].[TAP_CONTACT].[namepreferred]) LIKE '%NYGMA%' OR UPPER([ONTAP].[DBO].[TAP_CONTACT].[nametitle]) LIKE '%NYGMA%' OR UPPER([ONTAP].[DBO].[TAP_CONTACT].[namesuffix]) LIKE '%NYGMA%' OR UPPER([ORG].[orgname]) LIKE '%NYGMA%' OR UPPER([NOTE].[notetext]) LIKE '%NYGMA%'))) AND UPPER([ONTAP].[DBO].[TAP_CONTACTLANGUAGE].[contactlanguage]) IN ('EN') ORDER BY [cr].[contactrelevance] DESC,[ONTAP].[DBO].[TAP_CONTACT].[namelast] ASC,[ONTAP].[DBO].[TAP_CONTACT].[namefirst] ASC,[ONTAP].[DBO].[TAP_CONTACT].[namemiddle] ASC

Aside from the fact that the sql abstraction layer escapes all the column names for me and ensures case insensitivity (even with a case-sensitive database collation), creating this query required me to write a remarkably small amount of actual code, much of which is even reused in other queries.

So anyway, I uploaded a new archive with the enhancement for multi-table and/or filters4.

1. Actually it's a bit more sophisticated than that even because it uses localized strings for the and/or keywords for international use, so for example, german users can use the keywords "und" and "oder" in their search.
2. And really, without a facade like this, and/or keyword searching is one of those features that just never gets implemented because, as nice as it may be for the users, it's tedious and never makes it into the list of priorities we can spend time working on.
3. Thanks to search.getSyntax() for the sql syntax - this lets you output the syntax instead of executing the query for use in generating scripts to be executed elsewhere, as in my case it's easier to debug a query sometimes by copying and pasting the SQL syntax into an IDE provided by the database vendor like Query Analyzer. This way I don't have to sit through the tedium of copying all the query-param values, sifting through the query and replacing question-marks, because getSyntax() performs the merge for me.
4. I also discovered that when using filterGroups (they allow you to wrap parenthesis around groups of filters like in the query above) if you misspelled the name of a table alias, it was producing an unusual error -- it's supposed to throw a custom "column not found" error, which it wasn't doing, so I fixed that also.

Apparently I somehow managed to get a copy of an archive called "tapmce.zip" uploaded to the RIAForge server containing all the sourcecode for the Members onTap plugin project. Someone let me know about it, so I recreated the archive for the project & uploaded it and made sure it's the correct source this time. I'm sure this is related to the recent switch from my own server to RIAForge and having had to change all my batch files for distribution.

For a long time now the onTap framework has included features for managing timezones, something for which ColdFusion's support is traditionally somewhat weak. But timezones are a messy issue and notorious for creating unexpected challenges in software development. For example, it's generally considered a best practice to store dates in UTC (formerly GMT) timezone because that time zone is neutral, having no offest and no daylight savings adjustments, meaning that 5-o-clock is 5-o-clock is 5-o-clock. Really what this means is that it's a good baseline from which times for other timezones can be extrapolated.

That's actually the intention of UTC in the first place and why it is that many platforms like ColdFusion and SQL Server have native functions for getting the current time in UTC or converting times to and from UTC. SQL Server has a GetUTCDate() function for example although I'm not aware that it has any native functions for converting to or from. So the onTap framework by default will assume that you want to store dates in UTC and when you enter dates into forms, if you've added a time, it will convert that date/time value to UTC before it's stored. If there's no time, it leaves the date alone.

(Of course if you don't like the fact that the framework handles this time-zone converting for you and you'd rather do it all yourself, you can disable it by setting the timezone for the request to UTC. If there's no offset and no DST adjustment for the current timezone, then it will only be adding or subtracting zeros from your dates, effectively canceling out the feature.)

A while back I implemented a feature in the SQL Abstraction tools that allows you to use "now" as an alias for the current time in UTC when performing inserts or updates. I did that because I was finding that I wanted to insert the current UTC time frequently (you might say routinely) and at the time it was a bit of a challenge because the tools' default behaviors were converting the date from the current timezone of the request (which belongs either to the user or to the server). So in order to get the "now" UTC I was having to first convert the UTC date to the local time zone so that it would be the current time in UTC when it was converted back. So the fix that made it much easier to handle that common task was allowing me to assign it via "now", i.e. myobject.update(SomeDateColumn="now"). That's pseudocode -- the real thing is a bit more involved but you get the idea.

I was just noticing a similar issue with the forms this evening. It wasn't quite as simple as I would like to create a date input in a form with a default value of the current time (or date). So I decided to fix that by adding a feature to the input validator that will interpret values of "now" or "today" as the current date, with or without the time respectively. So in my case, the input element is

And that gives the input element a default value of today's date plus time. If everything is working properly, it should match the clock in my system tray. :) It's important that the input be validated as a date, otherwise the framework doesn't even try any date massaging. They also get massaged a second time when you use the framework tools to validate the form on the server side, so it's important also make sure you are performing the server-side validation unless you're doing your own date massaging. On the other hand if you use tap:dbtable in your form to automate validation for the form based on your database metadata, then you won't need to include the tap:validate attribute you see in this snippet since the automation will add that for you.

Part of the upshot of this is that if javascript is disabled in their browser, users might be able to type "now" or "today" into the form to get those values. (If they have javascript enabled, the default javascript validation will stop them, although if you really like the idea of using "now" and "today" in forms, you can change the regular expression the framework uses for javascript validation.

Oddly, while I was in there adding this, I noticed also that the localization directories were being included in the wrong order... So /_application/l10n/en/us/100_dateformat.cfm that overwrites the date format for the united states to be mm/dd/yyyy was being included before instead of after /_application/100_i18n.cfm where the default format of dd/mm/yyyy for the rest of the world is set. Basically it wasn't using US date formatting in the US. So I fixed that. It seems odd that I hadn't seen that earlier, since I know it was working before. Anyway it's fixed now (and for most folks wouldn't have been a deal-breaker anyway, since most of us in the US could just set the value application-wide to use US dates).

p.s. I updated the zip archive as usual.

So I realized today that the last couple of zip archives I posted were probably *not* updated from the new svn repository on the riaforge site but rather older copies from the svn repository I had on my local machine. They were properly zipped (which was the last issue, unrelated to svn) and therefore would install properly into the framework via the plugin manager... Thing is, exporting and zipping archives and uploading them is tedious and frustrating -- or at least I find it to be... and so I have these batch files on my local machine that I use to pack them up and at one time the bat would even upload the archive to the public server when I maintained a dedicated server for the framework.

So when I was done committing my changes to svn all I had to do was double-click on my bat and walk away.

Instead of

1. delete folder
2. create folder
3. right-click -> svn export
4. enter repo url (I maintain several)
5. wait for export to complete
6. right click -> send to -> compressed folder
7. right click -> ws ftp upload
8. select server -> select subdirectory -> upload

So I had these bats which worked fairly well for me for a while, and may or may not have been a product of my never having learned ANT.

And when I set up the projects on RIAForge I apparently forgot to update the distro.bat (which is called by the bats for individual projects so they're not all copy/pasted) to use the new svn repositories on the RIAForge server. So when I realized this today I started updating the projects again and getting ready to upload new zip archives (current archives) and immediately started getting this error from svn:

Honest to god, I was using Google to solve this and it took me over an hour of pulling my hair out (and Google was no help)...

I'd like to take this opportunity to reiterate that CASE SENSITIVITY IS A BOIL ON THE ASS OF SOFTWARE!

We are well past the point at which there is any significant performance gain from the choice for a language to be case sensitive, which was the original impetus. Back in the day, when "no one would ever need more than 640k of memory" things were different, and a person might actually notice in some cases when software took the time to convert all its data to upper or lower case before performing comparisons if there was a lot of data being moved around... But on today's machines, no one will ever notice.

We keep case sensitivity around today because programmers forgot the original impetus and are now married to this disgusting notion that case sensitivity is better because it's "precise". There is not now, nor has there ever been a "value added" to a piece of software as a result of its being case sensitive. Software that supports *nix should for everyone's health and sanity be case insensitive and force their users to save files in lower case.

Harsh? This isn't the first time I've seen it happen specifically with svn. The last time it took 3+ hours of hair pulling from another friend of mine before he realized that their svn repository was royally f*ed up because some file had been given a name that varied only by case from a file next to it on a *nix machine (i.e. application.cfm vs. Application.cfm) on a project that had developers using both *nix and windows platforms for development. Again in his case as in mine, NO VALUE WAS ADDED by the support of case sensitivity. It took him 3+ hours to isolate that issue because the errors caused by case issues almost never indicate that case might be an issue, and it's always the furthest thing from your mind when you start troubleshooting. Which immediately sends you off into one of these 3 hour hair-pulling frenzies. Nor does any appeal to Google or any user-groups or mailing lists help either, because it's also the furthest thing from their minds.

And just to underscore my point... There's always that case in development when you're having to convert data from one language that's not case sensitive to a language that is... like for example, SerializeJSON! Great function! Awesomely helpful! Totally easy to F*** UP unintentionally! I'm not complaining about the folks at Adobe (although honestly I would have wrapped an lcase() around all the keys being serialized for everyone's sanity). Thing is, what you get out of a SerializeJSON function will vary based on something that you're usually not thinking about... for example:

struct = XMLParse("<x foo='bar' />").x.xmlAttributes;

foo is lower case

struct = { foo = "bar" };

foo is UPPER case (I think)

struct = structNew(); struct["foo"] = "bar";

foo is lower case

struct = structNew(); struct.foo = "bar";

foo is UPPER case

So ... if someone else is editing a totally separate section of your application and decides to change for example:

<cfset foo.a = "" />
<cfset foo.b = "" />
<cfset foo.c = "" />
<cfset foo.n = "" />... 

to

<cfloop index="c" from="#asc('a')#" to="#asc('z')#">
  <cfset foo[char(c)] = "" />
</cfloop>

Suddenly your javascript receiving the SerializeJSON data blows up in your face because somebody decided to replace 26 lines of code with 3 further up for legibility. You come back in and fix it by putting in [ucase(char(c))] (so your javascript is getting capitalized letters again). So the next guy comes along and being a ColdFusion guy thinks "gee, aren't those the same thing?" and removes the ucase() because it's not needed, not realizing that it's necessary for the SerializeJSON call in another template.

In my case? My subversion repository is named MembersOnTap. Again... it's always the furthest thing from your mind... When I set up the riaforge projects, it didn't occur to me that having the url "http://onTap.riaforge.org" might mean that the svn repository would be case-sensitive, and again the same for the sub-projects, which are now

• http://svn.riaforge.org/onTap
• http://svn.riaforge.org/plugtap
• http://svn.riaforge.org/MembersOnTap
• http://svn.riaforge.org/tapmce

So... once again... new zip archives just published for the individual projects.

Enjoy!

And apparently if you use SVN, mind your p's and q's.

Just updated the Members onTap plugin (also uploaded new archives for tapMCE and the Members onTap plugin because I realized the archives I uploaded before were improperly zipped).

There are all manner of things that can go wrong with an installer that's as complex as the Members onTap installer especially when it uses AJAX to break up the request (which was done in an attempt to get past the fact that the ColdFusion administrator settings frequently seem to override the requesttimeout setting with the cfsetting tag, causing long-running requests in the installer to terminate early leaving the installer in an ungainly broken state)... There are just a huge number of things happening when you install the Members onTap plugin, so while I added some support in the latest version for it to use cfthread (which in itself was limited by the fact that most of these things can't be done out of sequence), it still takes a couple minutes to install. (And I've only realized in retrospect that someone who was having problems with it over a year ago was probably being fouled up by his hosting provider limiting requests to 60 seconds. I hate that setting.)

So a while back I decided to redraft the installer to allow you to install it with javascript disabled. Apparently I neglected to test it in my dev environment with script disabled the last time I made changes. So while it worked for me with script enabled it was throwing a rudimentary variable undefined error if you turned it off. So I've just fixed that and committed to SVN and uploaded a new zip (that incidentally will also work with the plugin manager's archive upload utility, since it got zipped correctly this time). :)