Drupal: open-source CMS

Drupal has been pretty popular with the various Belgian governments. Any new websites created for a government agency are supposed to use an open-source CMS. Drupal's popularity (and its Belgian roots?) have made it the perfect candidate. Heck, even the king uses it. A new initiative is Drupal-as-a-service: pick your features and click "Create website". Off you go!
With the government's zeal to convert everything to open-source the buzz has been increasing. Universities request websites built on Drupal, marketing people want their new website in "Druple".

So what's up with this love for open source? I will now wield my razor-sharp intelligence to shamelessly make up what I think is most likely:

• Avoiding vendor lock-in
  Every few years, the government is required to field a project to get price quotes from a number of parties, and then pick the best price quote.
  If a project has been written using a proprietary framework/platform/language/etc the best candidate will likely be the candidate who initially set up the project. Any other candidate would need to either swallow the costs of a steep learning curve, or be more expensive.
  Not so for open source. Anyone who knows Drupal/Joomla/Wordpress/But mostly Drupal can pick up a website and go! Money savings galore!
• Open data
  Whether you have access to the source of your CMS or not, the data on your site is yours. If you switch to a different system, you want to keep your data. If a proprietary system perishes, your data could perish with it. In an open source system, you have the ultimate data spec: the source code.
• Security updates
  Whenever a security leak is found, noble developers will flock to it, and release a security patch in no time at all. Government sites need to be secure, right?
• Oh! The modules!
  If you have a Drupal-based website and you want to add a online shop, you only need to install Drupal commerce. Works instantly! Hardly any configuration required! Money savings galore!
  More to the government point: integrate with the Belgian electronic ID card (currently has some security issues).
• Open source means no license means cheap! Right?
  No it doesn't. And I doubt the government is stupid enough to believe this.

Some of those are pretty solid reasons. I want the government to spend my tax money wisely (frugally even), and store my personal data on secure servers! Who wouldn't?

But here's the thing: when we make a Drupal website, the customer comes back and tells us

"We don't like the way this editing form works. Can you make it do x and y instead?"

And we can. The customer's request makes sense: core Drupal is pretty spartan, and a lot of modules are a tad confusing. We just code up a nice custom module that does exactly what the customer wants. 

But here's the catch:

Those custom modules can be neither open-source, nor secure (nor very well-written for that matter).
That includes some modules that I've written. It's vendor lock-in all over again. 

We inherited a Drupal project from a competitor, and the last few weeks I've been poring over custom modules that implement some form of url rewriting using taxonomy terms. It's a very neatly commented and indented mess, and has a fair number of bugs - I'm not sure I can do better. I'm familiar enough with Drupal, but my employer has had to swallow these costs anyway. You can't pick up a major website and run with it. Open source doesn't matter that much here.

As far as the open data is concerned, I think that's a valid point. I have written a proprietary CMS in .NET, and I encrypt all website data using AES-256 before storing it in the database using an 'optimized' version of Base64. I double-dare customers to switch to a competitor (I kid, of course). But I could do such a thing, call it 'Enterprise-level security', and sell it to the banking industry.
Using open source, getting my data is not necessarily easier, but I know it can be done.

Lastly, about the security updates. The sysadmin is usually the weak link here. Nobody will likely ever find a security hole in my proprietary CMS, because nobody cares. There will, 100% sure, be security updates for Drupal. For all eternity: either you patch or you're vulnerable. My CMS may actually be safer than Drupal. Not by virtue of its technical qualities, but just because it's a tiny fish in a pretty small pond.
After all: wasn't Apple repeatedly the first to be hacked at pwn2own ? 

Menno

SSMS: OutOfMemoryException executing a large SQL batch

So you're executing some beast of an Sql batch using the Management Studio, for instance the result of "Generate Scripts" on another database. Then all of a sudden you get a OutOfMemoryException. Even when just keeping the entire script in memory, poor SSMS is hanging on by its fingernails. How could you expect it to also execute that file, and give you the results?

Thankfully there is a command-line Sql server client Microsoft thoughtfully provides together with Sql server. If you're not able to import a large sql file using SSMS, navigate to the Binn folder of your Sql server folder (C:\Program Files\Microsoft SQL Server\100\Tools\Binn here) in Command Prompt, and type the following:

osql -S databaseserver -U username -P password -d DemoDatabase -i c:\demo.sql

That executes the file C:\demo.sql on databaseserver in the context of DemoDatabase using the login data provided. This will scroll a bunch of query results in your command prompt window. If you'd rather examine these results in detail later, the -o parameter writes this info to an output file:

osql -S databaseserver -U username -P password -d DemoDatabase -i c:\demo.sql -o c:\output.txt

That should work, even when SSMS chokes on the sheer size of your query file.

text-transform: uppercase subtleties

So last week I received an interesting question from a customer. He complained that in Chrome on Mac, the category 'Soßen und Dips' ("Sauces and dips") on the german (obviously) version was rendered as 'SOSSEN UND DIPS' (wrong), rather than 'SOßEN UND DIPS' (right). Notice that ß and SS are semantically identical - a latin1 collated database will consider those strings to be identical.
On Internet Explorer, the ß was displayed fine.
It didn't take long to figure out that the menu item had text-transform: uppercase applied, and that this caused the transform of ß to SS. What's more, Safari and Firefox also displayed SS instead of ß. My Google search led me to page https://bugzilla.mozilla.org/show_bug.cgi?id=354451 . This page indicated that transforming ß to SS was deliberate, and not a bug at all. Der Spiegel suggests that ß should, in capitals, always become SS.
So now I only need to convince the customer that their browser can spell their language better than they can.

Menno




As an interesting aside: a basic test case to check browser behavior sees Internet Explorer 9 also rendering SS, not ß. Changing the document mode down to IE7 shows ß again. The actual online page is in IE9 mode, and does not have a X-UA-Compatibility meta tag. The question then becomes, why does it show ß? No idea yet.

In which I learn about multithreading performance

... or an exercise in optimizing IndexOf on List<T> for multicore.

So I had a great plan.
.NET 4 introduced AsParallel() into LINQ (well, 'PLINQ' , but try using that word in conversation without giggling uncontrollably). So how about re-implementing some of these methods in a multithreaded way for .NET 3.5?
To get my feet wet, I decided to start off simple: I re-implement IndexOf on List<T>. Should be easy:

• Perfect for splitting up, each thread just takes a range of the list
• No need for critical sections

So I wrote an extension method IndexOfParallel<T>:

• Creates as many threads as there are logical cores,
• pass them a list range, and a status object
• each thread checks its range of the list, sets the status object to found and returns
• main thread calls .Join() on each created thread

Done! Let's check how much better it performs compared to the regular IndexOf!

Looking for a thousand random ints in a list of a million ints:

• Regular version: 8565064 ticks
• Multithreaded version: 542255931 ticks

Whoops! Only sixty times slower!

Perhaps a million ints is too small to have the proper effect. Let's try ten million:

• Regular version: 9131999 ticks
• Multithreaded version: 149267210 ticks

Well, that's only sixteen times slower now. Progress!

I check the source for IndexOf on List<T>, which uses Array.IndexOf on its internal array. Turns out it uses a native method for basic types. Clearly I can't improve on that. Perhaps I should compare against a List<string> (100,000 items):

• Regular version: 7044903 ticks
• Multithreaded version: 93284309 ticks

Thirteen times slower. This is starting to piss me off. Why is it slow? What is slow?

Perhaps I shouldn't be trying to reinvent the wheel. What happens if I just use the overload of IndexOf on List<T> that takes a range instead of writing my own loop? Clearly I'll be losing my early exit, but with enough cores maybe it evens out:

• Regular version: 7563221 ticks
• Multithreaded version: 140339244 ticks

Nope! Twenty times slower! If I fire this version off against a List<int> of a million, I gave up waiting for it to finish at all. Extremely slow. I suspect the native method lock the List's internal array in memory, and that may be marked as a critical section.
Whatever the reason, let's scrap this, and go back to my own loop. What if we unroll the loop, say, four times? If that speeds it up significantly, we can deduce the loop implementation is slowing everything down:

• Regular version: 7122743 ticks
• Multithreaded version: 89569552 ticks

A small gain: twelve times slower. Clearly the loop is fine. So the thread creating is probably to blame. Let's not create our own threads, and use the ThreadPool:

• Regular version: 7034966ticks
• Multithreaded version: 5879994 ticks

Whaaa! Success! It's not much, but I finally beat the built-in version. Let's see if we can improve it a bit more. How about turn the status class into a struct, and avoid the getter for the properties by turning them into public fields? Obviously we need to ref the method parameter:

• Regular version: 6979197 ticks
• Multithreaded version: 4006500 ticks

Awesome! If we increase the number of strings in the list to a million, the effect increases too:

• Regular version: 76802790 ticks
• Multithreaded version: 36655462 ticks

Twice as fast! I'm sure this can still be improved significantly. I still need to figure out at what list size it makes sense to switch to multithreaded. If you're working on a List<T> where T's implementation of Equals is slow, it should do better. There are a number of lessons learned already though:

• This method's only worth it in a few situations. Mostly just not.
• Only create new Threads if you will hold onto them for a long time. Creating new threads takes long. 
• Prefer using the ThreadPool.
• Measure Measure Measure!

It'll be interesting to see how we manage reimplementing .Where(). The Predicate delegate could be pretty expensive. IEnumerable<T> is forward only. Will exporting to List<T> and splitting up be faster? Excitement!

So there we are.We beat the built-in IndexOf.

Menno

Here's, for now, the final result:

using System;
using System.Collections.Generic;
using System.Threading;

namespace Tabeoka.Extensions
{
    public static class ExtensionMethods
    {
        public static int IndexOfParallel<T>(this List<T> source, T item)
        {
            int threadCount = GetOptimalThreadCount(source.Count);

            if (threadCount == 1)
                return source.IndexOf(item);

            SearchStatus status = new SearchStatus()
            {
                Found = false,
                FoundIndex = -1
            };

            // Looks like the ThreadPool always hangs onto at least 
            // # of cores threads, if left unset otherwise
            using (ManualResetEvent resetEvent = new ManualResetEvent(false))
            {
                int threadsFinished = 0;
                for (int i = 0; i < threadCount; i++)
                {
                    int fromIndex = (source.Count * i) / threadCount;
                    int toIndex = (source.Count * (i + 1)) / threadCount;

                    ThreadPool.QueueUserWorkItem(new WaitCallback(delegate(object t)
                    {
                        SearchListRange(source, item, fromIndex, toIndex, ref status);
                        if (Interlocked.Increment(ref threadsFinished) == threadCount)
                            resetEvent.Set();

                    }));
                }

                resetEvent.WaitOne();
            }

            return status.FoundIndex;
        }

        private static int GetOptimalThreadCount(int listCount)
        {
            // needs more sophisticated logic
            return Math.Min(listCount, Environment.ProcessorCount);
        }

        private static void SearchListRange<T>(List<T> source, T item, int fromIndex, int toIndex, ref SearchStatus status)
        {
            int i;

            for (i = fromIndex; i < toIndex; i += 4)
            {
                if (source[i].Equals(item))
                {
                    status.FoundIndex = i;
                    status.Found = true;
                    return;
                }

                if (source[i + 1].Equals(item))
                {
                    status.FoundIndex = i + 1;
                    status.Found = true;
                    return;
                }

                if (source[i + 2].Equals(item))
                {
                    status.FoundIndex = i + 2;
                    status.Found = true;
                    return;
                }

                if (source[i + 3].Equals(item))
                {
                    status.FoundIndex = i + 3;
                    status.Found = true;
                    return;
                }

                // if some other thread found it; quit searching!
                if (status.Found)
                    return;
            }

            // finishing up loop
            for (i = i - 3; i < toIndex; i++)
            {
                if (source[i].Equals(item))
                {
                    status.FoundIndex = i;
                    status.Found = true;
                    return;
                }
            }
        }
    }

    internal struct SearchStatus
    {
        public bool Found;
        public int FoundIndex;
    }
}

Complaining about Drupal

Kindly allow me to bitch some more about Drupal. I'm a bit frustrated.

• Array parameters: just about every function in Drupal takes an associative array:
  

  array(
        'field_name' => 'publication_datum',
        'cardinality' => 1,
        'type'        => 'datetime',
        'settings'    => array (
          'granularity' => array (
            'month' => 'month',
            'day' => 'day',
            'year' => 'year',
            'hour' => 0,
            'second' => 0,
            'minute' => 0,
          ),
        )
      )
  

  
  This effectively defeats any auto-complete you might have had in your php editor. Of course I see why they did it; it makes everything 'neat', and it certainly is flexible. Except now you're not only unclear about what you should pass into a parameter, you don't even know the parameters. Add-on modules could look for any key, and there's no way to find out, except by proper documentation (which is rare) or poring over the source code.
  Distinct advantage for the lazy coder: if you need to send an extra variable, you can just hitchhike along with any array that's headed in the right direction if you make sure the key isn't taken.
• Template naming: do you want to override the rendering of a particular element? You could hook into a theme process function and mess around with the render array or whatever, but the easier solution is actually to give a template file a cryptic name with lots of dashes. This file name functions somewhat as a css selector: the more specific ones override the more general ones. Thankfully there's a module to help you pick a template name: Theme developer offers a plethora of  possible template names for each part of your page. That's pretty handy, but also indicates I'm not the only one who has trouble keeping track.
• The hooks! Oh, the hooks! There are hundreds of them, and you can go ahead and create your own if the fancy strikes you. Hooks takes different numbers of parameters, of different types (and with or without &), and there is no way to find out without, once again, checking the documentation or the source code.
  I know, PHP does not have strong typing, but the small amount of meta data that would otherwise be available is eradicated by kinda sorta duck typing that's going on in Drupal. Again: I can see why they did it, and it is even clever. But it's still frustrating.

The basic problem it comes down to is this: a lack of discoverability of features. When I'm developing a Drupal module I will have dozens of tabs open in my browser, looking for a clue how to use a particular module. Open source is good; it allows me to figure out what's wrong, even if I did not write the code. But just providing the source code is no replacement for proper documentation and sample code. It's just lazy.

Menno

Drupal 7 and Asp.NET webforms

I've been manhandled into writing some modules for Drupal lately. Weeping and gnashing of teeth abounds.
Php is not exactly my favorite language to begin with, and Drupal is extensive and complex and wholly alien to me.
One thing has struck me though, working through the Drupal hooks madness: just how much some of it resembles Asp.NET webforms. A recurrent criticism of Asp.NET has been its confusing event pipeline, and how WebControl abstractions give less control over the generated html.
But lo and behold: Drupal 7's "render array":

$form['taal'] = array(
    '#type' => 'radios',
    '#options' => array(
      'nl' => 'Nederlands (NL)',
      'en' => 'English (EN)',
    ),
    '#required' => FALSE,
    );

This generates a list of <input type"radio">. Change type to 'select', and it generates a <select>. It's also possible to pass simple html string into a render array, like a LiteralControl. The point is to be able to change properties about generated content from other modules, without having to do lots of string parsing.
And then I read in my "Building Drupal Modules" book about how you can hook into the rendering process to change things. Here's the list of available functions:

• template_preprocess()
• template_preprocess_[NAME OF HOOK]()
• [NAME OF MODULE]_preprocess()
• [NAME OF MODULE]_preprocess_ [NAME OF HOOK]()
• [NAME OF THEME]_preprocess()
• [NAME OF THEME]_preprocess_ [NAME OF HOOK]()
• template_process()
• template_process_ [NAME OF HOOK]()
• ... you know, forget it

Total of twelve, for one possible hook, and there will be lots of hooks - add six for each. This is also an event pipeline of sorts. It's just that event handlers can only be added by AutoEventWireup, and it's for everything, not just page events.

Mind you, I'm not criticizing Drupal. Whenever you aim to provide a flexible web platform, you will come up with solutions that have to be either complex, or not enough.

Still, pain.

Menno

About those animated ajax page loads in MVC ...

A little while ago I showed a way to chain animation functions with callbacks in javascript. That was part of a web site in MVC the requirements of which were as follows:

• It needs to use awesometastic animation prettiful swooping panel dynamified load superlicious "Html5" etc.
• It needs to do well in search engines, and
• work reasonably well in IE7

The animations were part of the ajax-loading of the page content, but -of course- people needed to be able to link to any page directly as well.

So here's what I did:

• I created a ViewConfig class. This class contains the current configuration of the browser screen: what menus are shown, which background(s) are showing, etc etc:

      public class ViewConfig
      {
          [JsonConverter(typeof(StringEnumConverter))]
          public MenuDisplay Menu
          {
              get { return _showMenu; }
              set { _showMenu = value; }
          }
  
          [JsonConverter(typeof(StringEnumConverter))]
          public BackgroundsDisplay Backgrounds
          {
              get { return _backgrounds; }
              set { _backgrounds = value; }
          }
  
          public string Root
          {
              get { return _root; }
              set { _root = value; }
          }
  ...
  

• I created a BaseModel class that includes a ViewConfig property (and my menu data and other data shared among all models):

      public abstract class BaseModel
      {
          public List<Business.MenuItem> MainMenuItems
          {
              get { return _mainMenuItems; }
              set { _mainMenuItems = value; }
          }
  
          public ViewConfig ViewConfig
          {
              get { return _viewConfig; }
              set { _viewConfig = value; }
          }
  
          [JsonIgnore]
          public string ViewConfigJson
          {
              get { return  _viewConfigJson; }
              set { _viewConfigJson = value; }
          }
  ...
  

• Each controller takes a boolean json parameter that determines if the Model will be sent to the view, or rather simply returned as a JsonActionResult:

      public class InhoudController : BaseController
      {
          public ActionResult Index(string taal, string inhoudId, string json)
          {
              bool returnJson = "true".Equals(json);
              Models.InhoudModel model = new Tabeoka.Epsilon.Web.Models.InhoudModel();
  
  ...
  
              if (returnJson)
              {
                  var jsonResult = new JsonNetResult();
  
                  jsonResult.SerializerSettings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore;
                  jsonResult.SerializerSettings.MaxDepth = 1;
                  jsonResult.Data = model;
  
                  Response.Expires = 0;
                  Response.CacheControl = "no-cache";
  
                  return jsonResult;
              }
              else
              {
                  model.ViewConfigJson = model.ViewConfig.ToJson();
                  return View(model);
              }
  

• In the pages I then simply send an ajax request, get my model as json, and move, step by step, from my current ViewConfig to the new ViewConfig, using supertastic fantalicious animations.

There are just two obvious drawbacks to this approach:

1. As you can see, I need to send my ViewConfig twice; once as a string for the initial ViewConfig when a page is loaded through a View. The second when the entire model is serialized to json. It's ugly. I could serialize in my View to fix this.
2. Worse: I have a bunch of html rendering code in javascript. Ideally I would be able to use the same template in javascript and .Net. I'm not sure how and if that could work though.

Search engines and javascriptless (or javascript-poor) clients, I can just present a static version of the site. Other browsers will automatically have links on the page 'ajaxified'.

Conceivably in the future I can take this approach, and improve it by POSTing my current ViewConfig in my ajax request, build the exact delta between that and the requested page, and only fill up my Model with data to the extent my ajax code needs it.

Menno