Code Highlighting

Friday, December 28, 2012

SELECTing the comedy horror genre

Whenever we get a new intern, I like to poke and prod him*, to see what he's made of (bones and gooey bits, usually). To test SQL proficiency, I use the following problem. I like it because it's something that comes up occasionally in projects, and is surprisingly complex, though it seems like it shouldn't be:

Suppose you've got a movie database. You've got a table with movies T_MOVIES:

CREATE TABLE [dbo].[T_MOVIES](
 [movie_key] [int] NOT NULL,
 [movie_name] [nvarchar](500) NOT NULL,
 CONSTRAINT [PK_T_MOVIES] PRIMARY KEY CLUSTERED 
(
 [movie_key] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]

and there's also a table for movie genres T_GENRES:

CREATE TABLE [dbo].[T_GENRES](
 [genre_key] [int] NOT NULL,
 [genre_name] [nvarchar](500) NOT NULL,
 CONSTRAINT [PK_T_GENRES] PRIMARY KEY CLUSTERED 
(
 [genre_key] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]

To link both tables there is a third table T_MOVIE_GENRES:

CREATE TABLE [dbo].[T_MOVIE_GENRES](
 [movie_key] [int] NOT NULL,
 [genre_key] [int] NOT NULL,
 CONSTRAINT [PK_T_MOVIE_GENRES] PRIMARY KEY CLUSTERED 
(
 [movie_key] ASC,
 [genre_key] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]

Assume the proper foreign key constraints have been applied, and you have a pretty sensible lay-out. To select all movies within the horror genre you only have to join T_MOVIES to T_MOVIE_GENRES and filter down your genre key in the WHERE clause to whatever the horror genre is, suppose it's 5:

SELECT
 T_MOVIES.movie_key,
 T_MOVIES.movie_name
FROM
 T_MOVIES INNER JOIN
 T_MOVIE_GENRES ON
 T_MOVIES.movie_key = T_MOVIE_GENRES.movie_key
WHERE
 T_MOVIE_GENRES.genre_key = 5

The question now is: What if I want to filter by two genres? How do I find movies that are not only horror, but also comedy? What query will yield "Shaun of the Dead"?
More generally, how do we find those records in a table that have more than one matching record in another table, where those matching records have a field set to a set of specific values?

Obviously WHERE T_MOVIE_GENRES.genre_key IN (1,5) is going to yield movies that belong to either genre.
At this point usually the intern scratches his chin, gives it a bit of thought, and comes up with this:

SELECT
 T_MOVIES.movie_key,
 T_MOVIES.movie_name
FROM
 T_MOVIES INNER JOIN
 (
 SELECT  T_MOVIE_GENRES.movie_key
 FROM T_MOVIE_GENRES
 WHERE T_MOVIE_GENRES.genre_key = 5
 ) AS Q_HORROR
 ON T_MOVIES.movie_key = Q_HORROR.movie_key
  INNER JOIN
 (
 SELECT  T_MOVIE_GENRES.movie_key
 FROM T_MOVIE_GENRES
 WHERE T_MOVIE_GENRES.genre_key = 1
 ) AS Q_COMEDY
 ON T_MOVIES.movie_key = Q_COMEDY.movie_key

(Or some other solution involving a subquery)
Okay, fine, you found "Shaun of the Dead". But what if I'm actually in the mood for Zombieland, a comedy / horror / road movie? In fact, I want a stored procedure that can take an arbitrary number of genres, and filter by it. You can go the dynamic sql route, and build a string with an arbitrary number of subqueries. It'd be ugly and hard to maintain, but it would work.
The following shows what I think is the best solution though:

SELECT
 T_MOVIES.movie_key,
 T_MOVIES.movie_name
FROM
 T_MOVIES INNER JOIN
 T_MOVIE_GENRES
 ON T_MOVIES.movie_key = T_MOVIE_GENRES.movie_key
WHERE
 T_MOVIE_GENRES.genre_key in (1,5)
GROUP BY
 T_MOVIES.movie_key,
 T_MOVIES.movie_name
HAVING
 COUNT(*) = 2

To refactor this into forementioned stored procedure you need to add a table-typed variable, and use a little bit of dynamic sql - because SQL does not have anything like arrays. Something like this:

CREATE PROCEDURE GetMoviesByGenres
 @genre_keys nvarchar(2000)
AS
BEGIN
 DECLARE @genre_keys_table table(genre_key int)
 DECLARE @genre_count int
 
 INSERT INTO @genre_keys_table
   EXEC(N'SELECT DISTINCT genre_key FROM T_GENRES WHERE genre_key IN (' + @genre_keys + N')')
 
 SELECT @genre_count = COUNT(*) FROM @genre_keys_table
 
 SELECT
  T_MOVIES.movie_key,
  T_MOVIES.movie_name
 FROM
  T_MOVIES INNER JOIN
  T_MOVIE_GENRES
  ON T_MOVIES.movie_key = T_MOVIE_GENRES.movie_key
 WHERE
  T_MOVIE_GENRES.genre_key IN (
   SELECT genre_key FROM @genre_keys_table
  )
 GROUP BY
  T_MOVIES.movie_key,
  T_MOVIES.movie_name
 HAVING
  COUNT(*) = @genre_count
END

If it wasn't clear from the code; @genre_keys takes a comma-delimited list of keys. That leaves an obvious injection vulnerability, so I would advise to have your calling code take an array of int, and not a string.
So that pretty much concludes that. If any of you ever end up as an intern at Tabeoka, you can now impress me with your mad SQL skillz.

Menno




* "him" because we've only ever gotten male interns. I'm prety sure I could get in trouble for poking and prodding a girl intern anyway. Both with the law, and my wife.

Monday, December 17, 2012

Html and Svg: handling events to and fro

I've been working on a website where I embed a couple of svg's in an html page. Actually, I embed the same svg in three places on one page. The svg in question has some script-driven animation, so I needed to use an object tag, rather than a simple img.
First challenge was this: I needed to trigger the svg animation for all svg's one after the other, which meant I had to call into the svg's script:

        function bounceBall(ball) {
            var svgView = getSvgView(ball);
            
            if(svgView)
                svgView.startBounce();
        }

        function getSvgView(ball) {
            var svgDoc;
            try {
                if (ball.getSVGDocument)
                    svgDoc = ball.getSVGDocument();
                else if (ball.contentDocument)
                    svgDoc = ball.contentDocument;

                if (svgDoc) {
                    return svgDoc.defaultView;
                }
            } catch (e) { }
            return null;
        }

When I initially call this function, the SVG dom might not be loaded yet. That's why I include the try/catch, and return null if an error occurs. Elsewhere in the code I set a setTimeout to retry 200 milliseconds later. The onload event is not quite reliable enough.

Next up was the click handler. If you simply add an onclick to the object tag, nothing happens when you click the svg. That makes perfect sense: the onclick is registered and handled within the svg, and never makes it to the html dom. I needed to have a click on the image open a little div in html though. From svg I could call the html javascript functions using top.someFunctionName(). I had the same svg image three times though, and it needed to do something different each time. Here's what I came up with:

Html:

<object type="image/svg+xml" data="/Content/Images/ball.svg" class="ball" style="left: 130px; top: 160px;" onclick="showPopup('homepopup2');"></object>

"But wait!" you say, "You just told me that doesn't work!". And it doesn't, but it would be pretty convenient if it did:

Html dom javascript:

        function setClickEvent(ball) {
            if (ball.onclick) {
                var svgwin = getSvgView(ball);
                if (svgwin) {
                    svgwin.eventHandler = ball.onclick;
                } else {
                    // if the svg view is not available,
                    // try again in 200 ms.
                    window.setTimeout(function() {
                        setClickEvent(ball);
                    }, 200);
                }
            }
        }

There you go. On load I simply funnel the onclick handler into the svg dom. The svg implementation is trivial:

    var eventHandler = null;

    function handleClick(){
      if(eventHandler)
        eventHandler();
    }


<circle cx="15" cy="15" r="5" id="ball" onclick="handleClick();" />

If you need this sort of thing more often - or for more events - you could work out a neat wrapper with  registerEventHandler(eventName, eventHandler) and triggerEvent(eventName) methods to reuse (and a html-side script that automatically hooks it up). I don't currently foresee a need for it myself though.

Menno

Tuesday, December 4, 2012

Anything but restful

Last week I was asked to integrate some "webservices" into a .NET application. I say "webservices", but that word can mean anything from SOAP to custom format XML to some icky CSV that uses the asterisk for a delimiter.
I was lucky though: the url made obvious that this webservice was WCF-based, some methods returning json, and equivalent methods returning xml. All request parameters are to be sent as json through the querystring.
I'm an optimistic fool, so I simply use the "Add Service Reference" to add a link to the service, and all seems well. Until I try to actually use the service: my application throws up the following:

Could not find default endpoint element that references contract 'IServiceInterface' in the ServiceModel client configuration section. This might be because no configuration file was found for your application, or because no endpoint element matching this contract could be found in the client element. 

Sure enough, when I check in my configuration file, the following has been helpfully added:

<configuration></configuration>

Well thank you very much, Visual Studio. You're a great help.
Google turns out to be more of a help, and points me to the following article telling me: what you're trying to do does not work and will not work. Because they're so RESTful. Instead I should be using the WebChannelFactory class to generate a channel and use that to call the webservice.
No problem.
I  re-use the proxy classes generated in my non-working service reference (but delete the wrapper classes), and point the WebChannelFactory to the generated service interface. Does that work now?
No of course it doesn't work: "405 http method POST is not supported by this url".
Sure enough, Fiddler shows that the service is being called using a POST of the request paremeter serialized as xml. "Stupid boy, " Google tells me again, "you should be adding the WebGet attribute to your method, and if you want json, you need to add a behavior to your endpoint that selects the JsonQueryStringConverter". "And while you're on it, " Google continues, "don't forget to define a UriTemplate for your WebGet attribute".
Done, done and done.
Now does it work? No of course it bloody well doesn't work. No more errors, but no deserialized data either. The json serialized data appears to be wrapped in a single key 'd' for all requests. The class structure doesn't match up to the json hierarchy, so nothing gets deserialized. So I wrap all my return types in a small generic class:

  public class JsonWrapper<T>
    private T _d;
    public T d {
      get { return _d; }
      set { _d = value; }
    }
  }

Finally I start seeing data. WCF and the one side, WCF on the other side and nothing works by itself. Some of the articles I was reading had the gall to tell me this was because WCF has great extensibility. Is "extensible" newspeak for "does nothing useful out of the box"?
Sure, it is very extensible, but in the time it took me to figure out how to get it to call a simple json webservice, I could just as easily have used the WebClient class to call the url, and run the return data  through  NewtonSoft's Json.NET. In fact, I could have done it three times.
If the service reference is able to properly generate proxy classes, why can't the service reference also send the necessary metadata to configure all that stuff I had to do manually? Where was the added value in all that WCF stuff? Is this a useful abstraction, or needless obfuscation?
Now that I've got a bit more of a handle on how it works, I'll give it another chance. But I'm not sold.

Menno

Monday, November 26, 2012

SVG is such a tease

A couple of days ago I was asked to look into animations that would work on iPad. You could use CSS3 tranformations, but I was excited to see what I could do in SVG instead. Why? Because why not!
Off I went, and not too much longer I had this: http://www.tabeoka.be/downloads/svg/blokskes.svg . Cute. Works smoothly on all browsers that matter. Is a little jittery on Opera.
Turns out SVG is a lot of things I've wanted HTML to be in the past. Have you ever created a DIV just to make a rectangle? SVG is for you. It's HTML for making pretty pictures. It supports css and most of the javascript DOM methods you're used to. Pick it up and go.
Back to my image though: I could slap an image on there, and pretend the website is being swallowed by a maelstrom. Right?
Well, here's a unicorn with an unfortunate case of being torn into rectangles:  http://www.tabeoka.be/downloads/svg/blokskes_fill.svg . The most unfortunate part though, is that the unicorn is being torn up very slowly. Unfortunate for the unicorn, certainly, but more unfortunate for my ambition to animate stuff using SVG. What's up with that?

I must be doing something wrong. I can't believe my computer can render this, but not chop up a unicorn (where are its priorities?). Internet Explorer 9 is supposed to have hardware accelerated svg rendering. If this is the result, it must have needed that acceleration real bad. And I can believe Microsoft messed up SVG performance in IE, but every damn browser is slow!
In the past I did some animation where I would suspend rendering while updating the object hierarchy, and then restart rendering. But this article suggests browsers are not stupid; that's too bad. I think I am to conclude that fully SVG-driven animations are not quite ready from prime time. But when the performance catches up, the object model will be ready. I like SVG, but it's such a tease.

Menno

Monday, November 19, 2012

Drupal: open-source CMS

Drupal has been pretty popular with the various Belgian governments. Any new websites created for a government agency are supposed to use an open-source CMS. Drupal's popularity (and its Belgian roots?) have made it the perfect candidate. Heck, even the king uses it. A new initiative is Drupal-as-a-service: pick your features and click "Create website". Off you go!
With the government's zeal to convert everything to open-source the buzz has been increasing. Universities request websites built on Drupal, marketing people want their new website in "Druple".

So what's up with this love for open source? I will now wield my razor-sharp intelligence to shamelessly make up what I think is most likely:

  • Avoiding vendor lock-in
    Every few years, the government is required to field a project to get price quotes from a number of parties, and then pick the best price quote.
    If a project has been written using a proprietary framework/platform/language/etc the best candidate will likely be the candidate who initially set up the project. Any other candidate would need to either swallow the costs of a steep learning curve, or be more expensive.
    Not so for open source. Anyone who knows Drupal/Joomla/Wordpress/But mostly Drupal can pick up a website and go! Money savings galore!
  • Open data
    Whether you have access to the source of your CMS or not, the data on your site is yours. If you switch to a different system, you want to keep your data. If a proprietary system perishes, your data could perish with it. In an open source system, you have the ultimate data spec: the source code.
  • Security updates
    Whenever a security leak is found, noble developers will flock to it, and release a security patch in no time at all. Government sites need to be secure, right?
  • Oh! The modules!
    If you have a Drupal-based website and you want to add a online shop, you only need to install Drupal commerce. Works instantly! Hardly any configuration required! Money savings galore!
    More to the government point: integrate with the Belgian electronic ID card (currently has some security issues).
  • Open source means no license means cheap! Right?
    No it doesn't. And I doubt the government is stupid enough to believe this.
Some of those are pretty solid reasons. I want the government to spend my tax money wisely (frugally even), and store my personal data on secure servers! Who wouldn't?

But here's the thing: when we make a Drupal website, the customer comes back and tells us

"We don't like the way this editing form works. Can you make it do x and y instead?"

And we can. The customer's request makes sense: core Drupal is pretty spartan, and a lot of modules are a tad confusing. We just code up a nice custom module that does exactly what the customer wants. 
But here's the catch:
Those custom modules can be neither open-source, nor secure (nor very well-written for that matter).
That includes some modules that I've written. It's vendor lock-in all over again. 
We inherited a Drupal project from a competitor, and the last few weeks I've been poring over custom modules that implement some form of url rewriting using taxonomy terms. It's a very neatly commented and indented mess, and has a fair number of bugs - I'm not sure I can do better. I'm familiar enough with Drupal, but my employer has had to swallow these costs anyway. You can't pick up a major website and run with it. Open source doesn't matter that much here.

As far as the open data is concerned, I think that's a valid point. I have written a proprietary CMS in .NET, and I encrypt all website data using AES-256 before storing it in the database using an 'optimized' version of Base64. I double-dare customers to switch to a competitor (I kid, of course). But I could do such a thing, call it 'Enterprise-level security', and sell it to the banking industry.
Using open source, getting my data is not necessarily easier, but I know it can be done.

Lastly, about the security updates. The sysadmin is usually the weak link here. Nobody will likely ever find a security hole in my proprietary CMS, because nobody cares. There will, 100% sure, be security updates for Drupal. For all eternity: either you patch or you're vulnerable. My CMS may actually be safer than Drupal. Not by virtue of its technical qualities, but just because it's a tiny fish in a pretty small pond.
After all: wasn't Apple repeatedly the first to be hacked at pwn2own ? 

Menno

Monday, October 22, 2012

SSMS: OutOfMemoryException executing a large SQL batch

So you're executing some beast of an Sql batch using the Management Studio, for instance the result of "Generate Scripts" on another database. Then all of a sudden you get a OutOfMemoryException. Even when just keeping the entire script in memory, poor SSMS is hanging on by its fingernails. How could you expect it to also execute that file, and give you the results?

Thankfully there is a command-line Sql server client Microsoft thoughtfully provides together with Sql server. If you're not able to import a large sql file using SSMS, navigate to the Binn folder of your Sql server folder (C:\Program Files\Microsoft SQL Server\100\Tools\Binn here) in Command Prompt, and type the following:

osql -S databaseserver -U username -P password -d DemoDatabase -i c:\demo.sql

That executes the file C:\demo.sql on databaseserver in the context of DemoDatabase using the login data provided. This will scroll a bunch of query results in your command prompt window. If you'd rather examine these results in detail later, the -o parameter writes this info to an output file:

osql -S databaseserver -U username -P password -d DemoDatabase -i c:\demo.sql -o c:\output.txt

That should work, even when SSMS chokes on the sheer size of your query file.

Monday, October 15, 2012

text-transform: uppercase subtleties

So last week I received an interesting question from a customer. He complained that in Chrome on Mac, the category 'Soßen und Dips' ("Sauces and dips") on the german (obviously) version was rendered as 'SOSSEN UND DIPS' (wrong), rather than 'SOßEN UND DIPS' (right). Notice that ß and SS are semantically identical - a latin1 collated database will consider those strings to be identical.
On Internet Explorer, the ß was displayed fine.
It didn't take long to figure out that the menu item had text-transform: uppercase applied, and that this caused the transform of ß to SS. What's more, Safari and Firefox also displayed SS instead of ß. My Google search led me to page https://bugzilla.mozilla.org/show_bug.cgi?id=354451 . This page indicated that transforming ß to SS was deliberate, and not a bug at all. Der Spiegel suggests that ß should, in capitals, always become SS.
So now I only need to convince the customer that their browser can spell their language better than they can.

Menno




As an interesting aside: a basic test case to check browser behavior sees Internet Explorer 9 also rendering SS, not ß. Changing the document mode down to IE7 shows ß again. The actual online page is in IE9 mode, and does not have a X-UA-Compatibility meta tag. The question then becomes, why does it show ß? No idea yet.