I've moved!

For various reasons I couldn't maintain this blog so I've created a new one and migrated some of the posts (the ones that had actually had more than single digits numbers of hits!).

The new blog lives at http://edboiling.blogspot.co.uk/.

Points & Pointlessness: W3G lessons in GeoUsability

I've migrated this blog post to my new blog at http://edboiling.blogspot.co.uk/2010/10/points-pointlessness-w3g-lessons-in.html.

A Century-old Map Tiling Scheme

I've migrated this post to my new blog at http://edboiling.blogspot.co.uk/2010/08/a-century-old-map-tiling-scheme.html.

FuncUnit: Getting Funky with automated JavaScript testing

I’ve recently been attempting to make the testing of the software I’m working more agile, as the team I work with use Scrum as the development methodology. The problems could be summarised as:

• There is always a lag between development and testing: the software isn’t ready enough to test until late in the sprint, so sprints either overrun, or a “clean up” sprint is required to finish the testing, or fix bugs found very late.
• JavaScript functional testing (as opposed to unit testing) has traditionally been hard, especially with the kind of UI we typically build: a web GIS with lots of map interaction:pan, zoom, click to query, draw shapes.

The holy grail of my search has been to reduce the amount of manual regression testing we do, and increase the amount of unit testing. To make any impact this requires automation of complex UI tests in multiple browsers, linked to our Continuous Integration system (a hybrid of Cruise Control, nAnt, MSBuild and Team Foundation Server). Achieving this will allow the team to focus on more exploratory testing, and reduce sprint overruns. It should also highlight bugs at or shortly after check in time, so we don’t find them only after a huge round of manual regression tests.

This week I came across a new tool, FuncUnit. This is created by an outfit called Jupiter IT, who also produce JavaScriptMVC. I liked the look of it straight away because it ties together existing frameworks I am familiar with: jQuery, qUnit, Selenium – but offers automated functional tests of modern, whizzy JavaScript map UIs – i.e. not just unit tests of static components (no bad thing in themselves of course). You even write the tests in jQuery-like syntax – perfect for our development team who are fluent in jQuery and Dojo. Best of all it allows the specification of user interactions like drag – so we can automate scripts to pan our maps and draw zoom boxes or sketch shapes, something that has always eluded us.

I shan’t repeat too much detail on how FuncUnit works – you can go and read this for yourself. But an example might be useful to indicate how powerful and yet simple to implement this is. I had my first tests running within 15 minutes, performing an address look up using our Gazetteer client (running via a REST service against a remote web service), drilling down through a pick list of potential address matches, selecting one from the list, zooming to it on the map and kicking off a map search. The automation of this test is almost no further effort: create a batch file to call it from the command line instead of opening the test harness in a browser, and set up a scheduled task to run it after our nightly staging server deployment.

Installing FuncUnit

You can get the latest download here. Installation is just a matter of extracting the files. I put them in a web folder so I can set up a server that the whole team can access.

For my first test suite I used the template HTML file and copied the contents of the demo folder to a new folder, replacing the demo test page and JS file with my own.

Copying the demo scripts and changing to find the HTML elements in my application, I ended up with the following script:

module("wp",{
    //Set up: Open the map page
    setup : function(){
        S.open("/LocalViewWeb/Sites/LV/")
    }
})

//Main test. NB each test reloads the application being tested, so all my logic is in a single test.
test("Address Search", function(){
    //wait for the Gazetteer search box to become visible.
    S("#partialAddress").visible(function(){
        //Click to clear the hint. Type a street name.
        S("#partialAddress").click().type("Walton Way");
        //Click the "Search" button.
        S("#searchButton").click(function(){
           //Wait for the results container show. 
            S("#resultsContainer").visible(function(){ 
               //check for at least 1 result
                var items = S(".picklist_row").size();
                //use a qUnit assertion

                ok(items > 0, "pick list has items");
            }); 
        })
    });
    //Now look in the results list. 
    //use jQuery syntax to click the first in the list
    S(".picklist_row:first").click(function(){
        //When selected address is displayed, check the text
        S("#selectedAddress").visible(function(){
            var a = S("#selectedAddressText > a").text();
           //assert it's not an empty string
            ok(a!="", "Selected address: " + a);
        });
    });
    //Now choose a map search to run.
    S("#searchHeader").visible(function(){
        S("#searchHeader").click(function(){
            //this only appears when results have been found
            S(".result_row").visible(function(){
                var divs = S("#resultPanel > div").size();
                ok(divs>1, "Feed search results found");
            });
        });
    });
})

The results are displayed in the test page.

Running this is just a matter of opening the test harness in a browser. Automating it is just a matter of using EnvJs to run it from the command line, supplying the browser(s) you wish to test with. In about 30 minutes I’ve got to a state where as of tomorrow, I no longer need to regression test address searching, zooming to the address on a map or running a geographic search. In another 30 minutes I can get map zooming and panning covered off and have a stab at a map-based click-to-identify-features test. Hurrah – this saves me and the whole team days of time each sprint!

ShortcominGS

So far so good, but to get this to be as useful as possible I need a bit more:

• Alerts when the tests fail – run tests from Cruise Control server instead
• Ability to test multiple IE versions – create new VMs for IE6 and IE7

I’m really impressed at how far I’ve got so quickly though – it’s going to have an immediate impact. I’ll post more examples as I go along.

Watermarking your maps with the ESRI ArcGIS Server JavaScript API

If you work with digital maps on the web outside the USA then the chances are you use copyright-protected data that must meet certain conditions for display on the web or printed page. Often this can be a combination of copyright statement and watermarking.

There are a number of ways to implement watermarking on your maps when you are working with the ESRI ArcGIS Server JavaScript API. In this article I am going to discuss the options and describe the implementation I have chosen.

Watermarking involves displaying an image and/or some text over the map in a way that permits the data owner or map producer to be identified and prevents the map image being reused in a way that circumvents the data license. So, the trick is to get a transparent image to display over the map in a way that the map consumer cannot circumvent. This could be achieved in the following ways:

• Stamping dynamically created images with a watermark when they are written to disk, using a server technology such as Java or .Net.
• Adding a “watermark layer” to the map service, for example a large polygon that covers the full extent of the map, symbolised with a raster fill symbol.
• Drawing a watermark image over the map on the client, overlaying it using HTML and CSS.

Each approach has its potential drawbacks, and the “correct” approach will depend on the nature of your application and the type of map data you are working with.

Which option works best?

With the JavaScript API, we typically want base mapping to come from a tiled, cached source, as it is static data, and user expectation is of Google and Bing-like map performance. In this scenario we can’t stamp a watermark when each map is requested (Option 1), but we could when each map image is written to the cache folder. The folder structure may be a bit too complex to make this practical. So, we could add a watermark layer (Option 2) above at the point of map creation, when the cache is generated. This is robust as the user cannot turn it off, but there can be display issues – what if your application displays other data over the top? The watermark will not be the top level layer. What if your maps are overlaid with other watermarked maps? The overlay of several repeated transparent logos could get messy.

If we’re working with dynamic map services only, then there are plenty of opportunities to include a watermark, but we need a server implementation (such as a Windows Service that watches an output folder) to stamp the image when it’s written to disk – which won’t work if we’re streaming the image as MIME data – or, we have to include the watermark as a layer, which the user could turn off with a simple JavaScript call.

Alternatively we could use a proxy to intercept all map requests and stamp a watermark before passing to the client. This could be done at the map server level using a technique such as an HTTP Module for IIS –intercepting the request and adding a watermark to the image when it is streamed in the response. But, in my case, I am building a web application that consumes data from many sources, so I need to have an option to add a watermark in the browser for those situations where the map service doesn’t contain a watermark layer, or where multiple map sources are combined, or where I have no access to the map server to add watermarking to the HTTP pipeline.

So, it looks like option 3 may have a role to play where the base maps are coming from a tiled, cached source, or where we don’t have the access or maybe technical resources to develop a server-side watermarking application, or where a map watermark layer could be easily turned off by someone with basic JavaScript skills.

Implementing simple client-based watermarking

Given that we now think a client-based solution is worth considering, how might we go about it?

With an ArcIMS application we’d use the Acetate layer, and with a Web ADF application we could add a Graphics Layer – does this work for the JavaScript API? Well, kind of. But JS API Graphics draw as Vector graphics in the browser – Web ADF ones render as transparent PNGs. So we’d need to bubble any mouse events that the graphics captured to ensure that map navigation through mouse interaction still works, and that other map graphics could still be drawn over the watermark. I have to say I don’t like the feel of this approach.

So, that leaves us with the need to tile or centre an image over the map (the image could contain text and/or a logo). What is the best way to achieve this?

Well, one relatively easy way is to add the watermark as if it were a map service “layer” (the JS API treats services as Layers within the map – each map Layer also has Layers of its own, potentially. I had a read of Sathya Prasad’s excellent articles on extending the DynamicMapServiceLayer and TiledMapServiceLayer objects, and thought that if the methods that returned a URL just loaded a watermark, then it’d be relatively easy to add watermarks from a static image as if they were coming from a map layer. And it was. Here are two examples of watermark layers; implemented as a Dynamic layer (that gets one image that fits the map size) and a Tiled layer (that tiles the watermark in rows and columns).

Dynamic watermark layer:

dojo.provide("myScripts.WatermarkLayer");

dojo.declare("myScripts.WatermarkLayer", esri.layers.DynamicMapServiceLayer, {

startExtent: new esri.geometry.Extent({ xmin: 0, ymin: 0, xmax: 700000, ymax: 1300000, spatialReference: { wkid: 27700} }),

//construct the layer

constructor: function(args) {

this.initialExtent = this.fullExtent = this.startExtent;

this.spatialReference = this.initialExtent.spatialReference;

dojo.mixin(this, args);

//set layer loaded property and fire onLoad event

this.loaded = true;

this.onLoad(this);

},

getImageUrl: function(extent, width, height, callback) {

try {

callback(this.url);

}

catch (ex) {

console.debug("Error getting Watermark image. " + ex.message);

}

},

_errorCallback: function(err) {

console.error(err.message);

}

});

Tiled Watermark Layer:

dojo.provide("myScripts.WatermarkTiledLayer");

dojo.declare("myScripts.WatermarkTiledLayer", esri.layers.TiledMapServiceLayer, {

watermarkUrl: "",

//construct the layer

constructor: function(args) {

dojo.mixin(this, args);

//set layer loaded property and fire onLoad event

this.loaded = true;

this.onLoad(this);

},

getTileUrl: function(level, row, col) {

return this.watermarkUrl;

}

});

Usage

This sample code checks to see if the map’s base layer is tiled, and if so, creates a tiled watermark. If not, it creates a dynamic one.

var wm;

var base = map.getLayer(map.layerIds[0]);

if (base.tileInfo == null)

{

wm = new myScripts.WatermarkLayer({ url: “http://www.mydomain.com/somefolder/images/watermark.png” });

}

else {

wm = new myScripts.WatermarkTiledLayer({

watermarkUrl: “http://www.mydomain.com/somefolder /images/watermark.png”,

tileInfo: base.tileInfo,

initialExtent: base.initialExtent,

fullExtent: base.fullExtent,

spatialReference: base.spatialReference

});

}

wm.setOpacity(0.15);

map.addLayer(wm);

What’s going on here then?

Basically, the Dynamic watermark layer is requesting an image from a static URL each time the map is redrawn. Browser caching means this image is only downloaded once, so it performs well. But unless you know what size the map image is, the watermark may get stretched or squashed.

The Tiled layer implementation works a bit better. When the layer object is created, it copies the tileInfo property from the base layer (read: base map service) of the map control. So the watermarkimage assumes the dimensions of each tile image, and is neatly tiled across the screen in rows. You will often know or be able to predict the tile image sizes: 256 x 256 is pretty common, so you can get the watermarks looking a lot better than with the Dynamic layer approach.

When the map is loaded, we just add a layer on top of any others in the stack of services we’re using, and hey presto – we get watermarking over the top of them all.

Drawbacks

This approach is clearly not without its limitations. Here are the main ones I can think of:

• Dynamic images will stretch the watermark and distort it, unless your map application has a fixed viewport size.
• Tiled images will only work if your base map layer is also tiled, otherwise you’ll need to invent a tiling scheme for the map, and it will behave as if it’s using a cached, tiled map source.
• Any JavaScript developer worth their salt will be able to traverse your map object’s layers collection, find the watermark layer, and switch it off. So it’s your decision whether this approach is robust enough for the application you’re developing.

Performance Testing with Visual Studio 2008

I'm back at the controls of Visual Studio Team Test edition (2008 this time, woo hoo) loading and running performance tests against some software I'm developing. This time round I am doing basic analysis to see where potential bottlenecks lie, and to spot any memory issues whilst there is still plenty of time to fix these.

However, I had some difficulty getting my tests recorded - turns out there are a few things you need to bear in mind if your test client is Windows 7, Server 2008 or Vista, mainly around the enhanced security configuration (Server 2008) and IE security. Basically, if during your test you switch from one internet "zone" to another - for example, from "about:blank" to an actual website - then IE will kill the original iexplore process and launch a new one. Very secure, but it kills the ability to record web tests until you sort it out.

I found a great blog site run by Michael Taute that lists pretty much all the reasons why this behaviour can happen, and what to do about it. In my case I switched off the enhanced security configuration for administrators, and added "about:blank" to the same internet "zone" as the site I was testing.

See Michael's blog page for the full list of issues and fixes.

Oh, and if, like me, you're an idiot, you'll find your load test doesn't work if you don't create the load test database before you start. There are a few steps to remember before you start testing, as detailed by Travis Lingenfelder.

RAW DATA NOW! My first LocalGovCamp

I've migrated this post to my new blog at http://edboiling.blogspot.co.uk/2009/10/raw-data-now-my-first-localgovcamp.html.