Boiler Wadding

400 million maps in a month? Well, why not?

2010-12-22T10:16:00.001Z

I am surprised to hear that so much is being made of the claim by Tim O’Reilly, whilst on a visit to ESRI, that ArcGIS.com handled 400 millions maps in October.

Particularly odd are the comparisons too Flickr’s 200 million user uploads a month. Not just because two of the blog posts doing so, from @anderitr and @ianhaynes, contain identical text!

A Flickr user is not doing anything remotely like an ArcGIS.com user. A map request from ArcGIS.com is not anything like a user upload to Flickr. ArcGIS.com users are not uploading 400 million maps a month, they are requesting them.

Unique visitors to ArcGIS.com consist in a large part of professional desktop and web GIS users, who will spend most of their working day looking at maps, some of which will have a basemap or other overlay requested from ArcGIS.com.

In that context, assume the 30,000 or so unique visitors are GIS professionals, working on average 22 days a month, for 7 hours a day.

400 million maps divided by 33,000 visitors is just over 12,000 each. But if each visitor visits every working day for 22 days a month, that’s only 550 maps a day each. In a 7 hour working day, that’s requesting 1.3 maps from ArcGIS.com per minute on average.

In my performance testing I work with a long-held assumption that a GIS user will average 6 maps per minute, not 1.3. So this doesn’t seem hugely unreasonable to me.

Points & Pointlessness: W3G lessons in GeoUsability

2010-10-01T11:18:00.002+01:00

This year’s inaugural AGI W3G unconference – “The Three Ws of Geography” was a breath of fresh air on the Geo-related conference circuit, which I am afraid to say I have happily ignored for some years now. There are only so many vendor pitches and PR presentations a man can take before deciding it’d be time better spent to go into the office and do some work instead.

The unconference format promises to cut to the chase. People present, not organisations, and the format is more social, open and democratic. This of course makes the content difficult to predict, but I found that despite the apparent potential for a chaotic day of unrelated sessions, in fact W3G had some really strong themes running through it. Most of these touch on my “pet subjects”: pushpin overkill, poor usability, maps for the sake of it.

1. Your data is rubbish

This came up in Steven Feldman, Charles Kennelly and Peter Batty’s talks, albeit in subtly different ways:

Your data doesn’t always need to be mapped: think first: does it say anything useful?
Your data is not sufficiently self-describing. Don’t waste time and money thinking you can manage metadata as a separate task – tell your users more about your data’s provenance IN the dataset.
Publishing meaningless data - even in a pretty iPhone or iPad app - is a serious usability issue. Do more user testing first!

2. The Usability of your geo-app is poor

Consumer geo applications beat old-school GIS ones in terms of usability, because of the mantra to simplify, and then simplify again. Peter Batty quoted from Krug and make Jacob Nielsen’s 20-year-old point about usability testing: get anyone to do it, tell them nothing, and shut up and watch. 5 to 7 users will find 80% of your defects this way, as well as telling you way more about usability than you could ever guess as the designer or developer of an application.

Why is this? Why do GIS applications come across so poorly? I have a theory here – it’s one of my pet subjects after all. Because GIS applications are usually sold to GIS people, they have one template in mind of what the UI should be: ArcVIew 3 effectively.

Fill the screen with icons on tool bars and GIS people will want to buy it. Make it more simple or appear to “do less” and you may well sell fewer. GIS teams may then want the same thing as a web application to show off their lovely 250 datasets. But the people who use the application will suffer: the “toolbox” approach doesn’t tell you where to start or what to do: it forces the average user to have to think too much, and on the web, attention spans are too short to guarantee any sort of success. If you’re used to using desktop GIS, or even the legacy web GIS applications, then you won’t spot this. GIS apps need to focus far more on focussed, simpler workflows, not lists of data and tools, if they are to be used successfully by large amounts of ordinary users. In essence, they should not bear any visual resemblance to a GIS app, I’d say.

3. Make the complex simple

I especially liked Richard Treves’ presentation, coming from a design and visualisation aspect, but touching on the age-old cartographic problem of how to present millions of points in a clear and readable way, without distorting the pattern or introducing bias.

For my money I’d offer the user a choice of settings wherever a design decision has been made that could affect the interpretation. If you use a heatmap at small scales, say something about how grid size affects the pattern, and even allow the user to change it. IF you use point clusters, explain that you cannot tell what the distribution of the points is – allow teh raw data to be switched on, colour coded to show which cluster they represent. But don’t avoid these techniques just because of potential ambiguity, the alternative is the full extent map with one million markers that says absolutely nothing.

The Big Deal about W3G

So what made it such a different experience? I think for me, the fact that organisations such as Google, Nokia, and Yahoo were present and actively contributing put what can otherwise be a parochial, academic, insular sort of event into a much wider context. And made the point, repeatedly, that location is a feature – not the be all and end all of any application. Even a hardcode GIS one! Well, so say I.

Many of the points that were made – about geographic visualisation and analysis –were made by people who don’t come from the GIS tradition, but instead have expertise in usability, design, non-geographic visualisation: as a GIS professional I can benefit from extremely relevant experience from outside my field, that a traditional GIS conference cannot offer.

Compare this to the GeoCommunity sessions: what’s the difference, at first glance at least? Well, to me the major contrast was the focus on consumer applications of geotechnology, and the huge emphasis this inevitably places on usability (meaning many things, but including data quality, a tested application, simple rendering of complex data – making the user have to think less). The “traditional” GIS community is usually far more preoccupied with a business-to-business or academic focus, where standards and data interoperability and dry discussion of formats and functionality and features will necessarily dominate. The danger for me is that the the huge convergence of the two approaches, and the lessons each can teach to the other, are in danger of being ignored by “traditional” Geo community events. This could perhaps risk missing an enormous opportunity to reach a new consumer audience who now want to do a bit more with their location than spray it around Twitter in Foursquare spam-tweets, and may well want to take advantage of a bit of old-skool GIS analyis, if only it can be presented to them without them realising that that is what it is.

A Century-old Map Tiling Scheme

2010-08-23T22:34:00.001+01:00

I love it when my interests collide in unexpected ways. I spend a lot of my professional life thinking about web map tiling schemes for ESRI ArcGIS Server web mapping applications: tile size, map projection, levels of detail, image format and so on. Outside of work I indulge my childish fascination with big, loud, fast moving machines. On a recent holiday to North Yorkshire I couldn’t resist ~~dragging~~ taking the family to the spectacular North Yorkshire Moors Railway where I came upon a really rather beautiful map.

The map in question is a reproduction of an early twentieth century original,one of twenty or so produced by the North Eastern railway to show off its impressive railway network – now long since reduced to cycleways and overgrown embankments.

By itself it’s a thing of beauty, but as a geography nerd and lover of maps I rather like the unexpected parallels with modern web mapping techniques: tile size (8” square), levels of detail (special tiles for greater detail) and image format (ceramic tiles!) were all considerations for the railway cartographers of 1903 just as they are for the GIS developer in 2010. The final product is just a little more static and permanent than its modern equivalent. Thanks to a little photoshoppery by the ever-obliging Matt Toon at Google we can see how the North Eastern Railway might have approached the task today:

I was also impressed by the implicit supreme confidence in the permanence of not just the map but the railways it showed: ironic considering the surviving originals (examples remain at Whitby and Scarborough) have long outlived the lines they depict. If only I could say the same of the maps I work on whose life expectancy is usually 3 years at best, and of which no trace remains when they get decommissioned.

FuncUnit: Getting Funky with automated JavaScript testing

2010-08-13T16:58:00.001+01:00

I’ve recently been attempting to make the testing of the software I’m working more agile, as the team I work with use Scrum as the development methodology. The problems could be summarised as:

There is always a lag between development and testing: the software isn’t ready enough to test until late in the sprint, so sprints either overrun, or a “clean up” sprint is required to finish the testing, or fix bugs found very late.
JavaScript functional testing (as opposed to unit testing) has traditionally been hard, especially with the kind of UI we typically build: a web GIS with lots of map interaction:pan, zoom, click to query, draw shapes.

The holy grail of my search has been to reduce the amount of manual regression testing we do, and increase the amount of unit testing. To make any impact this requires automation of complex UI tests in multiple browsers, linked to our Continuous Integration system (a hybrid of Cruise Control, nAnt, MSBuild and Team Foundation Server). Achieving this will allow the team to focus on more exploratory testing, and reduce sprint overruns. It should also highlight bugs at or shortly after check in time, so we don’t find them only after a huge round of manual regression tests.

This week I came across a new tool, FuncUnit. This is created by an outfit called Jupiter IT, who also produce JavaScriptMVC. I liked the look of it straight away because it ties together existing frameworks I am familiar with: jQuery, qUnit, Selenium – but offers automated functional tests of modern, whizzy JavaScript map UIs – i.e. not just unit tests of static components (no bad thing in themselves of course). You even write the tests in jQuery-like syntax – perfect for our development team who are fluent in jQuery and Dojo. Best of all it allows the specification of user interactions like drag – so we can automate scripts to pan our maps and draw zoom boxes or sketch shapes, something that has always eluded us.

I shan’t repeat too much detail on how FuncUnit works – you can go and read this for yourself. But an example might be useful to indicate how powerful and yet simple to implement this is. I had my first tests running within 15 minutes, performing an address look up using our Gazetteer client (running via a REST service against a remote web service), drilling down through a pick list of potential address matches, selecting one from the list, zooming to it on the map and kicking off a map search. The automation of this test is almost no further effort: create a batch file to call it from the command line instead of opening the test harness in a browser, and set up a scheduled task to run it after our nightly staging server deployment.

Installing FuncUnit

You can get the latest download here. Installation is just a matter of extracting the files. I put them in a web folder so I can set up a server that the whole team can access.

For my first test suite I used the template HTML file and copied the contents of the demo folder to a new folder, replacing the demo test page and JS file with my own.

Copying the demo scripts and changing to find the HTML elements in my application, I ended up with the following script:

module("wp",{
    //Set up: Open the map page
    setup : function(){
        S.open("/LocalViewWeb/Sites/LV/")
    }
})

//Main test. NB each test reloads the application being tested, so all my logic is in a single test.
test("Address Search", function(){
    //wait for the Gazetteer search box to become visible.
    S("#partialAddress").visible(function(){
        //Click to clear the hint. Type a street name.
        S("#partialAddress").click().type("Walton Way");
        //Click the "Search" button.
        S("#searchButton").click(function(){
           //Wait for the results container show.
            S("#resultsContainer").visible(function(){
               //check for at least 1 result
                var items = S(".picklist_row").size();
                //use a qUnit assertion

                ok(items > 0, "pick list has items");
            });
        })
    });
    //Now look in the results list.
    //use jQuery syntax to click the first in the list
    S(".picklist_row:first").click(function(){
        //When selected address is displayed, check the text
        S("#selectedAddress").visible(function(){
            var a = S("#selectedAddressText > a").text();
           //assert it's not an empty string
            ok(a!="", "Selected address: " + a);
        });
    });
    //Now choose a map search to run.
    S("#searchHeader").visible(function(){
        S("#searchHeader").click(function(){
            //this only appears when results have been found
            S(".result_row").visible(function(){
                var divs = S("#resultPanel > div").size();
                ok(divs>1, "Feed search results found");
            });
        });
    });
})

The results are displayed in the test page.

Running this is just a matter of opening the test harness in a browser. Automating it is just a matter of using EnvJs to run it from the command line, supplying the browser(s) you wish to test with. In about 30 minutes I’ve got to a state where as of tomorrow, I no longer need to regression test address searching, zooming to the address on a map or running a geographic search. In another 30 minutes I can get map zooming and panning covered off and have a stab at a map-based click-to-identify-features test. Hurrah – this saves me and the whole team days of time each sprint!

ShortcominGS

So far so good, but to get this to be as useful as possible I need a bit more:

Alerts when the tests fail – run tests from Cruise Control server instead
Ability to test multiple IE versions – create new VMs for IE6 and IE7

I’m really impressed at how far I’ve got so quickly though – it’s going to have an immediate impact. I’ll post more examples as I go along.

Watermarking your maps with the ESRI ArcGIS Server JavaScript API

2010-02-12T21:44:00.005Z

If you work with digital maps on the web outside the USA then the chances are you use copyright-protected data that must meet certain conditions for display on the web or printed page. Often this can be a combination of copyright statement and watermarking.

There are a number of ways to implement watermarking on your maps when you are working with the ESRI ArcGIS Server JavaScript API. In this article I am going to discuss the options and describe the implementation I have chosen.

Watermarking involves displaying an image and/or some text over the map in a way that permits the data owner or map producer to be identified and prevents the map image being reused in a way that circumvents the data license. So, the trick is to get a transparent image to display over the map in a way that the map consumer cannot circumvent. This could be achieved in the following ways:

Stamping dynamically created images with a watermark when they are written to disk, using a server technology such as Java or .Net.
Adding a “watermark layer” to the map service, for example a large polygon that covers the full extent of the map, symbolised with a raster fill symbol.
Drawing a watermark image over the map on the client, overlaying it using HTML and CSS.

Each approach has its potential drawbacks, and the “correct” approach will depend on the nature of your application and the type of map data you are working with.

Which option works best?

With the JavaScript API, we typically want base mapping to come from a tiled, cached source, as it is static data, and user expectation is of Google and Bing-like map performance. In this scenario we can’t stamp a watermark when each map is requested (Option 1), but we could when each map image is written to the cache folder. The folder structure may be a bit too complex to make this practical. So, we could add a watermark layer (Option 2) above at the point of map creation, when the cache is generated. This is robust as the user cannot turn it off, but there can be display issues – what if your application displays other data over the top? The watermark will not be the top level layer. What if your maps are overlaid with other watermarked maps? The overlay of several repeated transparent logos could get messy.

If we’re working with dynamic map services only, then there are plenty of opportunities to include a watermark, but we need a server implementation (such as a Windows Service that watches an output folder) to stamp the image when it’s written to disk – which won’t work if we’re streaming the image as MIME data – or, we have to include the watermark as a layer, which the user could turn off with a simple JavaScript call.

Alternatively we could use a proxy to intercept all map requests and stamp a watermark before passing to the client. This could be done at the map server level using a technique such as an HTTP Module for IIS –intercepting the request and adding a watermark to the image when it is streamed in the response. But, in my case, I am building a web application that consumes data from many sources, so I need to have an option to add a watermark in the browser for those situations where the map service doesn’t contain a watermark layer, or where multiple map sources are combined, or where I have no access to the map server to add watermarking to the HTTP pipeline.

So, it looks like option 3 may have a role to play where the base maps are coming from a tiled, cached source, or where we don’t have the access or maybe technical resources to develop a server-side watermarking application, or where a map watermark layer could be easily turned off by someone with basic JavaScript skills.

Implementing simple client-based watermarking

Given that we now think a client-based solution is worth considering, how might we go about it?

With an ArcIMS application we’d use the Acetate layer, and with a Web ADF application we could add a Graphics Layer – does this work for the JavaScript API? Well, kind of. But JS API Graphics draw as Vector graphics in the browser – Web ADF ones render as transparent PNGs. So we’d need to bubble any mouse events that the graphics captured to ensure that map navigation through mouse interaction still works, and that other map graphics could still be drawn over the watermark. I have to say I don’t like the feel of this approach.

So, that leaves us with the need to tile or centre an image over the map (the image could contain text and/or a logo). What is the best way to achieve this?

Well, one relatively easy way is to add the watermark as if it were a map service “layer” (the JS API treats services as Layers within the map – each map Layer also has Layers of its own, potentially. I had a read of Sathya Prasad’s excellent articles on extending the DynamicMapServiceLayer and TiledMapServiceLayer objects, and thought that if the methods that returned a URL just loaded a watermark, then it’d be relatively easy to add watermarks from a static image as if they were coming from a map layer. And it was. Here are two examples of watermark layers; implemented as a Dynamic layer (that gets one image that fits the map size) and a Tiled layer (that tiles the watermark in rows and columns).

Dynamic watermark layer:

dojo.provide("myScripts.WatermarkLayer");

dojo.declare("myScripts.WatermarkLayer", esri.layers.DynamicMapServiceLayer, {

startExtent: new esri.geometry.Extent({ xmin: 0, ymin: 0, xmax: 700000, ymax: 1300000, spatialReference: { wkid: 27700} }),

//construct the layer

constructor: function(args) {

this.initialExtent = this.fullExtent = this.startExtent;

this.spatialReference = this.initialExtent.spatialReference;

dojo.mixin(this, args);

//set layer loaded property and fire onLoad event

this.loaded = true;

this.onLoad(this);

getImageUrl: function(extent, width, height, callback) {

try {

callback(this.url);

}

catch (ex) {

console.debug("Error getting Watermark image. " + ex.message);

}

_errorCallback: function(err) {

console.error(err.message);

}

});

Tiled Watermark Layer:

dojo.provide("myScripts.WatermarkTiledLayer");

dojo.declare("myScripts.WatermarkTiledLayer", esri.layers.TiledMapServiceLayer, {

watermarkUrl: "",

//construct the layer

constructor: function(args) {

dojo.mixin(this, args);

//set layer loaded property and fire onLoad event

this.loaded = true;

this.onLoad(this);

getTileUrl: function(level, row, col) {

return this.watermarkUrl;

}

});

Usage

This sample code checks to see if the map’s base layer is tiled, and if so, creates a tiled watermark. If not, it creates a dynamic one.

var wm;

var base = map.getLayer(map.layerIds[0]);

if (base.tileInfo == null)

{

wm = new myScripts.WatermarkLayer({ url: “http://www.mydomain.com/somefolder/images/watermark.png” });

}

else {

wm = new myScripts.WatermarkTiledLayer({

watermarkUrl: “http://www.mydomain.com/somefolder /images/watermark.png”,

tileInfo: base.tileInfo,

initialExtent: base.initialExtent,

fullExtent: base.fullExtent,

spatialReference: base.spatialReference

});

}

wm.setOpacity(0.15);

map.addLayer(wm);

What’s going on here then?

Basically, the Dynamic watermark layer is requesting an image from a static URL each time the map is redrawn. Browser caching means this image is only downloaded once, so it performs well. But unless you know what size the map image is, the watermark may get stretched or squashed.

The Tiled layer implementation works a bit better. When the layer object is created, it copies the tileInfo property from the base layer (read: base map service) of the map control. So the watermarkimage assumes the dimensions of each tile image, and is neatly tiled across the screen in rows. You will often know or be able to predict the tile image sizes: 256 x 256 is pretty common, so you can get the watermarks looking a lot better than with the Dynamic layer approach.

When the map is loaded, we just add a layer on top of any others in the stack of services we’re using, and hey presto – we get watermarking over the top of them all.

Drawbacks

This approach is clearly not without its limitations. Here are the main ones I can think of:

Dynamic images will stretch the watermark and distort it, unless your map application has a fixed viewport size.
Tiled images will only work if your base map layer is also tiled, otherwise you’ll need to invent a tiling scheme for the map, and it will behave as if it’s using a cached, tiled map source.
Any JavaScript developer worth their salt will be able to traverse your map object’s layers collection, find the watermark layer, and switch it off. So it’s your decision whether this approach is robust enough for the application you’re developing.

Performance Testing with Visual Studio 2008

2009-11-24T07:41:00.002Z

I'm back at the controls of Visual Studio Team Test edition (2008 this time, woo hoo) loading and running performance tests against some software I'm developing. This time round I am doing basic analysis to see where potential bottlenecks lie, and to spot any memory issues whilst there is still plenty of time to fix these.

However, I had some difficulty getting my tests recorded - turns out there are a few things you need to bear in mind if your test client is Windows 7, Server 2008 or Vista, mainly around the enhanced security configuration (Server 2008) and IE security. Basically, if during your test you switch from one internet "zone" to another - for example, from "about:blank" to an actual website - then IE will kill the original iexplore process and launch a new one. Very secure, but it kills the ability to record web tests until you sort it out.

I found a great blog site run by Michael Taute that lists pretty much all the reasons why this behaviour can happen, and what to do about it. In my case I switched off the enhanced security configuration for administrators, and added "about:blank" to the same internet "zone" as the site I was testing.

See Michael's blog page for the full list of issues and fixes.

Oh, and if, like me, you're an idiot, you'll find your load test doesn't work if you don't create the load test database before you start. There are a few steps to remember before you start testing, as detailed by Travis Lingenfelder.

RAW DATA NOW! My first LocalGovCamp

2009-10-26T17:54:00.007Z

And so to Lincoln's multi-coloured new Innovation Centre, the Think Tank, for my first ever LocalGovCamp. Wow, what a way to run a conference - spread the word on Twitter, allow delegates to set the agenda, and let the group sessions moderate themselves. It was a world away from the sleeping delegates on a day away from work who attend most of the presentations I've given at user conferences and technical workshops over the years. With about 45 people there it's a good deal smaller too, and all the more effective for it.

Why was it so good? For me, it was a combination of factors:

A pretty unique mix of delegates, not the usual corporate conference crowd at all
No big corporate presence, just open discussion (I wondered - unnecessarily as it turned out - if being an ESRI employee would actually upset people)
Great presentations, especially Stuart Harrison's Easy Mapping demo, and Paul Canning's reworking of Jakob Nielsen's still-relevant Discount Usability ideas (well, 20 years after the original was published, only one person in the room had done any user testing of any sort!)
The chance to get past the GIS departments and speak to the webmasters, PR people, and a whole slew of innovative minds - who are now busily filling the web with maps and spatial data, often without a GIS system in sight.

I think the key lessons for me were related to what I do, i.e. web mapping and GIS:

Simple is good. By and large the public don't want really complex map data like OS MasterMap, they want pushpins on a Google map. MasterMap has a place though, for marking up planning applications for example.
GIS Data needs to be indexable and mashable. Webmasters want embeddable maps and raw data: as KML or GeoRSS, not in a black-box, closed system that runs as a standalone web page. At one point the whole room was shouting "Raw data now!"
Usability of old-school web maps, based on GIS systems, is poor. Usability analysis has been neglected for too long, but with just 5-7 users testing, you can find 80% of your site's defects.

So, kudos to Andrew Beeken for organising it, to Stuart Harrison for showing us how to just get on with getting the data out there (and for some constructive criticism of LocalView :-)), and to the whole LocalGovCamp crowd for making it happen.

More RAF connections

2008-06-06T11:12:00.002+01:00

Looks like there's a Boiling connection up the road at RAF Winthorpe (now Newark showground and Air Museum):

http://www.worldwar2exraf.co.uk/Aircrew%20Notice%20Board/aircrew%20notice%20board%2077.htm

Sergeant S C Boiling was based there in 1945 and survived a crash of the Lancaster in which he was a crewman, for the 1661 Heavy Conversion Unit (HCU). The details on the post are wrong, it crashed at Hall Farm, Oxton (near Southwell, a few miles from Winthorpe).

An S C Boiling of Derby reportedly died in November 2007 - maybe the same one?

Spacemen 3 and Loop on YouTube

2006-07-24T13:50:00.000+01:00

Relive the halcyon days of indie hypnomonotony through the wonder of YouTube:

Loop - Collision
A nice collection of Spacemen 3 videos

RT Ponting lbw b Boiling 1

2006-07-04T16:19:00.000+01:00

One of many international batsmen to fall to the wily offspin of James Boiling - Ricky Ponting lasted a mere 6 balls in 1993.

Secret Students On Parade

2006-07-04T16:02:00.000+01:00

Hot off the press...Graham Boiling's memoir of life as a Cold War National Serviceman in the RAF, learning Russian at Crail in Fife. Read the background to the book on the Dundee Courier website.

You can get the book from Amazon or better still, direct from the publisher (Plane Tree): .

Or you can go to Crail airfield to look at the control tower , or even have a race.