Archive for the ‘techie’ Category

Wisdom of Crowds Mark II

May 21, 2010

After twitVoteUK comes soccervote. You may know the FIFA World Cup kicks off in South Africa in June, and so we’ve put together a World Cup prediction site at The backend is pretty much the same as twitVoteUK, but Chris has put a lot of work into the frontend to offer a global view, refine the form and add lots of language support, given we want a global submitting audience.

There is a subtle difference behind this effort. Last time it was about obtaining people’s opinions in order to get a poll to try to reflect how the nation might vote. Those people were an attempt to see if that sample set would predict the outcome of the election. They were representing the people who would go to the polls earlier this month and place their votes for real. This time the people who add their predictions on our site are not going to be playing in South Africa, they have no influence on the outcome of the World Cup. Instead this is a wisdom of crowds experiment, ie can the majority view of people who submit an opinion on our site accurately predict the winners of the World Cup? Indeed, will the second, third and fourth listed teams make the semi-finals and so on? Once the tournament begins and teams get knocked out we’ll use the rules in our event processing system to prevent folks from voting for invalid teams, and if volumes of votes increase, then we’ll modify rules to highlight particularly interesting countries of origin in terms of frequency of voting or frequency of change of leader.

What would be good, though, was if we could include Scotland in the list of contenders. Unfortunately we missed out on qualification again. 😦


TwitVoteUK: the aftermath

May 16, 2010

It’s over a week since the election, and how did TwitVoteUK end up? On a pure results level it was completely wide of the real result. Overall we had around two-thirds of votes for the Liberal Democrats, against 23% in the election. Labour and the Conservatives came second and third, but with 10% and about 7% respectively. Lots of the seats that we had votes recorded for only had one vote, meaning that the likes of the Pirate Party UK would have won seats according to our poll. The overall percentages were similar to other online polls, such as CountMyVote, so I’m not disappointed with the final stats. We did reflect the mood of Twitter, which anyone would have seen by following either the #ge2010 or #ukelection tags. We received just over 1500 votes from unique Twitter users, with many people voting more than once, and taking advantage of our facilities to change your vote without having duplicates. In my opinion, to have produced good numbers for each constituency we would have needed at least 100 votes per seat, or 65000 overall, which was far too difficult without some relentless advertising and press coverage. We did get a piece in the Isle of Wight County Press and the Ventnor Blog, both of which managed to drive the Isle of Wight seat to the top of our leaderboard for number of votes.

We received some positive comments in technical blogs, such as Andy Piper and James Governor. I was very pleased to get respect bloggers and social media people giving positive feedback on what we’d done on the site. Also, Ben Marsh, whose uksnow website was the inspiration, tweeted some very positive comments:

If I was going to build a #uksnow inspired election map app, I’d probably end up with something like @twitvoteuk. Nice work chaps. #ge2010

In terms of the result, okay, it was not where near the election result, but it did show that you could gather seat-by-seat data and display that on a map. The gap between each vote and it appearing on the web was no more than two minutes, and it would be fairly straightforward to reduce the window quite considerably. The technology we were using allowed us to restrict each Twitter user to holding only one vote, and to only voting within intervals we set, normally 24 hours. It was quick to change that window, for example during the leaders’ debates we cut it to 5 minutes, so that people could be influenced by what they were watching.

Going forward, we’ve got something on the go that will have more world wide appeal, and some improvements to the backend in order to make it more robust and increase the response time between the vote and the map being updated.

Finally, thanks to Chris Phillips and Ben Hardill, with whom I’ve been working on this with. There is no way I could have done this on my own because I have insufficient knowledge about web technologies to have scoured Twitter for the votes or to do the website, and Chris is really pushing the rest of us on the next rev.

UK Election Polling on Twitter

April 24, 2010

Ok, this post is way off beam compared to what’s gone before, so if you are neither interested in a) Twitter; b) the UK General Election, then don’t read any further.

Over the past couple of weeks I’ve been spending lots of time polishing off an idea I had a couple of months ago. It was inspired by the site that Ben Marsh put together at, using a hashtag on twitter to gather the observations of folks looking out their window at the snowfall when the UK was hit by a couple of snowstorms in January/February of this year. You could look at his map of the UK and at the rainfall radar charts on the Met Office website and see a direct correlation between the two, verifying the accuracy of the crowd-sourced data on the uksnow website.

I thought that a similar idea could be applied to people’s opinion on who they would vote for at the upcoming general election. The main problems were:

  • Ben used the first part of people’s postcode to put a symbol on a map, but quick research showed this wouldn’t be possible for mapping to constituencies, which I wanted to do. Postal districts and constituency boundaries do not match. Therefore the full postcode would be needed and would people stick that in a tweet?
  • Ben had a simple score out of ten for folks to measure how heavy the snow was. This, in my system, would be the party they’d vote for, but there are many of those, and the variety of spellings and abbreviations people might use could lead to missed votes.

Ideally I’d like to have just parsed tweets to establish sentiment, but I quickly dismissed this as being far too difficult, although I note there are a couple of sites experimenting with this approach, eg tweetminster.

So the journey began when I spoke to the renowned Twitter expert at work, IBM Distinguish Engineer and fellow Isle of Wight resident, Andy Stanford-Clark. He created the twittering ferries and twittering house. He reckoned it was a worthwhile idea, and put me in touch with a couple of other folks who had done web and twitter based applications, Chris Phillips and Ben Hardill. After discussing my idea with them, they were sufficiently interested to join and help me to put it together, and I especially needed their help because, hey, I’m not particularly technically proficient, and know nothing about websites beyond very basic html.

We didn’t do a great deal for a few weeks, mostly due to any of us being away from the office, other than putting together an architecture picture. Fast forward through the three of us finally pulling stuff together, and Chris registering our domain and pulling in one of his friends to design the website, and twitvoteuk was born. The how to page on the site describes how the internals work and so does a blog posting from another Andy from work (Piper this time), so I won’t cover it here.

How will this play out? I don’t know, but the most important is to get as many opinions recorded in our system as possible. Two interesting things have happened. The first was last Monday, shortly after we’d activated a form on the website so tweeps didn’t have to post their full postcode to the web, and some LibDem mailing list must have published our website and encouraged folks to use it, and for a while that afternoon Chris and I were chatting on instant messanging, asking each other what was wrong with our system because every single vote coming in was for the LibDems. Fortunately all was good with the system, it was only a party mobilising its supporters. Ever since, they’ve stayed ahead.

The other incident was on the evening of the same day when I noticed that someone had posted an opinion for the Pirate Party UK in a constituency where the BBC didn’t show they had a candidate. I tweeted about this from our @twitvoteuk id, and quickly got a response back from one of their supporters saying we didn’t state whether we were polling for parties people would vote for, or who they would like to vote for. Ever since, there has been the occasional tweet encouraging Pirate Party supporters to grab available constituencies and they remain in second place in our poll.

A final note – while I have political opinions, I do not support any particular party and this endeavour has no political aim. It simply offers a technical solution to the problem of polling a population of people and visualising that data in a geographical way.


January 15, 2010

I’ve been using Twitter for a several months now, but never mentioned it here. If you use Twitter and would like to follow me, I’m @kieronb1.

Shiny New Phone

July 10, 2009

So I have a new phone, with new phone number (got to text that out), and I’ve spent most of the day playing around with it (much to Hannah’s annoyance). It’s quite a change, even for me who has had a smartphone (Windows) for several years. The most important difference, apart from the interface and 3G, is the fact that it is location aware. It is a HTC Magic, a Google Android based phone (Android being the operating system). Obviously it ties into Googles web-based apps, like Maps and Mail (making me sync my Yahoo mail into Google to get the best email support on the phone). Using the touchscreen is very different too. My old phone used a stylus, but this one doesn’t work with a tap from an inanimate object, instead it uses the electical contact from your fingers to sense the weight and length of the contact. This is what the iPhone does and is all the rage. It’s nice to experience it. I do have rather fat fingers, though, so hitting the right key in the on-screen keyboard is tricky, but the predictive text also checks for likely spelling errors, so often the phone corrects my bad typing.

I’ve downloaded a few apps – simple things like a battery display and compass, as well as a Weather Channel app, task/to-do app, bar code reader (which hasn’t recognised anything I’ve shown it yet), My Tracks (which records where you’ve been, and promises to be excellent), and loads more. The Places Directory ought to be very useful when visiting new places, as it gives a directory of shops, entertainment etc based on your location, which is established using your mobile cell or the in-built GPS. I’ve registered for Google Latitude, so if you’re a user of that, let me know and then we can each know where the other is at all times. Spooky.

So overall, I’m liking it as a device. Plus it’s lighter and smaller than my bulky old smartphone, and with the 16Gb micro SD card (they are shockingly tiny), I’ve now got a new MP3 player as well.

New Phone on the Horizon

July 1, 2009

My phone is (I think) 3 years old. It’s a Windows Smartphone with touchscreen, and does most of what a modern smartphone does. It is a bit long in the tooth, though and quite blocky, and Windows Smartphone 2003 is not the best mobile OS. Plus I’d like to merge my MP3 player (a Creative Zen; also aging: I bought it for Hannah and I’s holiday to Mauritius in 2006) and therefore reduce my small device charging requirement by 1 and have one less device to carry around.

I’d had a look at the HTC Touch 3G on Orange, as that’s the network I’m on now, and the phone I have now is a rebadged HTC, so I’m used to the manufacturer. It’s another Windows Smartphone and that would mean being able to easily access stuff on my Windows desktop (files, music etc), install Yahoo Go for email and contacts, and get all the normal things off the internet that you can for Windows. It’s nothing special as a smartphone, though. I don’t want an iPhone, as I don’t have an iPod (most of my music is in WMA format), don’t have an Apple Mac and don’t use iTunes. I had looked at the HTC Magic, one of the Google Android phones, but that would mean switching network to Vodafone, and Hannah would still be on Orange, meaning higher charges for her calling me, and so had dismissed it. But… two of the guys in my team have just got one (with rebate from the IBM Rewards scheme for employees) and both rave about it and the apps it comes with. The major advantage of getting a new phone seems to be location awareness, ie the phone knowing where it is and being able to link you to local services etc.

I am tempted by the Magic, we’ll see what transpires.