EEEPC's power is in the network not the machine

Over here at Technology Treason we lurve ASUS' EEEPC. We liked the idea when they first came out and specifically trawled around Hong Kong computer markets to find one not long after they were launched. It's not the Apple Air or iPhone kind of aesthetic lust, we're talking about true "in sickness and in health" type love when it comes to the EEEPC.

Indeed for someone to now take this device off me it really would have to be from my cold, dead, rigamortis set fingers - and then only with a saw.

Go online and look at reviews. They fall into two camps - those who think it's great as a second machine that just happens to do a lot of funky things (see latest Linux Format June edition for a classic example) or those that just don't "get it" any wonder why the hell anyone would want a tiny-weenie machine when you can get a low spec dell for a few hundred quid now.

There is third camp however who are starting to realise that a linux based UMPC is truly a brilliant bit of kit and it's really because of the network it sits on not the thing plugged into it.

I've had mine for about 5 months and realistically I've installed about half a dozen bits of software - 10 at a push. I can do docs, review spreadsheets, skype, web browse hell even play games if I want and when hooked to a network I can do all of these things with all of the files I could possibly want.

My machine comes home and it auto connects to my home network, synchs to my media server and can play all my media files out of the box. I can check my mail and actually read it without squinting without firing up the laptop. I can connect from home to work via a VPN and mod some files for a client without leaving the sofa or the garden and be doing wht I want before the laptop has finished booting to a desktop.

At work I can use it for presentations and taking notes on projects without printing stupid amounts of documentation and hefting my laptop along with me.

It's not a replacement computer - it's a tool. A finely shaped, infinitely configurable tool. All the things I want in my phone but will never get because of the lack of keyboard,mouse and processing power and without it being much bigger.

The thing is I'm a techie, if I'm talking a walk down the street phone and wallet are it. If I'm going somewhere then it's satchel with camera, book, PSP and now EEEPC in place of a laptop.

ASUS have released details recently of a new version designed to hit off the people who think the other is too small. I don't know myself. Small is beautiful and in this case perfectly formed.

Ubuntu 8.04 - truly desktop Linux

I'm quite an Ubuntu fan, having followed the project since more or less it's original inception. Given the general lack of problems with it thus far you'll notice very few entries on this blog about it.

Indeed various clients of mine are running Ubuntu servers that are easily maintained, easily managed and just generally easy and have been for several years. It's not necessarily an industrial strength OS - see Fedora for example - but for quick deployment, great security and stability and a modifiable tool (thanks to its Debian base) that just gets the job done you don't really need to look much further.

But I think that's about to change. You see Ubuntu 8.04 (the latest version that also happens to be a Long Term Support version) has markedly shifted the goalposts of what I expect from a Linux distribution.

I've been wanting to be convinced to move to a complete Linux desktop for the better part of 10 years but there's always something holding it back - lack of support for a media type, lack of drivers for particular bit of hardware, issues to do with wireless, no power management for my laptop, I can't run some third party apps like Skype... but that has all now changed - and changed in a massive way.

As I'm want to do, every time a new version of Ubuntu comes out it is duly installed on my Acer Travelmate notebook - a very good test of whether an upstart OS "just works" or not. The machine is about 2 years old but it has some quirks such as it has an ATI 3D card embedded on it's motherboard that was difficult to get working properly even on XP, it also has an inbuilt webcam - again with proprietary Acer drivers and it has gigabit Ethernet. Oh and it's widescreen.

About 9 times in 10 I don't even get to a working desktop without some hackery of graphics drivers, x.org files and I've even had a couple of "bomb-proof" distros just not even boot up to a command line.

I've been around Linux for a long time and I know what to expect, I'm an enthusiast and advocate so none of this surprises me at all and I'm prepared to work through the issuses. Most of the time I get to a working desktop with some sort of graphical interface that is mostly not widescreen, with no 3d support, sometimes wireless and without the use of the webcam. Linux isn't aimed at desktop use - it's just a side effect of people using it for development who wanted some creature comforts whilst working - notably the Gnome and KDE bods.

Imagine my surprise when I booted Ubuntu 8.04 and I logged into a graphical desktop that detected wireless and gigabit ethernet, properly displayed my screen in widescreen mode, gave me the option to run my ATI drivers easily and then configured the 3D in a few seconds and on top of that gave me a working webcam that I'd never had running under Linux and you could tell from the whoops of joy that here was something worth formatting my hard drive for.

Every device I threw at it was auto detected and installed in moments, flash drives, USB devices, a weather station, even an old MP3 player than needed proprietary Sony software to synch on XP. All handled with aplomb and with scarcely a pause by the processor.

What the hell was going on? How did we go from solid and okay 7.10 to this awe inspiring 8.04 in just six months? Had Mark Shuttleworth finally given his soul to Beelzebub in exchange for the most promising distro to date?

Then the answer came to me in a word: Vista.

Vista - that problematic and misbegotten child of Redmond that has been causing havoc in the IT world for nearly a year now. I haven't installed it on a work machine, neither has any other techie I know that wants to "Get Things Done". My dad had it and tried it daily for 6 months - he's now back on XP. I know corporate users who've had it on new machines and reverted to XP in order to decrease the amount of support required for users.

Shuttleworth and his cabal of Elite Ubuntu coders have recognised a change is in the air - particularly in Europe that is Linux's stronghold - we have an opportunity to put Linux on the desktop of millions of users who might upgrade to Vista but are worried about its impact. Couple this with a slight economic downturn and people are worried their existing hardware just won't work with Vista thus leading to a higher upgrade cost.

This latest LTS version gives novice and power users alike the ability to do anything they want with their desktop and it just works. It gives corporate users the knowledge that they have the security of support for 5 years without the rug being pulled out from under them.

We bit the bullet this week and put all our support team onto this version exclusively - WinXP was nuked off their machines. The development team are all dual booting but the number of XP desktops seem to be fading from view at a very fast rate as the requirement to just "drop in" on Windows becomes less necessary.

We've been saying it for nearly a decade but "this year is the year for desktop Linux" and with 8.04 Ubuntu the excuses for moving OS can now be left at the door thanks.

Easy product or class rating system

So you've got a lovely little ratings system going on your site. All of a sudden though you get slashdotted, dugg or just your marketing starts working and you have thousands of users all rating your products / services / systems / posts / videos etc and your pages start to creak.

"It's the shared web space you're on," say your techies, "it can't handle the users" and duly bounce you to a better hosting environment at triple the cost along with the migration charges.

From time to time I come across this problem when I've either picked up code from someone else or else a techie asks me how to optimise a page that's running really slowly. In this particular instance it was caused by a ratings system in the style of Amazon or YouTube - basically a user is displayed a product and then people rate it as to whether it's any good. The real problem came when they had a list of products, each of which had it's individual ratings displayed.

The cause of this very slow page however had nothing to do with shared hosting or otherwise or direct server load - it was all down to some naive coding executing what my old CS lecturer would call an O(n)² process.

What the coder had done was get a list of products, then for each product gone back to the database and got a list of all the rankings ever made and then averaged them out. Nice and simple but frightfully inefficient and that which caused the problem I've highlighted.

This isn't the first time I've seen this and I've been asked how to build them numerous times as well so here's a well optimised method of doing it in general terms.

Consider first that calculating the average when you insert into the database is going to be computationally less expensive than calculating it every time you perform a select when a user hits the page. This sounds obvious but it's stunning how often it's overlooked.

Make two extra fields for your product table, one called average and the other called user_count or something. On your insert of the rating into the ratings table, run a trigger or else add some code that will update the product table with the updated count and a new average calculated from the ratings info.

Now when you select the product data you pull down the average and user count as part of that select and they are just simple static fields, thus adding no more computational load than the original select or view does already.

This gives you a nice little rating system that's not heavy in terms of processor load. However we can improve things once step further if you aren't interested in the data.

The option I'm providing below is good if you are just after a running average and don't care about the individual ratings being kept. I did a project recently where we weren't worried about keeping individual ratings data because the site wasn't going to be up for very long and it didn't add anything to our system to have it.

This option uses a running weighted average in order to just update the data in the product table without requiring a ratings table at all.

Some useful background maths though:

If I have a set {3, 4, 4} and take it's average I need to add the numbers and divide by the number of entries. Thus this set's average is (3+4+4)/3 = 3.67

Now suppose I've precalculated this average as I've suggested above and stored it without the individual ratings, I now want to add another rating, 2 to the set.

Intuition says to do something like this: (2 + 3.67)/2 = 2.83 which is actually wrong. Looking at the set {3, 4, 4, 2} we can guestimate that the average is going to be somewhere more between 3 and 4 than it is 2 and 3 as we've calculated above.

Thankfully a technique from statistics gives us an option here which is to use a weighted average instead. This is useful for adding sets together that have different numbers of elements within them but maintain the averages by skewing the data using proportional averages (or a weighted average).

The general formula for this is:

Avg_w = (Avg₁ * (n₁ / (n₁+n₂))) + (Avg₂ * (n₂/ (n₁+n₂)))

Where:

Avg₁ is the average of the first set
Avg₂ is the average of the second set
n₁ is the number of elements in the first set
n₂ is the number of elements in the second set

In our example this simplifies even further because our second set is actuall only one item. So let's work this through:

Avg_w = (3.67 * (3/(3+1))) + (2 * (1/(3+1)))

= (3.67 * 3/4) + (2 * 1/4)

= 2.75 + 0.5

= 3.25

Which is the answer we're after for our average.

As we know all the base line average data in the product table and we know the value of the rating we're tracking, it's a very simple function to update this instead of doing another insert into a ratings table and we just keep on doing it for every rating that has been added.

Computationally this is a very inexpensive process and whilst I'm more than happy to be shown otherwise I think this is about as good as it gets in terms of optimisation.

The key thing is we've now reduced an O(n)² operation to O(n) which is a drastic improvement as n tends towards infinity.

Phorm over function?

Phorm is, and will continue to be for some time I think a hugely divisive issue online. BBC have another story today about it, this time having spoken to the various security companies like F-Secure, McAffee etc about whether they will flag a message to the user about whether Phorm has been enabled or not.

Phorm management have come out saying "it's only a cookie", the same as many other sites use to provide tracking (such as Google Analytics), interactivity (such as shopping carts or ID maintenance on numerous retail sites), or a small amount of memory (configuration information for the BBC home page for example).

The difference, though, is that the information is being used differently because data is being shared.

This is what got the Information Commissioners Office's back up because sharing data between companies without users opting in is a breach of the Data Protection Act - "But not if it's anonymous data" say the legal eagles from Phorm - and technically they are correct. This is a case of adopting the letter of the law rather than the spirit of it.

Tim Berners-Lee came out saying he would move ISP if he found out they were using Phorm and whilst I admire his line I fear the vast majority of consumers won't care or rather just won't be bothered to switch - just see how many people actually switch bank or utilitiy companies.

For me this is a case of the slow erosion of privacy at the hands of our ISPs. In a massively competitive market where margins are being squeezed ever tighter, the sale of their user data to Phorm must have seemed like the proverbial golden goose.

It won't take long for someone to cotton onto the flip side of this and market aggressively on the privacy front. Talk Talk made huge inroads as an ISP on the back of their "The Internet should be free" campaign with regard to price (being bundled as it was with other services). Who will be the first to play the "Internet should be private" card and sign up to a deal not using Phorm or other tracking software?

In my cynical world view, I think the security firms have realised this and it is 99% of the reason for why they are looking at it all as the anti-spy, -mal and -virus software is worth billions.

In real terms Phorm isn't actually that clever a piece of technology - most of what has been achieved is in the brokering of deals between ISPs and content owners and then a bit of clever gluing in the middle.

In the end Phorm will either be a great white elephant and just slip off the radar the way many technologies and companies have done or else it may actually be a spur to drive privacy legislation forward in line with our digital behaviour - how long it will take to do this however is the question as government is typically a long way behind technology in terms of law-making.

Can Yahoo really get things so wrong?

Update - The guys at Yahoo came to our rescue after tracing through the "network" somewhat to find someone that knows someone at Yahoo to help us out. Unfortunately their techies couldn't explain why we'd been bloack listed either but we are now officially on their whitelist so big thanks to the guys for helping us out.

Yahoo are one of the original dotcoms. They've been around for a long time so they should know their business. Imagine my surprise when one of my clients starts complaining that their confirmation emails to yahoo email accounts are permanently being binned as is everything else they send - including personal communications.

Like most mail providers, free or otherwise, Yahoo have a spam policy that will look at an inbound email and then drop it in your inbox or spam folder depending on how it is classified.

As with most techies I have about a dozen email addresses at various providers in order to test exactly these sorts of issues. Especially given that the goalposts are changing all the time.

Sure enough even a personally addressed confirmation email was killed as it came into my yahoo account. "Ah ha," said I, "they've been blacklisted". So off one goes and checks the various blacklisting sites and there's nothing there. Hmmm.

It transpires that yahoo have just taken it on themselves to block that domain. Weirdly though, a personally addressed mail to me from the client with only the word "test" in the subject line is still considered Spam yet an email from some random address that doesn't reply, containing several instances each of the words "penis", "cock", "viagra" and "cialis" made it through to my inbox completely unscathed. At this point the phrase about arses and elbows definitely comes to mind.

Trying to get Yahoo to do anything about this issue is similarly problematic as there are no feedback channels to deal with this problem at all.

So overall we've just had to advise people to not use Yahoo or to check their junk mail periodically and read the mail there.

Security 101 : The user should be able to authenticate

Are you listening Barclays?

I like security - particularly data security and in very particular data security that protects my personal information (unlike a certain Uk government department a few months back).

However, I've been around this game long enough, worked for a bank long enough and built more web applications capturing user data for long enough that I know there is one fundamental truth when it comes to data security and that is: pragmatism.

When I was at Uni I was told, "The only secure system is one that has no network connection, no keyboard or mouse and most of all no users" (and I apologise Dr Fekete for bastardising your phrase but you can't have done a bad job for me to remember it 15 years later!).

However the flip side of all of this was that depending on the data being protected, the security protocol should be appropriate without undue burden placed upon the user. Which is why logging into flickr is trivial but logging into your bank should and is a more arduous affair.

Banks are very secure enviroments which is good because the last thing I want is some 13 year old script kiddie making off with the tens of pounds in my bank account. Having said that, the bank should never make it difficult for me to get to the tens of pounds in my account due to security reasons.

At the moment though banks are running very scared and they are nailing the customers because of it. On my recent trip to Australia I had my card stopped no less than three times because Barclays decided that the activity looked fradulent.

Initially I thought something serious had happened but a call to Barclays got them to right the problem which was part of their new security measures. The next time it happened was because Barclays decided that it was time for me to come home and that I shouldn't be using my card in a Fraud Capital of the world like Sydney. The third time it happened though it locked my account out entirely and I was told I would have to come into a branch with identification documents to sort it all out - except there aren't any in Australia and I was leaving the next day for Hong Kong. Luckily a very understanding parent lent some cash.

I applaud Barclays' sentiments - they really were trying to protect my account, however it would appear as though client / bank trust has disappeared and I can no longer say "I want access to my money globally" without alarm systems going off all over the place. If I was backpacking I'd have been in serious trouble as without a bailout I literally had about 10c in my pocket.

Upon return to the UK Barclays' statement was along the lines of "Sorry but we're dealing with a lot of fraud and it's better to be safe than sorry". Tell this to one of my employees who just had £3K wiped out of their account due to identity theft (spent on local UK products and didn't fire off a single warning) and they are being told they have to prove it wasn't them...

In a way I feel sorry for Barclays because they are damned one way or the other - on this issue though it should just be a case of phoning and doing a vocal authentication then saying "I'm abroad for 4 weeks allow any transactions from xyz country until I say otherwise". In this manner everything other than DDs occuring in my home country should be treated as fraudulent and everything authorised abroad should be fine...

Bring on the chip in my hand is what I say...

DVD Jon strikes again

At Technology Treason we love DVD Jon or Jon Lech Johansen as he is more commonly known. This great Norwegian famously broke the DVD encryption put in place by the big firms with the release of some software primarily aimed at allowing DVDs to be played on computers and unlocking the regionality of DVDs and DVD players.

When he released DeCSS he ran afoul of the US DMCA and was almost charged, he was then indicted by Norwegian authorities acting on behalf of the US who actually did go to court twice to try and convict him of hacking. Both times they failed and decided not to go to the Supreme Court.

Imagine our complete amusement in the office when we find out he's now trying it on with Apple via iTunes.

iTunes is a love it or hate it product - if you are part of the Apple / Steve Jobs faithful it is obviously the greatest thing on earth, if you know nothing about technology it's a simple product that allows you to use one of those "fangled new digital music type thingies".

If you are a techie you see it as a proprietary lock in and try and avoid it like the plague. The main issue for most techies is you can't play your music on anything other than your PC / Mac that has iTunes installed and your iPod / iPhone / iTouch.

I've railed against lock in for time immemorial - just a quick count of my personal items puts the following music players at my disposal - mobile phone (x2 because my wife has one that can play music too), MP3 capable stereo, PC (x3 - my office, my home and laptop), PSP, Xbox, a real MP3 player and my Nokia Internet Tablet - 10 devices at my personal disposal that I want to play music from and indeed do play music from.

The thing is, I know how to do all of this so I just push the files around on memory cards or over my network (streaming from my media server for example) onto the various devices. For many people this isn't possible and Apple's enforcement of the iTunes lock ins firmly violate the right I have to play my music (or video) on whatever device I choose at whatever time I choose. I also vote with my wallet and don't buy tunes from Apple.

What DVD Jon has done with his software (available from DoubleTwist for free) is allow you to take files that are locked into iTunes and essentially it plays the file, re-encoding it into a format you can play on other devices (I haven't looked properly but presumably OGG or MP3).

Just to rub salt into the wound he's going to cause Apple and the US music industry he's decided to let you share your files with friends as well. One wonders how long it will be before a writ arrive from the RIAA and Apple... I'm sure they'll be racing to get in first.

So well done Jon - keep up the good work and keep fighting the good fight - media we have legitimately purchased is ours to use on any device we own for our personal use.

Eventually the media industry will wake up and realise where they've been going wrong. Perhaps if EMI had taken notice of the way the world was going they wouldn't have had to cull a couple of thousand staff.