The Mystery of Safari To Go: why is it so bad?

A lot of applications make the IPad shine: they’re fast, responsive, elegant and take full advantage of the touch screen: Flipboard, Ocado, Twitter and most magazines (Wired, Economist, New York Times, …). Swiping is smooth, the whole screen is used.

And then you have things like BBC IPlayer, Amazon Touch Shopping or Safari To Go that in terms of user interface are plain rubbish.

Take Safari To Go: I was expecting to be able to easily flick through my collection of Safari books stored offline, it was the reason why I bought the IPad in the first place. Well it turns out you can’t even swipe the pages!!! You have to press left/right arrow buttons to navigate. It’s full of strange rounded dialogs that waste space. It’s very buggy. The table of contents appears in an annoying modal dialog right in the middle of the screen, the table hierarchy is flatten and slow to scroll down… I actually find the web version of Safari Online Books to be much more usable than the IPad app, even if the web version does not support offline storage.

Amazon Touch Shopping is not as buggy but you will find similar clunky rounded dialogs, wasted space, slow scrolling plus some very annoying sounds that acknowledge every user action.

Although it’s fantastic to be able to watch BBC programs on the IPad for the price of a TV license, the BBC IPlayer app suffers from the same sort of web-like sluggishness.

What’s frustrating is that we’re not talking about cheap farting apps here: Safari Books Online, BBC IPlayer and Amazon are all extremely useful platforms supported by companies that have a habit of being innovative. I guess it’s just poor technological choices. I don’t know how they’re designed but I suspect that all 3 apps are actually web browsers running javascript… At least that’s what they feel like.

Update: (13/02/2011)

• I read that OReilly removed Safari To Go from the app store on 24/11/2011, 3 weeks after it introduced it. They’re currently working on a new version (yeah!).
• There is now a proper BBC IPlayer (released 10/02/2011) designed for the IPad from the ground up. It’s much much much nicer than the previous one.

Currently Listening to…

• Mark Russinovich: Mysteries of Windows Memory Management Revealed (PDC10)
• Mads Torgersen: Inside C# Async (Channel 9 Going Deep)
• Anders Hejlsberg: The Future of C# and Visual Basic (PDC10)
• IPad Today 17: Radioliffic Apps, Mute Button Woes, iPad Cat Stands, Fruit Ninjas!
• Net@Night 174: Instagram
• Roy Osherove: Unit-Testing C++ with Isolator++ Webinar

Watching PDC10…

I’m  impressed with the live streaming video system that Microsoft set-up for the PDC.

You can watch live or past sessions. I’m watching the keynote right now that took place just a few hours ago.

The player lets you switch between slide view and stage view.

I wish they did that for the previous TechEd events… Too bad the Ipad doesn’t do Silverlight.

Update: when the US sleeps (now 07:54 BST), quality switches to HD, see below:

Notes from DDD8a

The DeveloperDeveloperDeveloper event at the Microsoft offices in Reading was split in two tracks of simultaneous sessions.

Notes from some of the sessions:

It’s Time to Look at Entity Framework

Speaker: Julie Lerman (twitter @julielerman) thedatafarm.com

The Entity Framework was initially designed to reverse engineer existing databases but you can now also create them from scratch.

The designer generates a DDL. It doesn’t actually create the DB, it creates a script. Then you give the script to a DBA (Not sure how you specify paths for database and log files ?).

Support for POCO classes: your classes don’t have to inherit from Entity Framework classes, you can keep your model totally separate from EF and still work with the designer. The way you do this is through T4 templates, some of them you can download from Visual Studio Gallery.

If you tweak the T4 template you can totally abstract the EF layer (ObjectSet, etc…). In other words your code can interact with EF only through interfaces, which makes unit-testing possible.

TODO:

• check out T4, CodeFirst

Packaging in the .NET World

Speaker: Seb Lambla (twitter @serialseb)

Seb demoed OpenWrap www.openwrap.org (the site is going live live in a few days), a packaging tool that you use to inject dependencies into your build. The dependencies must be available as “wraps” for this to be possible. You inject the dependencies with a simple command-line tool.  

The openwrap commands can also be used from within an msbuild file.

There is another open source tool called Nupack (maintained by Microsoft) released on codeplex.

TODO:

• Check out Scott Hanselman’s blog about Nupack

Is NoSQL the Future of Data Storage?

Speaker: Gary Short (twitter @garyshort)

• Term introduced by a programmer from LastFM.
• NoSQL
  • often does not implement ACID
  • avoids joins
  • no fixed schema
  • scales horizontally (adding more machines)
• Types of NoSQL DBs
  • document Store
  • graph storage: nodes and edges
  • key/value stores: on disk/on RAM
  • “eventually consistent”
  • object DBs. You provide your own indexing rules.
• Good for:
  • geographic regions, large quantities of data, game server sharding (what’s sharding?)
  • often written, rarely read > Key/Value
  • binary data
• Example: Twitter
  • they tried RDB > didn’t scale.
  • built FlockDB
    • not optimized for transversal, because not needed.
    • optimized for adjacency lists: graph stored as set of edges
    • idempotency: useful for computing set unions and intersections.
  • Lessons learned: use aggressive timeouts

TODO:

• check story of Twitter (DB angle)

Modern C#: this is not your grand-daddy’s language

Speaker: Jon Skeet and his pony. (twitter: @jonskeet)

Inspiring talk about the evolution of the language features of C#.

Jon illustrated how powerful C# has got by writing a MaxBy() implementation in various versions of C#. MaxBy() should return the element of a collection containing the maximum value of a specified field. It should work with any field and any collection.

He wrote the solution in C# 4 use generics, lambdas, method extensions and inferred types.

In C# 1.0, he started with a simple for loop: this lead to code specific to the class being compared. To make the code a bit more generic he used a delegate and some downcasting from System.Object to the type compared: it worked but was somewhat verbose and not as type safe as the C# 4 version.

Jon’s quotes:

“Knowing how things work under the hood is important. However you need to be able to shift gears so that you think at a higher level when you need to”

“We need to be jolted out of the idea of what a language should look like.”

“You can express yourself more clearly without having to write things you don’t need. What needs to be expressed is expressed only once.”

“Learning F# helps you understand the new features of C# 4.”

Things you should know about SQL as a developer

Speaker: Simon Sabin (twitter: @simon_sabin)

• Do not truncate the transaction log > you loose point-in-time recovery
• Re-indexing
  • You may just need to update the statistics.
• Shrinking
  • usually bad, files grow for a reason
  • shrinking makes sense after a big import job only
• Clustered indexes on dates
  • make them small and unique
• Types of joins
  • Loop: for small datasets
  • Hash: large datasets
  • Merge: requires sets to be ordered
• user-defined functions
  • bad performance!
    • interpreted
    • can’t use parallelism
  • demoed a padding operation done with UDF vs one done with a CLR function. The CLR one significantly faster.
  • Prefer CLR functions over UDF.

TODO:

• Look into taking snapshots vs restoring from backup.

 

More blog posts about the event:

Danny-T.co.uk

The Ninja Ferret

Slides:

Chris Hardy’s WP7 talk

Ipad/Iphone Apps List

IPad

• Ocado

 

Very well suited to the Ipad, here is a typical use case:

1. Book your delivery slot (slots for the following evening are usually free).
2. Add all items from the previous order
3. Display basket and go to your kitchen
4. Remove unwanted items (check your fridge and cupboards)
5. Add items you know are missing
6. Go back to your sofa and browse your favourites for things you might have forgotten
7. Checkout: type in the voucher (Ocado often emails 15% or 20% reduction vouchers)
8. Type in the 4-digit pin to authorise the transaction, and that’s it.

 

• Reeder

• Quickly syncs with Google Reader.
• Elegantly displays groups of feeds
• How I use it: I star articles I want to read during the week and take the time to read them at the week-end.
• No pictures, no colors :-(
• Too bad you can’t subscribe to a new RSS feed from within Reeder.

 

•  Pulse

Unlike Reeder, this one will show the pictures included in the RSS feeds.

You can create tabs to arrange your feeds. Inside each tab you can add feeds imported from Google Reader. Note you can only import from Google Reader, it does not sync.

Pleasant to look at but from a functional point of view, Reeder is still better because it syncs with any change you make to your subscriptions in Google Reader.

• DropBox

There is a small set of files I always want to have with me. I use DropBox to keep them in sync. I installed it on my PC, 2 laptops, an Iphone and an Ipad. Changes made on any device are immediately propagated to the others if they’re on.

The Windows version gives a version history of each file.

IPhone

• 2Do Lite

Clever little app for TODOs. You can associate a TODO with a geographical location. Click the nearby button to filter TODOs by current location. This is good for the situations where you say “Next time I go to Boots, I should get some shampoo”. Elegant, fast, good-looking.

• TimeOut London

Search films by location, by date, by cinema.

I use it to look-up the films available in a short list of of favorite cinemas.

• FourSquare

This is what Google Latitude should have been. Check-in to pre-defined places (restaurants, bars, companies, hotels, train station platforms, forests…) to tell where you are or add tips about a particular place. This app would be great if only more people used it: the nearby tips tend to age a bit.

It’s all I wanted to do with Google maps: see a place I like, make a note of it to come back later or go to a place and leave a note about it. I used to do it with Google maps, it’s much more clever with FourSquare.

A friend talks to you about this new restaurant he tried: look-up the name, add it as todo and attach a tip “Remi recommended the Savoy bar because it was recently refurbished and just re-opened”

A number of little apps are available to make it easier to use FourSquare, in particular to automate the check-ins: CheckMate and FourMinder.

• TimeOut London

Search films by location, by date, by cinema.

I use it to look-up the films available within my short list of favorite London cinemas.

• DropBox

The Iphone version allows you to cache some of the files for offline view (by making them favorites). You can’t view the version history though.

UK TechDays Special Event with Steve Ballmer

I’ll go hear about Windows Phone 7 and Azure on Tuesday. It will happen at the ICC in the Docklands (Royal Victoria Dock).

It seems it’s still possible to register.

Notes about Snapshot Isolation

MSDN: Using Snapshot Isolation

The SQL92 standard defines four isolation levels, two of them being READ_COMMITTED and REPEATABLE_READ.

READ_COMMITTED prevents the current transaction from seeing data changed by uncommitted transactions.

REPEATABLE_READ guarantees that the data read since the start of the current transaction will not be changed by other transactions. This way the state of the database remains the same from the point of view of the current transaction.

By default, SQL Server uses locks to implement READ_COMMITTED. That means any query attempting to read a row that is being changed by a transaction will block. Not great really. Oracle uses rollback segments to achieve the same thing and queries are never blocked by writes!

Snapshot Isolation (from SQL Server 2005 onwards) is another way to implement REPEATABLE_READ and READ_COMMITTED without using locks. Instead row versions are saved in tempdb every time an insert or update takes place.

To allow all future sessions to use snapshot isolation:

ALTER DATABASE MyDatabase
SET ALLOW_SNAPSHOT_ISOLATION ON

ALTER DATABASE MyDatabase
SET READ_COMMITTED_SNAPSHOT ON

To enable REPEATABLE_READ in snapshot mode for the current connection:

SET TRANSACTION ISOLATION LEVEL SNAPSHOT
Articles

MSDN:
Isolation levels in SQL Server 2005
Understanding Row Versioning-Based Isolation Levels
Cost of Row Versioning
Stack Overflow:
Why is READ_COMMITTED_SNAPSHOT not on by default?

A week-end in the South of France using the Ipad

I enjoyed a long and sunny week-end in the south of France at a friend’s place (Gordes, Provence). I spent many hours near the swimming pool toying with the Ipad (the first Apple device I’ve ever bought).

I like the sexiness of the hardware but plenty of little things keep annoying me.

Intended home usage:

• read my RSS feeds
• browse Oreilly Safari
• read kindle books

I’m still looking for:

• A better browser, because Safari lacks some really obvious features such as tabs or page text search.
  • A1 Perfect browser: would be great if it didn’t reload the last page every time you switch back to it.
  • ICab Mobile
    • To be tried
• a RSS Feed reader that compares to Feedly. I use Feedly on the desktop but it does not exist on the Ipad yet so I looked for alternatives:
  • Feedler:
    • the experience is similar to Google Reader, hence functional but a bit boring.
  • Reeder: could not install
  • Early Edition:
    • Visually pleasant although not as much as feedly
    • Very slow to refresh
    • imports feeds from Google Reader but does not sync automatically.
    • No mark for later function
    • Doesn’t remember what you read.
    • Displays original posts with embedded browser instead of opening Safari (which a good thing).

Some random observations:

• You can’t see anything on the screen if you wear polarising sunglasses. :D Not a problem but it’s the first time I notice something like this on any device…. Update: (12/09/2010) When using polarised RayBan sunglasses, the IPad screen becomes invisible if you hold it in portrait mode. In landscape mode it’s fine.
• The absence of notice means you have to play around to discover some features of the hardware/OS/virtual keyboard, that’s actually quite fun.
• The WIFI issue
  • I observed that my Ipad loses the Wifi connection if the connection is unused for more than 5 minutes (does not occur at 4 minutes).
  • Router tweaks I tried with no success:
    • Disabled security
    • Changed the wireless channel
    • Switched down to 802.11b.
    • Used a static IP address
    • Updated the firmware of my Belkin F5D8635v1 from 1.00.17 to 1.00.23

Apps reviews:

Wired
PadGadget

Update: (07/08/2010)
Safari Books Online plans on releasing an app for the Ipad this summer.

SQL Server and Notifications

Chaps from Microsoft gave us a presentation of StreamInsight at work the other day.

Following this I started digging around the whole SQL Server notification/event thing: how can an application be notified of a data change instead of polling the database like a drowsy lobotomised lemming?

Service Broker

Messaging infrastructure for SQL Server. To enable it on a database, right-click the database in SSMS, click Properties > Options > Broker Enabled

Service Broker
Servicer Broker Programming

T-SQL has a few constructs dedicated to using the service broker.

Query Notifications

Requires Service Broker to be installed on the current SQL Server instance and enabled for the current database.

Using query notifications
Query Notifications in SQL Server (ADO.NET)

SqlDependency

This is the class you use to implement Query Notifications in a .NET app. It is supposed to be used by server apps (ASP.NET for instance). The documentation warns it is not designed for client apps. It probably doesn’t scale too well.

SqlDependency class
Detecting Changes with SqlDependency (ADO.NET)

Notification Services

SQL Server 2005 only, not 2008. This is a publish/subscribe messaging service.
Notification Services Tutorial
What does Notification Services do?

StreamInsight

Big Diagram
StreamInsight Installation
Download
Deployment model
Concepts

Other Stuff…

• Change Data Capture / Change Tracking

This is more for auditing purposes: allows you to easily query historical changes made to the data without having to write tons of error-prone code in triggers (as I saw being done on an Oracle database by a client I used to work for…)

Data Capture and Change Tracking compared

• WAITFOR

Can be used to delay a statement: block for 2 minutes before running a statement for instance or can be used to schedule a statement to run at a specified time.

Using WAITFOR
WAITFOR (Transact-SQL)

FOWA Dublin 2010 – Notes

Some notes scribbled down during the talks.

Twitter and Geolocation

Raffi Krikorian (twitter, blog)

• Yaketee: very simple web app that automatically detects your location and allows you to create chat groups with people in the same location.

What makes a place?

• longitude, latitude. Sometimes this is a bit too precise.
• A polygon (area rather than exact location). According to Raffi defining locations with polygons is currently what Twitter does for named areas.
• Area name / town / country
• WOEID: Where on Earth Identifier, defined by Yahoo.
• TwID (?)
• IP-based location
• w3c location
• brokered location: redistributes location info to other apps. Example: Yahoo’s fire eagle.
• location databases
  • Yahoo (WOE db)
  • Twitter

Cloud Computing: Innovation vs Commodity

Simon Wardley (twitter, blog)

Simon based his talk on a graph showing the relationship between ubiquity and certainty (slides). It’s an asymptotic curve. On one end of the curve innovative technologies have low ubiquity and low certainty. At the other end of the graph, commoditized technologies have infinite certainty (no more change) and a capped ubiquity.

How can a technology move from the innovation stage to the commodity stage and become a service? Four requirements:

• concept
• attitude (willingness to see IT as a commodity)
• suitability
• technology

Hierarchy and componentisation: stable components act as building blocks and accelerate innovation.

Any technical progress that makes something more efficient increases the demand for its use.

Risks associated with move from innovation to service:

• highly disruptive transition, how do you manage the new supplier?
• outsourcing risks: lock-in, second sourcing…

Myths around cloud computing:

• You have a choice (actually you don’t).
• It will reduce IT spending. It doesn’t because you’ll end-up spending the same amount of money, you’ll simply do more stuff with it.
• Cloud is an innovation. It’s not, it’s a commodity.

Accessibility

Robin Christopherson (twitter, blog)

• Flash is really bad for accessibility: nothing can be done with the keyboard.
• Internet Explorer has a function to bring up all links of a page inside a dialog. Not all browsers support that.
• Chrome not accessible.

Robin did his talk and demos without being able to see the screen. He used keyboard shortcuts and text-to-speech. When he scrolls a list of items the system enunciates the links at very high speed (much too fast for me to understand, but I guess if it’s the only way you can use a PC you must end up getting pretty good at it).

• Automated caption and automated timing. Youtube makes it easier to produce videos with captions: you upload the video and the transcript. Youtube uses voice recognition to associate time markers with captions (so you don’t have to do it yourself). It sort of works, enough to be useful.
• Internet TV: projectcanvas.info: joint venture between the BBC, ITV, C4 and Five to build an internet TV platform with accessibility as a prime requirement.
• Avoid horizontal scrolling: Opera mini can force single column browsing.
• Siruna: can crunch down a website for any mobile platform.
• Textcaptcha.com. Accessible CAPTCHAs. The distorted pictures of random words are not usable by blind people! This site suggests an alternative using logic questions. They even provide a REST web service.
• Accessibility testing budget encompasses usability testing. i.e. if you test an app for accessibility for the blind you will spot usability issues affecting everyone else.

FOWA Dublin 2010 – Overall Impressions

The Irish edition of FOWA in Dublin lasts only one day. But because the presentations are short (about 20 minutes) you end-up seeing a lot of talks. And as usual what makes FOWA different is the opportunities you get for casual conversations with the speakers.

• FOWA Dublin
• Full list of speakers

The venue was a building in the grassy campus of University College Dublin. Everyone could sit outside in the sun for lunch.

The talks that stood out for me were those by Raffi Krikorian (Twitter) and Christian Heilmann (Yahoo), both dealing with geolocation. When it’s about geo I always think of Google Latitude and Bing Maps but Yahoo actually does a lot too, especially in the area of APIs and web services.

IPv6: we’ve been hearing about the exhaustion of IPv4 addresses for a long time. I remember my teachers talking about this at school back in 1996. Well Owen Delong explained that this is now less than two years away. More info on the ARIN site. You can even get counter badges that give the current estimate for the exhaustion date.

I was sitting next to the CEO of TicTacDo, a collaborative how-to site that looks very useful. You could think of it as a Stack Overflow where each question would get a single answer in the form of an actionable TODO list. The app has been live for a while and the enterprise version will launch soon.

Databases: there are many alternatives to the classic relational database. Chris Lea described the NoSQL approach (slides): a range of tools (CouchDB, MongoDB, JackRabbit) should be considered before jumping on MySQL, SQL Server or Oracle.

Some excellent non technical talks by professional speakers: Alex Hunter about taking care of your brand and the lovely Relly Annett-Baker about content creation.

The after conference party at Solas was great: free bar from 7.30pm to midnight (you can never go wrong with that), very good atmosphere and good crowd.

One thing I noticed: during the last FOWA I attended there was an army of bloggers taking notes live on their laptops and posting as the conference was progressing (I could find the posts in Google 30 min after a presentation was finished). Not this time! I think I know why: three years ago people were using blogs to post live information, now they use Twitter, which is better suited for this kind of thing. Somehow Twitter has made blogs more quiet.

Maps at the British Library

The Jubilee line is working this Sunday. I’ll make the most of this exceptional event by going to the new maps expo at the British Library.

In September there will be a talk by Ed Parsons geospatial technologist at Google Maps and Steve Chilton from OpenStreetMap.

Insight Into an Agile Team

 

It’s one thing to read about agile in books. It’s another to see it working in practice. And it’s impressive!

I had the opportunity to take part in meetings with a team that applies agile methods: they’ve been doing this for years and their process is very mature. I was impressed with the discipline. The methods the team uses are agile in spirit and re-use some elements from scrum, extreme programming and lean.

They hold at least three types of meetings:

• stand-up (daily):
  • quick status update by each pair
  • decide the new pairs for the day
  • quickly go over yesterday’s annoyances (written on a board)
• planning (every iteration): give a time estimate to each piece of work in the next iteration.
• retrospective (every two iterations): list what went well, what didn’t and what needs to change.

Six programmers work in pairs and interact with two business analysts. The pairs change every day.

The business analysts identify early little chunks of functionality called stories. Stories are small enough to be designed/coded/tested in less than two days and still provide business value. The BAs prepare a list of the stories that will make it into the next iteration. After discussion with the developers they assign a rough weight to each story. A story is weighted in relation to a “standard” story, which takes about 1.5 day for a pair to complete. A story with weight 1/3 takes a third of the time of a standard story. Later the devs will give a more precise estimate to the story during the planning meeting.

The most striking aspect of the approach is that the BAs write and complete the FIT tests before each iteration. That means even before the dev pair starts coding a story, there is already a FIT test for that story. Obviously at this stage the test fails because the solution does not yet exist. Then the job of the devs is to get the FIT tests to pass by actually coding the functionality (writing the unit-tests first and using TDD’s Fail>Pass>Refactor cycle). If the devs feel a need to change something to the FIT test itself, they discuss it with the BAs.

The dev pairs change every day and the new pairs are decided during the stand-up.

More about the planning meeting: before each new iteration the team meets to discuss the stories going into the next iteration. They sit around a table and go through the stack of stories one by one. It’s literally a stack of cards with a brief description of the story on it. Because the stories are very lightweight both in terms of functionality and coding complexity, you can get away with a very short description on the card. A few lines of handwritten text are enough to express what the story is about. No heavy specs documents: fine level story granularity + lots of team communication replace the need for detailed specs. The FIT tests and unit-tests capture the detailed specifications.

The process they use is self-evolving: once a month they hold a formal meeting called a retrospective. They discuss what went well, what didn’t and then fine-tune their methods.

The structure of the retrospective is straightforward: they write three lists of items on a board:

• GOOD
• BAD
• TODO

They also bring the board with last month’s TODOs to tick off those that were actually done and carry over the rest. Everyone sits around a table with tea, coffee and biscuits. Not surprisingly that meeting can get quite animated, especially given that those guys are not exactly the shy type. You can tell they’re used to doing that by the way they structure the discussions: list the items first on the board, close the list and then discuss the items one by one rather than dwell on the first items and run out of time.

Feedly, Google Map Layers and Bing Maps

I’m an avid user of Google Reader because it’s fast and straightforward. Unfortunately the layout is a bit boring. The problem with a boring layout is that you can end up missing posts either because there is no pic to catch your attention or you fell asleep.

While I was in Istanbul I tried feedly. It works as a Chrome extension and uses your Google Reader feeds (so no need to import the feeds, they’re all there). It lays out RSS posts with a picture, a title and a preview text. Several layouts are available, digest, cover, popular and latest, all using some obscure algorithm to display a subset of the posts available in your RSS feeds. The result is visually appealing and the keyboard shortcuts are similar to those in Google Reader.

The digest layout (notice how you get some Kandinsky-like abstract painting when a picture is not available from the post):

The cover layout:

Compare that with the equivalent Google Reader layout:

Google maps for mobile

At last, the latest version of Google maps for Windows mobile is what it should have been at the beginning: the places you save in My Maps on the desktop appear on your mobile. The places you create as starred items on the phone appear on the desktop. It’s good because I create starred items on my phone whenever I walk past a restaurant I want to remember. But until this new version of Google Maps I couldn’t save them in the cloud, which was a bit daft.

So on the mobile Google maps you can now display layers. A layer can be anything: your saved maps, saved searches, maps of favorites places by famous people, your friends on latitude, tube lines… There is a Wikipedia layer that displays an icon on places related to a geolocated article and a Buzz layer that displays icons where people left a buzz message…

New Bing maps

The new bing maps available at www.bing.com/maps/explore work with Silverlight and are clearly faster than Google maps. Try bringing up London in two chrome sessions side by side, one with Google and one with Bing. It’s pretty obvious: panning, zooming in and zooming out feels faster and more fluid with Bing.

The map apps are pretty good: there is a Restaurants finder that works really well (at least for London that is) and a Twitter Map to display geolocated tweets on the current map in real time. Today’s front pages is exciting if you’re into worldwide news.

Currently the data in the new Bing lags behind… For instance the Restaurant Finder didn’t have anything to show for Dublin or Istanbul. But in terms of functionality, it’s very promising.

This is some of the available Map apps:

The Restaurants finder with category filtering:

The Ordnance Survey maps are not displayed in the new Bing maps apparently. However they still are in the normal bing maps maps.bing.com. It still blows my mind that the raster version of those 1/25000 maps (those that cost you an arm and two legs at the bookshop) are up there for free.

Comparing Result Sets with Linq

Linq is really handy. Once you’re familiar with the syntax and the way it works, it becomes very useful.

Recently I had to write a front-end to compare lists of financial instruments coming from two different sources. One source was a SQL Server database, the other source was an in-house market data system.

The result set from the database was extracted by a stored procedure into a collection. The collection’s type is IEnumerable<Result1> where Result1 is a type automatically generated by the Linq designer using the stored procedure definition.

The other result set comes from the in-house market data system. It is extracted with an API that constructs Result2 objects and accumulates them into a collection List<Result2>.

The purpose of the little app is to show the instruments present in the first result set and not the other one. The types Result1 and Result2 are defined as follows:

 

/// <summary>
/// In my real project this class was generated by the Linq designer
/// </summary>
class Result1
{
    public string Isin { get; set; }
    public string Sedol { set; get; }
}

/// <summary>
/// Manually created class to store objects from the in-house market data system
/// </summary>
class Result2
{
    public string Isin { get; set; }
    public string Sedol { set; get; }
    public string Location { set; get; }
}

Fill both collections with some dummy data:

List<Result1> coll1 = new List<Result1>();
List<Result2> coll2 = new List<Result2>();

coll1.Add(new Result1 { Isin = "ABCD", Sedol = "123" });
coll1.Add(new Result1 { Isin = "EDFG", Sedol = "234" });

coll2.Add(new Result2 { Isin = "EDFG", Sedol = "234", Location = "Set2" });

I want to compute result1 MINUS result2. In Linq this is done with Except. So I’d like to write something like this:

var diff =
    (from c in result1
     select c).
    Except
    (from c in result2
     select c);

…but that does not compile because result1 and result2 have two different types. Therefore I need to do a projection, like this:

var diff =
    (from c in result1
     select c).
    Except
    (from c in result2
     select new Result1 { Isin = c.Isin, Sedol = c.Sedol });

The above compiles fine, but there is a problem: the result of the diff is the original result1 collection. It doesn’t remove the elements present also present in result2! This is because at this stage Except does not know how to test Result1 objects for equality.

Except takes an IEqualityComparer<T> as second argument. Let’s define one:

class Result1Comparer : EqualityComparer<Result1>
{

    /// <summary>
    /// Two elements are considered equal if they match on at least one identifier
    /// </summary>
    /// <param name="x"></param>
    /// <param name="y"></param>
    /// <returns></returns>
    public override bool Equals(Result1 x, Result1 y)
    {
        return ((x.Isin == y.Isin) || (x.Sedol == y.Sedol) );
    }

    public override int GetHashCode(Result1 obj)
    {
        // Just re-use Object's default GetHashCode
        return obj.GetHashCode();
    }
}

The diff now looks like this:

var diff =
    (from c in result1
     select c).
    Except
    (from c in result2
     select new Result1 { Isin = c.Isin, Sedol = c.Sedol },
     new Type1Comparer());

It’s still not working! Why? When Except compares the result sets, it uses the comparer’s GetHashCode first. If two objects have two different hashcodes then it doesn’t bother calling Equals().

In the code above the GetHashCode() override calls the default Object.GetHashCode(). Object.GetHashCode() is useless: it generates an index as a function of the object reference. Even if two objects have the same value Object.GetHashCode()returns two different hashes. And Except thinks the objects are different!

No need to burn too much brainpower on writing the hash function as long as:

• it’s fast
• two objects with the same value return the same hashcode.

The following should do:

class Result1Comparer : EqualityComparer<Result1>
{

    /// <summary>
    /// Two elements are considered equal if they match on at least one identifier
    /// </summary>
    /// <param name="x"></param>
    /// <param name="y"></param>
    /// <returns></returns>
    public override bool Equals(Result1 x, Result1 y)
    {
        return ((x.Isin == y.Isin) || (x.Sedol == y.Sedol) );
    }

    public override int GetHashCode(Result1 obj)
    {
        // Objects with different Isins will be considered as different
        // If 2 objects have same Isin then Equals is used
        return obj.Isin.GetHashCode();
    }
}

No the MINUS operation with Except works.

Limits of Linq:

The SQL-like way of comparing collections is elegant but it’s nothing more than a nice way of writing for loops. Don’t expect miracles on the performance side…

The best place to perform queries is still in the database.

In the case of the app described above performing queries on the client is just fine. However there are situations where importing into the database and having all the queries take place in SQL Server would be faster.

Books I currently Flip through #8

Tech

• LINQ to Objects Using C# 4.0: Using and Extending LINQ to Objects and Parallel LINQ (PLINQ)

Others

• The Cul-de-Sac Syndrome: Turning Around the Unsustainable American Dream (Bloomberg) (John F. Wasik)  

• Outliers (Malcolm Gladwell)
  

Udi Dahan about Simplifying Multi-Tier Architecture

I listened to Udi Dahan talk about an alternative pattern to the classic multi-tier architecture during a Skills Matter event in London.

I knew the name was familiar but I couldn’t put my finger on it until I entered the room: he was at TechEd Barcelona in November 2008! Back then he gave a very sarcastic (and enjoyable) talk about interface-based programming, dependency injection and the abuses of the strategy pattern.

Udi is a good speaker. That’s quite a treat to have a TechEd-level presentation for free at Skills Matter.

The pattern is called CQRS. Despite the boring 4-letter acronym it is based on a few very simple ideas: take out from the standard UI/Services/Business logic/DAL everything that is related to queries (= pull data off the database to show it to the user) and leave only the commands (= use business rules to change data). Queries can simply use a 2-tier model with persistent views.

The video of the talk is available on Skills Matter website.

Command Query Responsibility Segregation

View more presentations from Skills Matter.

Presentation notes: (those are Udi’s ideas rephrased with my own words…)

Queries

• Data is always stale, but why not show how stale it is with a timestamp? Doesn’t cost anything.
• Representing data with objects is a bit of a waste because as far as queries go, data is not an object, it has no behaviour, it’s just data!
• Consequence: go back to the simplest thing that works, the 2-tier architecture. 
• The UI can talk directly to the DB since layers don’t add any value to queries. For each view in the UI there is one persistent view in the DB. The UI does a simple select *, no calculation. The persistent views have a column for every piece of data to be shown in the UI view.
• Data duplication between persistent views is fine. Entities are covered separately by an OLTP model.

Search

• Avoid generic search screens that allow users to compose an ad-hoc query with plenty of fields. Focus on the user’s intent only: design the most likely queries to be used and make them available by default with very limited amount of parameters.
• For anything that’s not covered by a pre-designed query, use a very simple google-style 1-field full text search. You’d need a data model dedicated to this kind of search (SQL Server full-text search service for instance).

Persistent view model

• The views don’t need foreign keys, but they do need indexes.

Security

• Use the database for role-based access: user, superviser, each one has its own view with its own set of columns. Data is duplicated but that’s ok.
• That’s actually more secure than a memory-based web cache.

Validation

• Use the persistent view model to do preliminary validation (as the user types his name for instance). No need to wait until submission.
• Validation is done within components identical on the client and the server.
• Examples of things that can be checked early: uniqueness, related entity existence…
• Validation result in correct in 99% of cases, which is good enough for preliminary validation. The purpose is to answer the question: does the data have chances to be good?
• Validation does not have to be mixed with business rules: ranges, lengths, etc…

Commands

• Commands encapsulate the business rules.
• Rules answer the question: should we do this based on what’s currently in the DB?
• Commands can use the usual UI/Services/Business Logic/DAL layers.
• Because commands are successful most of the time you can get away with giving an immediate positive answer to the user. Notify him asynchronously if something goes wrong (per email for instance). “Thank you, we’ll let you know if there is a problem”.

User interface design

Tips to capture the user’s intent in the UI while taking advantage of the preliminary validation on the client:

• a grid should be able to accept a column and reject another. There is not necessarily a transaction at the row level. Changing a user’s status is a different task from changing a shipping address and can be done separately.
• a reservation system should allow people to book blocks of seats rather than forcing the user to tick seats individually until the finds an available block. The system could even look for an available block and notify the user when one is found. A good command ends with: “Thanks, you’ll get a confirmation soon”.
• posting a comment to blog. The best way to confirm is to display the user’s comment in the page. You can do that on the client directly, no need to wait until the server updated the page.
• cash machine: you can do a preliminary validation too (credit card valid, user authenticated, yesterday’s balance good…). If the balance was not good and the user is overdrawn, no big deal, it makes money to the bank anyway (just make sure there is a clause in the credit card’s small print).

Domain Models

• They are not for validation (use components instead), not for queries (remove all objects used only to pull data and use a persistent view model instead).
• Keep the domain model for the interesting stuff: the business logic. Remove from the domain model everything (tables or columns) that’s not directly used by business rules.

Coding in Istanbul 3: Pair Coding with a Friend is Fun

Helping Jeff with his PHP work was a chance to practice some pair coding. I don’t get to do this at work!

I know Jeff since the engineering school. We graduated from the same place in Paris. At the time (about 14 years ago) we were always coding in pairs: nothing to do with agile, we simply had to! The school labs were equipped with one PC/Solaris station for two people at best. We often had to finish projects late at night. Eating pizza at someone’s place, three or four people on the same computer. Of course we couldn’t afford laptops. I even remember having to carry around my tower PC to a mate’s flat in order to finish a late project! Those were tough times… and pairing with people was natural: it didn’t look like an exotic feature from a new methodology. It was just what all students did. 

During our excursions in the coffee-shops and restaurants of Istanbul we both have our netbook. We tried phases where we worked separately, Jeff coding and me looking up some info. But what turns out to work best is to code on the same netbook taking turns at the keyboard. Less chances to get distracted this way. If one loses focus, the other puts him back on track. If one does a typo the other spots it immediately. If we both loose focus we just order another Latte.

We can sustain the pace for about two hours before we become useless.

Quite a good laugh actually…  Just like good old times! Too bad we don’t do that at work…

Coding in Istanbul 2: Reverse Engineering a LAMP Project

We had a salad at Belvu restaurant, Fenerbahçe then carried on coding in True Blue, a cafe overlooking the Sea of Marmara with a nice view to the Princes Islands.

The stack of Jeff’s site is based on PHP/MySql/Apache. I’m more used to the world of C#/ADO.NET/Linq/T-SQL/SQL Server.

It’s funny how web development differs from enterprise apps!

I got into the habit of encapsulating everything and separating layers but the approach used by some PHP devs is more pragmatic. Business logic, data retrieval, building SQL queries and presentation: everything is done in the same .PHP file!

In Windows client development you typically let a data layer retrieve some data objects (via ADO.NET, Linq or something else) then you assign the data object to a grid control. The control takes care of displaying the content of the object, dynamically creating columns and adding rows… In the PHP files I saw the data layer bit (building the SQL, performing the query, retrieving data) appears just a few lines before a while loop that echoes  <td> tags to generate a table for each row of the result set. All layers in the same place.

Also in the code I saw SQL queries were always built on the fly. The first downside of building SQL queries with string concatenation is vulnerability to SQL injection (followed by performance loss due to always rebuilding the query plan and no transactional integrity)… The devs protected the code from injections by using specific PHP functions that parse all user inputs. Why not use stored procedures? Apparently stored procedures are a new thing in MySql: they arrived only in version 5. So I guess they got into the habit of doing away without them.

Regarding the development tools, when working with PHP in Dreamweaver -apart from a bit of color coding- you get absolutely ZERO compiler support. Indentation is manual and there is no intellisense. Comes pretty close to handcarving on stone tablets.

Another funny thing with MySQL: it comes with a large choice of database engines, each one having its pros and cons. It turns out that the default db engine in MySQL is optimised for speed and does not support foreign keys. So by default you work in a system where you can delete parents without deleting childrens and where children can contain a reference to a non existing parent. You’d better be sure that’s what you want, most of the time it’s not…