Insight Into an Agile Team

 

It’s one thing to read about agile in books. It’s another to see it working in practice. And it’s impressive!

I had the opportunity to take part in meetings with a team that applies agile methods: they’ve been doing this for years and their process is very mature. I was impressed with the discipline. The methods the team uses are agile in spirit and re-use some elements from scrum, extreme programming and lean.

They hold at least three types of meetings:

• stand-up (daily):
  • quick status update by each pair
  • decide the new pairs for the day
  • quickly go over yesterday’s annoyances (written on a board)
• planning (every iteration): give a time estimate to each piece of work in the next iteration.
• retrospective (every two iterations): list what went well, what didn’t and what needs to change.

Six programmers work in pairs and interact with two business analysts. The pairs change every day.

The business analysts identify early little chunks of functionality called stories. Stories are small enough to be designed/coded/tested in less than two days and still provide business value. The BAs prepare a list of the stories that will make it into the next iteration. After discussion with the developers they assign a rough weight to each story. A story is weighted in relation to a “standard” story, which takes about 1.5 day for a pair to complete. A story with weight 1/3 takes a third of the time of a standard story. Later the devs will give a more precise estimate to the story during the planning meeting.

The most striking aspect of the approach is that the BAs write and complete the FIT tests before each iteration. That means even before the dev pair starts coding a story, there is already a FIT test for that story. Obviously at this stage the test fails because the solution does not yet exist. Then the job of the devs is to get the FIT tests to pass by actually coding the functionality (writing the unit-tests first and using TDD’s Fail>Pass>Refactor cycle). If the devs feel a need to change something to the FIT test itself, they discuss it with the BAs.

The dev pairs change every day and the new pairs are decided during the stand-up.

More about the planning meeting: before each new iteration the team meets to discuss the stories going into the next iteration. They sit around a table and go through the stack of stories one by one. It’s literally a stack of cards with a brief description of the story on it. Because the stories are very lightweight both in terms of functionality and coding complexity, you can get away with a very short description on the card. A few lines of handwritten text are enough to express what the story is about. No heavy specs documents: fine level story granularity + lots of team communication replace the need for detailed specs. The FIT tests and unit-tests capture the detailed specifications.

The process they use is self-evolving: once a month they hold a formal meeting called a retrospective. They discuss what went well, what didn’t and then fine-tune their methods.

The structure of the retrospective is straightforward: they write three lists of items on a board:

• GOOD
• BAD
• TODO

They also bring the board with last month’s TODOs to tick off those that were actually done and carry over the rest. Everyone sits around a table with tea, coffee and biscuits. Not surprisingly that meeting can get quite animated, especially given that those guys are not exactly the shy type. You can tell they’re used to doing that by the way they structure the discussions: list the items first on the board, close the list and then discuss the items one by one rather than dwell on the first items and run out of time.

Feedly, Google Map Layers and Bing Maps

I’m an avid user of Google Reader because it’s fast and straightforward. Unfortunately the layout is a bit boring. The problem with a boring layout is that you can end up missing posts either because there is no pic to catch your attention or you fell asleep.

While I was in Istanbul I tried feedly. It works as a Chrome extension and uses your Google Reader feeds (so no need to import the feeds, they’re all there). It lays out RSS posts with a picture, a title and a preview text. Several layouts are available, digest, cover, popular and latest, all using some obscure algorithm to display a subset of the posts available in your RSS feeds. The result is visually appealing and the keyboard shortcuts are similar to those in Google Reader.

The digest layout (notice how you get some Kandinsky-like abstract painting when a picture is not available from the post):

The cover layout:

Compare that with the equivalent Google Reader layout:

Google maps for mobile

At last, the latest version of Google maps for Windows mobile is what it should have been at the beginning: the places you save in My Maps on the desktop appear on your mobile. The places you create as starred items on the phone appear on the desktop. It’s good because I create starred items on my phone whenever I walk past a restaurant I want to remember. But until this new version of Google Maps I couldn’t save them in the cloud, which was a bit daft.

So on the mobile Google maps you can now display layers. A layer can be anything: your saved maps, saved searches, maps of favorites places by famous people, your friends on latitude, tube lines… There is a Wikipedia layer that displays an icon on places related to a geolocated article and a Buzz layer that displays icons where people left a buzz message…

New Bing maps

The new bing maps available at www.bing.com/maps/explore work with Silverlight and are clearly faster than Google maps. Try bringing up London in two chrome sessions side by side, one with Google and one with Bing. It’s pretty obvious: panning, zooming in and zooming out feels faster and more fluid with Bing.

The map apps are pretty good: there is a Restaurants finder that works really well (at least for London that is) and a Twitter Map to display geolocated tweets on the current map in real time. Today’s front pages is exciting if you’re into worldwide news.

Currently the data in the new Bing lags behind… For instance the Restaurant Finder didn’t have anything to show for Dublin or Istanbul. But in terms of functionality, it’s very promising.

This is some of the available Map apps:

The Restaurants finder with category filtering:

The Ordnance Survey maps are not displayed in the new Bing maps apparently. However they still are in the normal bing maps maps.bing.com. It still blows my mind that the raster version of those 1/25000 maps (those that cost you an arm and two legs at the bookshop) are up there for free.

Comparing Result Sets with Linq

Linq is really handy. Once you’re familiar with the syntax and the way it works, it becomes very useful.

Recently I had to write a front-end to compare lists of financial instruments coming from two different sources. One source was a SQL Server database, the other source was an in-house market data system.

The result set from the database was extracted by a stored procedure into a collection. The collection’s type is IEnumerable<Result1> where Result1 is a type automatically generated by the Linq designer using the stored procedure definition.

The other result set comes from the in-house market data system. It is extracted with an API that constructs Result2 objects and accumulates them into a collection List<Result2>.

The purpose of the little app is to show the instruments present in the first result set and not the other one. The types Result1 and Result2 are defined as follows:

 

/// <summary>
/// In my real project this class was generated by the Linq designer
/// </summary>
class Result1
{
    public string Isin { get; set; }
    public string Sedol { set; get; }
}

/// <summary>
/// Manually created class to store objects from the in-house market data system
/// </summary>
class Result2
{
    public string Isin { get; set; }
    public string Sedol { set; get; }
    public string Location { set; get; }
}

Fill both collections with some dummy data:

List<Result1> coll1 = new List<Result1>();
List<Result2> coll2 = new List<Result2>();

coll1.Add(new Result1 { Isin = "ABCD", Sedol = "123" });
coll1.Add(new Result1 { Isin = "EDFG", Sedol = "234" });

coll2.Add(new Result2 { Isin = "EDFG", Sedol = "234", Location = "Set2" });

I want to compute result1 MINUS result2. In Linq this is done with Except. So I’d like to write something like this:

var diff =
    (from c in result1
     select c).
    Except
    (from c in result2
     select c);

…but that does not compile because result1 and result2 have two different types. Therefore I need to do a projection, like this:

var diff =
    (from c in result1
     select c).
    Except
    (from c in result2
     select new Result1 { Isin = c.Isin, Sedol = c.Sedol });

The above compiles fine, but there is a problem: the result of the diff is the original result1 collection. It doesn’t remove the elements present also present in result2! This is because at this stage Except does not know how to test Result1 objects for equality.

Except takes an IEqualityComparer<T> as second argument. Let’s define one:

class Result1Comparer : EqualityComparer<Result1>
{

    /// <summary>
    /// Two elements are considered equal if they match on at least one identifier
    /// </summary>
    /// <param name="x"></param>
    /// <param name="y"></param>
    /// <returns></returns>
    public override bool Equals(Result1 x, Result1 y)
    {
        return ((x.Isin == y.Isin) || (x.Sedol == y.Sedol) );
    }

    public override int GetHashCode(Result1 obj)
    {
        // Just re-use Object's default GetHashCode
        return obj.GetHashCode();
    }
}

The diff now looks like this:

var diff =
    (from c in result1
     select c).
    Except
    (from c in result2
     select new Result1 { Isin = c.Isin, Sedol = c.Sedol },
     new Type1Comparer());

It’s still not working! Why? When Except compares the result sets, it uses the comparer’s GetHashCode first. If two objects have two different hashcodes then it doesn’t bother calling Equals().

In the code above the GetHashCode() override calls the default Object.GetHashCode(). Object.GetHashCode() is useless: it generates an index as a function of the object reference. Even if two objects have the same value Object.GetHashCode()returns two different hashes. And Except thinks the objects are different!

No need to burn too much brainpower on writing the hash function as long as:

• it’s fast
• two objects with the same value return the same hashcode.

The following should do:

class Result1Comparer : EqualityComparer<Result1>
{

    /// <summary>
    /// Two elements are considered equal if they match on at least one identifier
    /// </summary>
    /// <param name="x"></param>
    /// <param name="y"></param>
    /// <returns></returns>
    public override bool Equals(Result1 x, Result1 y)
    {
        return ((x.Isin == y.Isin) || (x.Sedol == y.Sedol) );
    }

    public override int GetHashCode(Result1 obj)
    {
        // Objects with different Isins will be considered as different
        // If 2 objects have same Isin then Equals is used
        return obj.Isin.GetHashCode();
    }
}

No the MINUS operation with Except works.

Limits of Linq:

The SQL-like way of comparing collections is elegant but it’s nothing more than a nice way of writing for loops. Don’t expect miracles on the performance side…

The best place to perform queries is still in the database.

In the case of the app described above performing queries on the client is just fine. However there are situations where importing into the database and having all the queries take place in SQL Server would be faster.

Books I currently Flip through #8

Tech

• LINQ to Objects Using C# 4.0: Using and Extending LINQ to Objects and Parallel LINQ (PLINQ)

Others

• The Cul-de-Sac Syndrome: Turning Around the Unsustainable American Dream (Bloomberg) (John F. Wasik)  

• Outliers (Malcolm Gladwell)
  

Udi Dahan about Simplifying Multi-Tier Architecture

I listened to Udi Dahan talk about an alternative pattern to the classic multi-tier architecture during a Skills Matter event in London.

I knew the name was familiar but I couldn’t put my finger on it until I entered the room: he was at TechEd Barcelona in November 2008! Back then he gave a very sarcastic (and enjoyable) talk about interface-based programming, dependency injection and the abuses of the strategy pattern.

Udi is a good speaker. That’s quite a treat to have a TechEd-level presentation for free at Skills Matter.

The pattern is called CQRS. Despite the boring 4-letter acronym it is based on a few very simple ideas: take out from the standard UI/Services/Business logic/DAL everything that is related to queries (= pull data off the database to show it to the user) and leave only the commands (= use business rules to change data). Queries can simply use a 2-tier model with persistent views.

The video of the talk is available on Skills Matter website.

Command Query Responsibility Segregation

View more presentations from Skills Matter.

Presentation notes: (those are Udi’s ideas rephrased with my own words…)

Queries

• Data is always stale, but why not show how stale it is with a timestamp? Doesn’t cost anything.
• Representing data with objects is a bit of a waste because as far as queries go, data is not an object, it has no behaviour, it’s just data!
• Consequence: go back to the simplest thing that works, the 2-tier architecture. 
• The UI can talk directly to the DB since layers don’t add any value to queries. For each view in the UI there is one persistent view in the DB. The UI does a simple select *, no calculation. The persistent views have a column for every piece of data to be shown in the UI view.
• Data duplication between persistent views is fine. Entities are covered separately by an OLTP model.

Search

• Avoid generic search screens that allow users to compose an ad-hoc query with plenty of fields. Focus on the user’s intent only: design the most likely queries to be used and make them available by default with very limited amount of parameters.
• For anything that’s not covered by a pre-designed query, use a very simple google-style 1-field full text search. You’d need a data model dedicated to this kind of search (SQL Server full-text search service for instance).

Persistent view model

• The views don’t need foreign keys, but they do need indexes.

Security

• Use the database for role-based access: user, superviser, each one has its own view with its own set of columns. Data is duplicated but that’s ok.
• That’s actually more secure than a memory-based web cache.

Validation

• Use the persistent view model to do preliminary validation (as the user types his name for instance). No need to wait until submission.
• Validation is done within components identical on the client and the server.
• Examples of things that can be checked early: uniqueness, related entity existence…
• Validation result in correct in 99% of cases, which is good enough for preliminary validation. The purpose is to answer the question: does the data have chances to be good?
• Validation does not have to be mixed with business rules: ranges, lengths, etc…

Commands

• Commands encapsulate the business rules.
• Rules answer the question: should we do this based on what’s currently in the DB?
• Commands can use the usual UI/Services/Business Logic/DAL layers.
• Because commands are successful most of the time you can get away with giving an immediate positive answer to the user. Notify him asynchronously if something goes wrong (per email for instance). “Thank you, we’ll let you know if there is a problem”.

User interface design

Tips to capture the user’s intent in the UI while taking advantage of the preliminary validation on the client:

• a grid should be able to accept a column and reject another. There is not necessarily a transaction at the row level. Changing a user’s status is a different task from changing a shipping address and can be done separately.
• a reservation system should allow people to book blocks of seats rather than forcing the user to tick seats individually until the finds an available block. The system could even look for an available block and notify the user when one is found. A good command ends with: “Thanks, you’ll get a confirmation soon”.
• posting a comment to blog. The best way to confirm is to display the user’s comment in the page. You can do that on the client directly, no need to wait until the server updated the page.
• cash machine: you can do a preliminary validation too (credit card valid, user authenticated, yesterday’s balance good…). If the balance was not good and the user is overdrawn, no big deal, it makes money to the bank anyway (just make sure there is a clause in the credit card’s small print).

Domain Models

• They are not for validation (use components instead), not for queries (remove all objects used only to pull data and use a persistent view model instead).
• Keep the domain model for the interesting stuff: the business logic. Remove from the domain model everything (tables or columns) that’s not directly used by business rules.

Coding in Istanbul 3: Pair Coding with a Friend is Fun

Helping Jeff with his PHP work was a chance to practice some pair coding. I don’t get to do this at work!

I know Jeff since the engineering school. We graduated from the same place in Paris. At the time (about 14 years ago) we were always coding in pairs: nothing to do with agile, we simply had to! The school labs were equipped with one PC/Solaris station for two people at best. We often had to finish projects late at night. Eating pizza at someone’s place, three or four people on the same computer. Of course we couldn’t afford laptops. I even remember having to carry around my tower PC to a mate’s flat in order to finish a late project! Those were tough times… and pairing with people was natural: it didn’t look like an exotic feature from a new methodology. It was just what all students did. 

During our excursions in the coffee-shops and restaurants of Istanbul we both have our netbook. We tried phases where we worked separately, Jeff coding and me looking up some info. But what turns out to work best is to code on the same netbook taking turns at the keyboard. Less chances to get distracted this way. If one loses focus, the other puts him back on track. If one does a typo the other spots it immediately. If we both loose focus we just order another Latte.

We can sustain the pace for about two hours before we become useless.

Quite a good laugh actually…  Just like good old times! Too bad we don’t do that at work…

Coding in Istanbul 2: Reverse Engineering a LAMP Project

We had a salad at Belvu restaurant, Fenerbahçe then carried on coding in True Blue, a cafe overlooking the Sea of Marmara with a nice view to the Princes Islands.

The stack of Jeff’s site is based on PHP/MySql/Apache. I’m more used to the world of C#/ADO.NET/Linq/T-SQL/SQL Server.

It’s funny how web development differs from enterprise apps!

I got into the habit of encapsulating everything and separating layers but the approach used by some PHP devs is more pragmatic. Business logic, data retrieval, building SQL queries and presentation: everything is done in the same .PHP file!

In Windows client development you typically let a data layer retrieve some data objects (via ADO.NET, Linq or something else) then you assign the data object to a grid control. The control takes care of displaying the content of the object, dynamically creating columns and adding rows… In the PHP files I saw the data layer bit (building the SQL, performing the query, retrieving data) appears just a few lines before a while loop that echoes  <td> tags to generate a table for each row of the result set. All layers in the same place.

Also in the code I saw SQL queries were always built on the fly. The first downside of building SQL queries with string concatenation is vulnerability to SQL injection (followed by performance loss due to always rebuilding the query plan and no transactional integrity)… The devs protected the code from injections by using specific PHP functions that parse all user inputs. Why not use stored procedures? Apparently stored procedures are a new thing in MySql: they arrived only in version 5. So I guess they got into the habit of doing away without them.

Regarding the development tools, when working with PHP in Dreamweaver -apart from a bit of color coding- you get absolutely ZERO compiler support. Indentation is manual and there is no intellisense. Comes pretty close to handcarving on stone tablets.

Another funny thing with MySQL: it comes with a large choice of database engines, each one having its pros and cons. It turns out that the default db engine in MySQL is optimised for speed and does not support foreign keys. So by default you work in a system where you can delete parents without deleting childrens and where children can contain a reference to a non existing parent. You’d better be sure that’s what you want, most of the time it’s not…

Coding in Istanbul 1: PHP in Turkey

I’m having a çoban kavurma in a restaurant with my old mate Jeff who lives in Istanbul. We stopped for lunch in one of the busy pedestrian streets of Kadiköy on the Asian side of the city.

We spent most of yesterday coding on Jeff’s doctors appointment website. We added a small admin functionality that will save him time when mailing doctors. He’s currently working on getting more doctors registered: it involves finding their details and emailing them one by one. Obviously spamming them all in one go is out of the question so each email is slightly personalised (quoting the doctor’s own website). Yet they all follow the same template so there is some room for automation.

Jeff also needs to track how doctors open and view the campaign emails. Each email contains a number of links: to the appointment’s website, press articles, facebook and to twitter. Pretty clever: each link calls a php on his website that logs the doctor’s identifier along with the identifier of the link. Then he can run a report that shows various stats about who clicked what link…

Good places to take the netbooks out in Turkey include a growing number of restaurants, Starbucks, Gloria Jeans… and Wifi is almost always free!

Gloria Jeans is a great place to sit down with the laptop: more comfortable than Starbucks and less crowded. The one in Ortaköy is perfect: beautiful view to the Bosphorus, 3 levels, smoothies, cheesecakes, tosts (paninis) and free wifi.

View Istanbul in a larger map

Setup, TearDown, When to Mock?

 

I’m writing this while eating at the Launceston Place, a very comfy restaurant near Gloucester Road. Warm decor, soft lighting and excellent service.

• Setup and TearDown

Only use those to respectively initialise then destroy mock objects. Methods with [Setup] and [TearDown] attributes are called before and after each test so they’re the only methods that guaranty that things get cleaned up.

Best to avoid the [SetupFixture] attribute (to mark a class that contains Setup and Teardown methods called just once before ALL tests), as well as [TestFixtureSetup]/[TestFixtureTeardown] (called once only before all tests in a fixture). I tried to be clever and used them and ended up having to debug nasty side effects. Unit tests are supposed to run in isolation, so just one [Setup] before and one [TearDown] after each of them, that’s it.

• What should be mocked?

Yesterday while writing a UT I bumped into this question: should I bother writing a mock for that object or can’t I just use the real thing?

One answer: objects that access external systems (DB, hard-drive, client API to some server-based system, …) should always be mocked (for unit-tests that is, not integration tests), otherwise the test is not repeatable, and it’s slow…

If a method under test accesses an injected object and this object does only in-memory stuff, then no need to mock it. It would make sense to mock it because after all this injected object is not actually the thing you want to test… Still, the more real objects are covered by the test, the better. As long as they all run in-memory and do not prevent the test from being repeatable, all is well. So no need to waste time writing a mock in that case.

Got to Treat Yourself

I bought myself a Christmas present. Next week I’ll receive this neat little thing:

It’s a VAIO X-Series.

This is a replacement for the VGN-SZ5 that I carried around every day in a rucksack for more than two years. Now the rucksack will disappear to be replaced by a tiny messenger bag.

Premium carbon finish, SSD drive, embedded 3G. Extremely thin and less than 900g. This is going to be pure pleasure.

In 2007 netbooks were not around yet and I needed something to write code on the go, not just in my lounge. I got the VGN-SZ5 and a Wenger SwissGear rucksack, both of which I’m still using today. I went everywhere with them: Hertfordshire, TechEd Barcelona, Egypt, the French Pyrenees (picture), Paris, Istanbul… In London I use the Vaio in the tube, in Starbucks, Costa, in pubs, in restaurants (which is extremely annoying to people I go out with).

Last year the rucksack felt a bit heavy on my shoulders so I started looking around for a netbook. I read reviews and benchmarks in magasines, tried out all sorts of models in the shop… Samsung, Asus Eee, Acer, Lenovo, MSI Wind… It wasn’t quite that. Either the 3G card was not embedded, or the thing was too bulky… Not sexy enough.

The first time I saw the X-Series in a Sony shop (in October last year two days before the official UK launch), I immediately thought “I want that”. I gave myself some time to think it over (it’s a bit of an investment). In its October 2009 review, PC Pro said:

But the real problem for the X-Series is its price. (…) Who, though, can lavish £1,130 exc VAT on a vanity laptop such as this?

I just ordered one.