Pages

Tuesday, 30 March 2010

Comparing Result Sets with Linq

Linq is really handy. Once you’re familiar with the syntax and the way it works, it becomes very useful.

Recently I had to write a front-end to compare lists of financial instruments coming from two different sources. One source was a SQL Server database, the other source was an in-house market data system.

The result set from the database was extracted by a stored procedure into a collection. The collection’s type is IEnumerable<Result1> where Result1 is a type automatically generated by the Linq designer using the stored procedure definition.

The other result set comes from the in-house market data system. It is extracted with an API that constructs Result2 objects and accumulates them into a collection List<Result2>.

The purpose of the little app is to show the instruments present in the first result set and not the other one. The types Result1 and Result2 are defined as follows:

 

/// <summary>
/// In my real project this class was generated by the Linq designer
/// </summary>
class Result1
{
    public string Isin { get; set; }
    public string Sedol { set; get; }
}

/// <summary>
/// Manually created class to store objects from the in-house market data system
/// </summary>
class Result2
{
    public string Isin { get; set; }
    public string Sedol { set; get; }
    public string Location { set; get; }
}

Fill both collections with some dummy data:

List<Result1> coll1 = new List<Result1>();
List<Result2> coll2 = new List<Result2>();

coll1.Add(new Result1 { Isin = "ABCD", Sedol = "123" });
coll1.Add(new Result1 { Isin = "EDFG", Sedol = "234" });

coll2.Add(new Result2 { Isin = "EDFG", Sedol = "234", Location = "Set2" });

I want to compute result1 MINUS result2. In Linq this is done with Except. So I’d like to write something like this:

var diff =
    (from c in result1
     select c).
    Except
    (from c in result2
     select c);

…but that does not compile because result1 and result2 have two different types. Therefore I need to do a projection, like this:

var diff =
    (from c in result1
     select c).
    Except
    (from c in result2
     select new Result1 { Isin = c.Isin, Sedol = c.Sedol });

The above compiles fine, but there is a problem: the result of the diff is the original result1 collection. It doesn’t remove the elements present also present in result2! This is because at this stage Except does not know how to test Result1 objects for equality.

Except takes an IEqualityComparer<T> as second argument. Let’s define one:

class Result1Comparer : EqualityComparer<Result1>
{

    /// <summary>
    /// Two elements are considered equal if they match on at least one identifier
    /// </summary>
    /// <param name="x"></param>
    /// <param name="y"></param>
    /// <returns></returns>
    public override bool Equals(Result1 x, Result1 y)
    {
        return ((x.Isin == y.Isin) || (x.Sedol == y.Sedol) );
    }

    public override int GetHashCode(Result1 obj)
    {
        // Just re-use Object's default GetHashCode
        return obj.GetHashCode();
    }
}

The diff now looks like this:

var diff =
    (from c in result1
     select c).
    Except
    (from c in result2
     select new Result1 { Isin = c.Isin, Sedol = c.Sedol },
     new Type1Comparer());

It’s still not working! Why? When Except compares the result sets, it uses the comparer’s GetHashCode first. If two objects have two different hashcodes then it doesn’t bother calling Equals().

In the code above the GetHashCode() override calls the default Object.GetHashCode(). Object.GetHashCode() is useless: it generates an index as a function of the object reference. Even if two objects have the same value Object.GetHashCode()returns two different hashes. And Except thinks the objects are different!

No need to burn too much brainpower on writing the hash function as long as:

  • it’s fast
  • two objects with the same value return the same hashcode.

The following should do:

class Result1Comparer : EqualityComparer<Result1>
{

    /// <summary>
    /// Two elements are considered equal if they match on at least one identifier
    /// </summary>
    /// <param name="x"></param>
    /// <param name="y"></param>
    /// <returns></returns>
    public override bool Equals(Result1 x, Result1 y)
    {
        return ((x.Isin == y.Isin) || (x.Sedol == y.Sedol) );
    }

    public override int GetHashCode(Result1 obj)
    {
        // Objects with different Isins will be considered as different
        // If 2 objects have same Isin then Equals is used
        return obj.Isin.GetHashCode();
    }
}

No the MINUS operation with Except works.

Limits of Linq:

The SQL-like way of comparing collections is elegant but it’s nothing more than a nice way of writing for loops. Don’t expect miracles on the performance side…

The best place to perform queries is still in the database.

In the case of the app described above performing queries on the client is just fine. However there are situations where importing into the database and having all the queries take place in SQL Server would be faster.

Monday, 8 March 2010

Books I currently Flip through #8

Tech

Others

  • The Cul-de-Sac Syndrome: Turning Around the Unsustainable American Dream (Bloomberg) (John F. Wasik)  

51mWGBwpq L._SL160_

  • Outliers (Malcolm Gladwell)

image

Tuesday, 16 February 2010

Udi Dahan about Simplifying Multi-Tier Architecture

I listened to Udi Dahan talk about an alternative pattern to the classic multi-tier architecture during a Skills Matter event in London.

I knew the name was familiar but I couldn’t put my finger on it until I entered the room: he was at TechEd Barcelona in November 2008! Back then he gave a very sarcastic (and enjoyable) talk about interface-based programming, dependency injection and the abuses of the strategy pattern.

Udi is a good speaker. That’s quite a treat to have a TechEd-level presentation for free at Skills Matter.

The pattern is called CQRS. Despite the boring 4-letter acronym it is based on a few very simple ideas: take out from the standard UI/Services/Business logic/DAL everything that is related to queries (= pull data off the database to show it to the user) and leave only the commands (= use business rules to change data). Queries can simply use a 2-tier model with persistent views.

The video of the talk is available on Skills Matter website.

Presentation notes: (those are Udi’s ideas rephrased with my own words…)

Queries

  • Data is always stale, but why not show how stale it is with a timestamp? Doesn’t cost anything.
  • Representing data with objects is a bit of a waste because as far as queries go, data is not an object, it has no behaviour, it’s just data!
  • Consequence: go back to the simplest thing that works, the 2-tier architecture. 
  • The UI can talk directly to the DB since layers don’t add any value to queries. For each view in the UI there is one persistent view in the DB. The UI does a simple select *, no calculation. The persistent views have a column for every piece of data to be shown in the UI view.
  • Data duplication between persistent views is fine. Entities are covered separately by an OLTP model.

Search

  • Avoid generic search screens that allow users to compose an ad-hoc query with plenty of fields. Focus on the user’s intent only: design the most likely queries to be used and make them available by default with very limited amount of parameters.
  • For anything that’s not covered by a pre-designed query, use a very simple google-style 1-field full text search. You’d need a data model dedicated to this kind of search (SQL Server full-text search service for instance).

Persistent view model

  • The views don’t need foreign keys, but they do need indexes.

Security

  • Use the database for role-based access: user, superviser, each one has its own view with its own set of columns. Data is duplicated but that’s ok.
  • That’s actually more secure than a memory-based web cache.

Validation

  • Use the persistent view model to do preliminary validation (as the user types his name for instance). No need to wait until submission.
  • Validation is done within components identical on the client and the server.
  • Examples of things that can be checked early: uniqueness, related entity existence…
  • Validation result in correct in 99% of cases, which is good enough for preliminary validation. The purpose is to answer the question: does the data have chances to be good?
  • Validation does not have to be mixed with business rules: ranges, lengths, etc…

Commands

  • Commands encapsulate the business rules.
  • Rules answer the question: should we do this based on what’s currently in the DB?
  • Commands can use the usual UI/Services/Business Logic/DAL layers.
  • Because commands are successful most of the time you can get away with giving an immediate positive answer to the user. Notify him asynchronously if something goes wrong (per email for instance). “Thank you, we’ll let you know if there is a problem”.

User interface design

Tips to capture the user’s intent in the UI while taking advantage of the preliminary validation on the client:

  • a grid should be able to accept a column and reject another. There is not necessarily a transaction at the row level. Changing a user’s status is a different task from changing a shipping address and can be done separately.
  • a reservation system should allow people to book blocks of seats rather than forcing the user to tick seats individually until the finds an available block. The system could even look for an available block and notify the user when one is found. A good command ends with: “Thanks, you’ll get a confirmation soon”.
  • posting a comment to blog. The best way to confirm is to display the user’s comment in the page. You can do that on the client directly, no need to wait until the server updated the page.
  • cash machine: you can do a preliminary validation too (credit card valid, user authenticated, yesterday’s balance good…). If the balance was not good and the user is overdrawn, no big deal, it makes money to the bank anyway (just make sure there is a clause in the credit card’s small print).

Domain Models

  • They are not for validation (use components instead), not for queries (remove all objects used only to pull data and use a persistent view model instead).
  • Keep the domain model for the interesting stuff: the business logic. Remove from the domain model everything (tables or columns) that’s not directly used by business rules.

Sunday, 7 February 2010

Coding in Istanbul 3: Pair Coding with a Friend is Fun

Helping Jeff with his PHP work was a chance to practice some pair coding. I don’t get to do this at work!

I know Jeff since the engineering school. We graduated from the same place in Paris. At the time (about 14 years ago) we were always coding in pairs: nothing to do with agile, we simply had to! The school labs were equipped with one PC/Solaris station for two people at best. We often had to finish projects late at night. Eating pizza at someone’s place, three or four people on the same computer. Of course we couldn’t afford laptops. I even remember having to carry around my tower PC to a mate’s flat in order to finish a late project! Those were tough times… and pairing with people was natural: it didn’t look like an exotic feature from a new methodology. It was just what all students did. 

During our excursions in the coffee-shops and restaurants of Istanbul we both have our netbook. We tried phases where we worked separately, Jeff coding and me looking up some info. But what turns out to work best is to code on the same netbook taking turns at the keyboard. Less chances to get distracted this way. If one loses focus, the other puts him back on track. If one does a typo the other spots it immediately. If we both loose focus we just order another Latte.

We can sustain the pace for about two hours before we become useless.

Quite a good laugh actually…  Just like good old times! Too bad we don’t do that at work…

Tuesday, 2 February 2010

Coding in Istanbul 2: Reverse Engineering a LAMP Project

DSC_9879 (640x426)

We had a salad at Belvu restaurant, Fenerbahçe then carried on coding in True Blue, a cafe overlooking the Sea of Marmara with a nice view to the Princes Islands.

The stack of Jeff’s site is based on PHP/MySql/Apache. I’m more used to the world of C#/ADO.NET/Linq/T-SQL/SQL Server.

It’s funny how web development differs from enterprise apps!

I got into the habit of encapsulating everything and separating layers but the approach used by some PHP devs is more pragmatic. Business logic, data retrieval, building SQL queries and presentation: everything is done in the same .PHP file!

In Windows client development you typically let a data layer retrieve some data objects (via ADO.NET, Linq or something else) then you assign the data object to a grid control. The control takes care of displaying the content of the object, dynamically creating columns and adding rows… In the PHP files I saw the data layer bit (building the SQL, performing the query, retrieving data) appears just a few lines before a while loop that echoes  <td> tags to generate a table for each row of the result set. All layers in the same place.

Also in the code I saw SQL queries were always built on the fly. The first downside of building SQL queries with string concatenation is vulnerability to SQL injection (followed by performance loss due to always rebuilding the query plan and no transactional integrity)… The devs protected the code from injections by using specific PHP functions that parse all user inputs. Why not use stored procedures? Apparently stored procedures are a new thing in MySql: they arrived only in version 5. So I guess they got into the habit of doing away without them.

Regarding the development tools, when working with PHP in Dreamweaver -apart from a bit of color coding- you get absolutely ZERO compiler support. Indentation is manual and there is no intellisense. Comes pretty close to handcarving on stone tablets.

Another funny thing with MySQL: it comes with a large choice of database engines, each one having its pros and cons. It turns out that the default db engine in MySQL is optimised for speed and does not support foreign keys. So by default you work in a system where you can delete parents without deleting childrens and where children can contain a reference to a non existing parent. You’d better be sure that’s what you want, most of the time it’s not…

Saturday, 30 January 2010

Coding in Istanbul 1: PHP in Turkey

DSC_9868 (640x426)

I’m having a çoban kavurma in a restaurant with my old mate Jeff who lives in Istanbul. We stopped for lunch in one of the busy pedestrian streets of Kadiköy on the Asian side of the city.

We spent most of yesterday coding on Jeff’s doctors appointment website. We added a small admin functionality that will save him time when mailing doctors. He’s currently working on getting more doctors registered: it involves finding their details and emailing them one by one. Obviously spamming them all in one go is out of the question so each email is slightly personalised (quoting the doctor’s own website). Yet they all follow the same template so there is some room for automation.

Jeff also needs to track how doctors open and view the campaign emails. Each email contains a number of links: to the appointment’s website, press articles, facebook and to twitter. Pretty clever: each link calls a php on his website that logs the doctor’s identifier along with the identifier of the link. Then he can run a report that shows various stats about who clicked what link…

Good places to take the netbooks out in Turkey include a growing number of restaurants, Starbucks, Gloria Jeans… and Wifi is almost always free!

Gloria Jeans is a great place to sit down with the laptop: more comfortable than Starbucks and less crowded. The one in Ortaköy is perfect: beautiful view to the Bosphorus, 3 levels, smoothies, cheesecakes, tosts (paninis) and free wifi.


View Istanbul in a larger map

Tuesday, 12 January 2010

Setup, TearDown, When to Mock?

 

I’m writing this while eating at the Launceston Place, a very comfy restaurant near Gloucester Road. Warm decor, soft lighting and excellent service.

  • Setup and TearDown

Only use those to respectively initialise then destroy mock objects. Methods with [Setup] and [TearDown] attributes are called before and after each test so they’re the only methods that guaranty that things get cleaned up.

Best to avoid the [SetupFixture] attribute (to mark a class that contains Setup and Teardown methods called just once before ALL tests), as well as [TestFixtureSetup]/[TestFixtureTeardown] (called once only before all tests in a fixture). I tried to be clever and used them and ended up having to debug nasty side effects. Unit tests are supposed to run in isolation, so just one [Setup] before and one [TearDown] after each of them, that’s it.

  • What should be mocked?

Yesterday while writing a UT I bumped into this question: should I bother writing a mock for that object or can’t I just use the real thing?

One answer: objects that access external systems (DB, hard-drive, client API to some server-based system, …) should always be mocked (for unit-tests that is, not integration tests), otherwise the test is not repeatable, and it’s slow…

If a method under test accesses an injected object and this object does only in-memory stuff, then no need to mock it. It would make sense to mock it because after all this injected object is not actually the thing you want to test… Still, the more real objects are covered by the test, the better. As long as they all run in-memory and do not prevent the test from being repeatable, all is well. So no need to waste time writing a mock in that case.

Sunday, 10 January 2010

Got to Treat Yourself

I bought myself a Christmas present. Next week I’ll receive this neat little thing:

4AC9FC19E4EE012DE10080002BC29BDF 4AC9FC55E4EE012DE10080002BC29BDF

It’s a VAIO X-Series.

This is a replacement for the VGN-SZ5 that I carried around every day in a rucksack for more than two years. Now the rucksack will disappear to be replaced by a tiny messenger bag.

Premium carbon finish, SSD drive, embedded 3G. Extremely thin and less than 900g. This is going to be pure pleasure.

In 2007 netbooks were not around yet and I needed something to SwissGearwrite code on the go, not just in my lounge. I got the VGN-SZ5 and a Wenger SwissGear rucksack, both of which I’m still using today. I went everywhere with them: Hertfordshire, TechEd Barcelona, Egypt, the French Pyrenees (picture), Paris, Istanbul… In London I use the Vaio in the tube, in Starbucks, Costa, in pubs, in restaurants (which is extremely annoying to people I go out with).

Last year the rucksack felt a bit heavy on my shoulders so I started looking around for a netbook. I read reviews and benchmarks in magasines, tried out all sorts of models in the shop… Samsung, Asus Eee, Acer, Lenovo, MSI Wind… It wasn’t quite that. Either the 3G card was not embedded, or the thing was too bulky… Not sexy enough.

The first time I saw the X-Series in a Sony shop (in October last year two days before the official UK launch), I immediately thought “I want that”. I gave myself some time to think it over (it’s a bit of an investment). In its October 2009 review, PC Pro said:

But the real problem for the X-Series is its price. (…) Who, though, can lavish £1,130 exc VAT on a vanity laptop such as this?

I just ordered one.

 

1237476547103

Monday, 4 January 2010

Cheap IoC in native C++

In a 2008 episode of dnrTV James Kovac describes how to create a very simple IoC container. All the container does is mapping the name of an interface to an instance of this interface.

Easy to do in .NET with a generic Dictionary, but what about native C++? It turns out it's not that much more difficult: all you need is a STL map and C++'s typeid. 

What is an IoC container?
A class that given an interface ISomething returns the concrete object implementing ISomething.


Why do you need an IoC container ?

It's one way to do dependency injection.

Let's say you want to unit-test some business layer that relies on a data layer. The business layer sees the data layer through an interface (let's call it IDataLayer). The actual instance of IDataLayer is created by either:

  • the entry point of the code that uses the business layer (an executable for instance). This is production code and it has to create the object connecting to the real database.
 
int _tmain(int argc, _TCHAR* argv[])
{
    // Create the concrete DataLayer object
    DataLayer dataLayer;

    // Register it with the resolver
    Resolver::Register<IDataLayer>(dataLayer);

    BusinessStuff businessStuff;
    businessStuff.DoStuff();

    return 0;
}
  • the unit-test code. The IDataLayer object in that case is a fake object that doesn't connect to the real database and therefore makes unit-tests easily reproducible and fast.

void BusinessStuffTest::DoStuff_Nominal_DoesNotRaiseException()
{
    // Arrange

    // Create the concrete DataLayer object
    FakeDataLayer dataLayer;

    // Register it with the resolver
    Resolver::Register<IDataLayer>(dataLayer);

    BusinessStuff businessStuff;

    // Act
    businessStuff.DoStuff();

    // Assert
    dataLayer.AssertStuffWasDone();

}

    Implementation of the Resolver class
    The Resolver class has two methods Register and Resolve. Here is the code for the Resolver class (very straightforward):


      #pragma once

      #include
      #include

      using namespace std;

      ///
      /// Allows you to Register and Resolve global objects
      ///
      class Resolver
      {
          static map<string, void* > typeInstanceMap;

      public:

          template<class T> static void Register(const T& object)
          {
              typeInstanceMap[typeid(T).name()] = (void*)&object;
          }

          template<class T> static T& Resolve()
          {
              return *((T*)typeInstanceMap[typeid(T).name()]);
          }
      };

      This is a just a starting point. In production Resolver could do with methods such as Unregister(), IsRegistered(), UnregisterAll()...


      An IoC container is one way to do dependency injection. Other ways include:
      • pass the injected object using the constructor
      • pass the injected object using a setter method
      Both methods above can become tedious very quickly in a big project.
      I tried all 3 methods and I find the IoC approach scales rather well. I've used it for 10 months in a production project and it works nicely with unit tests.

      Resources:

      Jean-Paul Boodhoo on the gateway pattern
      Roll your own IoC container (dnrTV)
      The Art of Unit-Testing where Roy Osherove tells everything about dependency injection and fake objects.

      Monday, 9 November 2009

      Windows 7 and the Press

      I was replacing Windows 7 beta with Windows 7 Release Candidate on my laptop a few months ago... And I was joking with colleagues about the future success of this new version. Having used Vista for two years
      Windows 7 didn't strike me as an earth-shattering evolution. Yet we predicted that people would rush on Windows 7 like bees on cupcakes for two main reasons:
      • the press was very positive about Windows 7 after it slaughtered Vista
      • the Release Candidate was as good as the final thing and free to use until March 2010. In other words people could use a fully working OS for free for 10 months.
      I don't think Vista deserved the bad press it received. I used it for two years: RC1 and RC2 on an old PC then the RTM 32-bit and eventually 64-bit on my main machine.
      Driver issues? A bit at the beginning but nothing that couldn't be overcome.
      Speed issues? Desktop: nothing that I could notice on a 4GB dual-processor Dell with raid-striped 15000rpm SAS drives. Laptop: browsing the web was fine, Visual studio was slow. After I moved to Windows 7 browsing the web was still fine and Visual Studio still slow.

      When I removed Vista 32-bit from my Vaio and installed Windows 7 RC, it indeed felt a bit snappier. But again, ANY version of Windows feels snappier after a fresh install.

      Whether you run XP, Vista or Windows 7 the OS you're running on a machine does not matter as much as the following factors:
      - how fresh the install is: re-install your OS often, with a bit of organisation it can be quick.
      - the amount of RAM: install more than you think you'll need.
      - the hard drive speed: one 15000rpm drive is good, two are better.


      I might oversimplify a bit, still I think there are mostly two reasons why you want to upgrade to a new OS:
      • the untold one: it looks better than the previous version.
      • the one you tell people to look clever: the OS is faster, contains bug fixes and new useful features.
      Thanks to unbalanced press opinions Windows 7 at last gives people an excuse to cave in to the first reason while being covered by the second.