No SQL and Big Data Resources

Overview

Martin Fowler's overview of No-Sql databases

Databases

• HBase (used by Facebook for Messenger).
  The storage technologies behind Facebook messenger.
• MongoDb. Replication, auto-sharding, accessible over REST.
  Introduction to MongoDB 
• Cassandra. Peer-to-peer read/write replication.

Analysis

• Hadoop. used to perform distributed computations over very large files.  Overview of Hadoop and related languages and databases.

Complex Event Processing

• StreamInsight

Integrates with Linq for queries. Can be used with reactive extensions.
Streaminsight on Channel 9

• Storm

Relational streaming and Hadoop

API Design

• REST

Introduction to REST

API Design with REST

Overview Of Asynchronous Tools in C# and C++

Asynchronous programming is satisfying, it feels good to write code that doesn't block.

The tools to do it have evolved a lot over the past few years, both in C# and C++.

Adapter And Dependency Injection Without The Pretty Diagrams

How I used Adapter, Singleton and Dependency Injection (in real life, without the dry diagrams).

Confidently Assert Fallacies with Statistics, Surveys and Luck

You can be as rigorous and thorough as you like, with a bit of luck you can prove relationships between things that don't actually exist.

Java vs C++ vs C#: What Is The Best Programming Language?

From Akihabara in Tokyo how to choose a programming language?

Hide legacy C++ APIs behind WCF to simplify deployment

The problem

We are using a number of in-house C++ APIs for various systems (static info repository, Excel calculation libraries, internal market data repository, configuration repository). Those APIs were originally designed to be run in Excel spreadsheets and date back to a time when .NET didn't exist.
Some of those APIs now come with 50 or more native Windows DLLs and sometimes require installation of database drivers on the client machines.

To use them in a .NET solution you have to write C++/CLI wrappers. Interop is not enough because APIs have complex data types so you must write the marshalling code in C++/CLI yourself. Adding mixed (managed/native) libraries to .NET solutions makes builds slightly more difficult to manage.

If you are creating Winforms or WPF GUIs that happen to use one of those APIs directly you have to deploy those native DLLs or ODBC drivers to users machines. It's just painful and ClickOnce doesn't really like that.

Proposed solution

Hide those APIs along with their native DLLs and db drivers behind WCF endpoints. Deployment complexity is limited to the server. All the clients have to do is make a WCF call.

This brings other problems though:

• how to avoid building a single point of failure whereby if one API call gets stuck then all clients trying to access the service are stuck.
• how to manage user's credentials to apply proper authorisation? (because the APIs are now called from a server).
• how to manage user's session state? (an example of state is in-memory cache which is sometimes maintained to increase performance between API calls)

In our case it turns out that some of those APIs

• have state. They require to connect and disconnect (create an environment for the duration of the user's session then destroy it). 
• are not thread-safe: they were originally designed to run inside single-threaded clients such as Excel and do not have a predictable behaviour when several instances of the API are created in different threads inside the same process. 
• can be very memory-hungry with long-running operations and direct access to various databases (Oracle, SQL Server). Because it's still 32-bit code, the hunger for memory can be a problem as we've seen situations when passing arguments in bulk to an API call breaks the 2GB memory limit.

WCF allows you to have a per-session instantiation mode, which sounds good here: it makes sense to create an instance of each API for each client WCF proxy. However we really can't run all those instances inside a single process with multiple threads. The code is not thread-safe and because it's 32-bit we would quickly run out of memory.

WCF automatically manages creation of a thread pool to run services concurrently but it doesn't manage the creation of separate processes, which is what we want to do here really. The only way to ensure the native APIs instances will not create side effect is to run them in the same way they were designed and tested: one per process.

How to do that?

Let's say the contract's interface is called IMyContract.

You could create a dispatcher service implementing IMyContract with binding netHttpBinding for instance. This would be addressable from any host in the company.
Then the implementation of IMyContract would just redirect the calls to other services, hosted on the same server in different processes.

The dispatcher service would be configured with InstanceContextMode = PerSession to have one instance created for each client proxy.

Every time a DispatcherService instance is created, it would spawn a process. That process in turn would create a WCF endpoint with binding netNamedPipesBinding. Named pipes would do fine for low overhead interprocess communication on the same machine.

Because there is only one dispatcher service instance per user session, there would also be one process per user session. This would ensure that those native C++ non re-entrant client APIs would be completely isolated from each other.

(To be continued...)

Algorithmic Trading Resources

Books

• Numerical methods in Finance with C++ very concise and programming-oriented. You have to be familiar with the methods before reading it (binomial method, finite difference, implied volatility with Black Sholes...)
• All About High Frequency Trading very unpretentious book, easy to read. Gives practical tips about the physical architecture of an automated trading platform.
• C# for Financial Markets very dense. Not much prose, many bullet points. Describes patterns for bond pricing, binomial method, finite difference method, interaction with Excel, C++/CLI... 
• Financial Instrument Pricing using C++ (review by QuantStart)
• Top 5 Essential books for algorithmic trading (QuantStart)
• Paul Willmot on quantitative finance
• Architects of Electronic Trading (SafariOnlineBooks)
• Encyclopaedia of Financial Models (SafariOnlineBooks)
• Large reading list from QuantStart

Overviews

• NumericalMethod courses the slides give a rough overview of the tools/strategies used for algorithmic trading.

Strategies

• Cheat sheet for algorithm trading (PDF) Deals with margin accounts.

Sites

• Quantopian algorithmic trading platform: web app that allows you to create algorithms and back-test them using historical data.
• Quantstart has a list of free resources here.

Open Source Projects

• Algo Trader: written in Java, there is a an open source version and full featured enterprise version. The architecture diagram is interesting.

Technology

• Trading floor architecture diagram from Cisco 
• Large file storage: HDF5
• High speed time series databases: kdb, OneTick, Reuters Velocity Analytics (formerly Vhayu)
• Big data in banking: how four financial giants crunch big data.

Courses

• Computational investing 8-week online course on coursera.

Learning Japanese

What can you do when you take the tube? Read the free version of the Evening Standard or learn Japanese with flashcards.

Same as for Chinese I use cram.com so store my vocab.

I started re-arranging the vocab list in 3 columns, separating Kanji from Romaji. like this:

You can import this format into cram.com which supports 3-sided cards.

Then you download those cards to Flashcards Deluxe where you can choose which side to use for the prompt: either English or Kanji.

One backup is never enough...

I had trouble with my backup lately when switching to a new MacBook. I tried restoring the data to the new machine as a test.

The WD Passport software restore failed (it restored a 6-month old version of my files for some reason). Since the WD Passport software doesn't have a backup tool for Mac, I tried to configure it with Timemachine instead. The WD Passport USB drive died. I reformatted it from Windows: impossible to write to it any more. Surface tests with the WD diagnostic tools failed. It probably didn't like the whole traveling thing.

The Mozy restore worked fine.

So here is the new setup:

Off-site backup

• Automatic backup to Mozy using local encryption with passphrase. 

On-site backup

• Automatic backup to a USB drive with TimeMachine and Apple encryption.
• Manual backup every now and then of sensitive files to a USB key protected with Truecrypt. It's good to have a simple copy of your important data. A simple, straight copy that doesn't require backup software. Because DVDs are out, a simple USB key should do.

The main disk is encrypted with FileVault. 

I'm still using Mozy but I'm considering moving to Backblaze that also offers local encryption with a passphrase that no one else knows and unlimited storage. Also Backblaze claims to be able to backup data from a USB drive as well without getting confused. I tried doing this with Mozy and it got mad, deleting all my files from the server as soon as I unplugged the drive.

Learning Chinese

I bought a couple of phrasebooks while traveling. Making out in Chinese is a fun one, the Rough Guide Phrasebook: Mandarin Chinese fits in your pocket.

I also used the flashcards by Brainscape. They contain some English > Chinese vocab (+audio), some Chinese > English, some grammar and even character radicals.

I maintain a vocabulary list in a spreadsheet which I export to flashcards using cram.com (used to be flashcardexchange) for long-term memorisation. When I'm in the tube I play the flashcards on the iPhone with Flashcards Deluxe, one of the many flashcards apps that support cram accounts. I haven't tried many flashcard apps actually. I like this one because it allows you to fine tune the spaced repetition algo. So you can set how long it takes before they prompt you with a word you got wrong. And when you get a word right you can set how long it takes before the app prompts you again to check you didn't forget.

Cram allows you to work with 3-sided cards. This is good for languages like Chinese or Japanese where you need to learn characters on top of the romanised version of the word.

For a while I listened to Serge Melnyk's free podcast as well as One Minute Mandarin from Radio Lingua network.

I gave a quick try to Rosetta Stone in a shop. It gave me the impression of being no more than a very polished overpriced phrasebook, I wasn't convinced.

So far the best tool I found is the course by chineseclass101. Courses are structured in a large number of short lessons, each lesson has a 15min podcast with pdf notes, online quizzes, vocabulary and grammar points. It's very progressive and they actually explain you the grammar from the ground up. The vocab always comes with the Simplified Chinese, Traditional Chinese and pinyin representations. The beginner lessons have 4 lines of dialog each. The advanced lessons have entire texts. You have to use at least the premium option for the material to be useful. The podcasts happen to be lively and not too scripted.

The same company that does chineseclass101 sells an iPhone app WordPower. This one is useful to learn vocab. The vocab is organised in 10 levels of flashcards of about 100 words each starting with the most basic. You can also tackle the flashcards by category.

Letting go of the pinyin

At the beginning I was learning the Roman representation of Chinese words while completly ignoring the actual Chinese characters. Then after using the line-by-line audio from chineseclass101, I realised recognising characters is not that difficult.

Pinyin exists only to give a phonetic representation of the sound for each Chinese character. However anything else than the International Phonetic Alphabet is actually misleading. It's just as lame as those English phonetic approximations you find in Lonely Planet phrasebooks. If you have access to an online resource / electronic dictionary that plays out a word out loud when you click it (such as Google Translate), then you don't need pinyin. All you have to do is read the characters at the same time as you listen to the way they're pronounced in order to associate each character with its sound and tone.

The objective is to learn to visually recognise the Chinese characters, pinyin is just a way to guess the sound associated with the character when the actual sound is not available. If you have access to the real sound, forget pinyin. By the way from my last trip to Asia I got the impression that most Chinese people don't know about pinyin anyway.

For pinyin to be useful you actually have to learn how to pronounce it with a chart like this one, that lists the 4 tones of each syllable combination.

To recognise characters it helps to know the radicals they're composed of. It helps memorisation as well. Sometimes they make sense... Here is a list of radicals together with their simplified version. Brainscape here is a good flashcard tool to learn them. You can find a fun visual way of associating radicals with characters on Chineasy.org.


Texting

I've also found out that a good exercise to train recognising the characters is to text on the iPhone using the Mandarin keyboard. The keyboard expects pinyin as input then suggests a list of matching homophone characters sorted by frequency. Choosing the right character from the list makes your brain work a little bit.