Tools I Use to Learn Tough Languages (1)

More than a year after I started, I'm still working on Chinese and Japanese.
I'm not fluent yet, far from it... But I'm better than I was 1 year ago so here is a list of tools I use.

No SQL and Big Data Resources

Overview

Martin Fowler's overview of No-Sql databases

Databases

• HBase (used by Facebook for Messenger).
  The storage technologies behind Facebook messenger.
• MongoDb. Replication, auto-sharding, accessible over REST.
  Introduction to MongoDB 
• Cassandra. Peer-to-peer read/write replication.

Analysis

• Hadoop. used to perform distributed computations over very large files.  Overview of Hadoop and related languages and databases.

Complex Event Processing

• StreamInsight

Integrates with Linq for queries. Can be used with reactive extensions.
Streaminsight on Channel 9

• Storm

Relational streaming and Hadoop

API Design

• REST

Introduction to REST

API Design with REST

Overview Of Asynchronous Tools in C# and C++

Asynchronous programming is satisfying, it feels good to write code that doesn't block.

The tools to do it have evolved a lot over the past few years, both in C# and C++.

Adapter And Dependency Injection Without The Pretty Diagrams

How I used Adapter, Singleton and Dependency Injection (in real life, without the dry diagrams).

Confidently Assert Fallacies with Statistics, Surveys and Luck

You can be as rigorous and thorough as you like, with a bit of luck you can prove relationships between things that don't actually exist.

Java vs C++ vs C#: What Is The Best Programming Language?

From Akihabara in Tokyo how to choose a programming language?

Hide legacy C++ APIs behind WCF to simplify deployment

The problem

We are using a number of in-house C++ APIs for various systems (static info repository, Excel calculation libraries, internal market data repository, configuration repository). Those APIs were originally designed to be run in Excel spreadsheets and date back to a time when .NET didn't exist.
Some of those APIs now come with 50 or more native Windows DLLs and sometimes require installation of database drivers on the client machines.

To use them in a .NET solution you have to write C++/CLI wrappers. Interop is not enough because APIs have complex data types so you must write the marshalling code in C++/CLI yourself. Adding mixed (managed/native) libraries to .NET solutions makes builds slightly more difficult to manage.

If you are creating Winforms or WPF GUIs that happen to use one of those APIs directly you have to deploy those native DLLs or ODBC drivers to users machines. It's just painful and ClickOnce doesn't really like that.

Proposed solution

Hide those APIs along with their native DLLs and db drivers behind WCF endpoints. Deployment complexity is limited to the server. All the clients have to do is make a WCF call.

This brings other problems though:

• how to avoid building a single point of failure whereby if one API call gets stuck then all clients trying to access the service are stuck.
• how to manage user's credentials to apply proper authorisation? (because the APIs are now called from a server).
• how to manage user's session state? (an example of state is in-memory cache which is sometimes maintained to increase performance between API calls)

In our case it turns out that some of those APIs

• have state. They require to connect and disconnect (create an environment for the duration of the user's session then destroy it). 
• are not thread-safe: they were originally designed to run inside single-threaded clients such as Excel and do not have a predictable behaviour when several instances of the API are created in different threads inside the same process. 
• can be very memory-hungry with long-running operations and direct access to various databases (Oracle, SQL Server). Because it's still 32-bit code, the hunger for memory can be a problem as we've seen situations when passing arguments in bulk to an API call breaks the 2GB memory limit.

WCF allows you to have a per-session instantiation mode, which sounds good here: it makes sense to create an instance of each API for each client WCF proxy. However we really can't run all those instances inside a single process with multiple threads. The code is not thread-safe and because it's 32-bit we would quickly run out of memory.

WCF automatically manages creation of a thread pool to run services concurrently but it doesn't manage the creation of separate processes, which is what we want to do here really. The only way to ensure the native APIs instances will not create side effect is to run them in the same way they were designed and tested: one per process.

How to do that?

Let's say the contract's interface is called IMyContract.

You could create a dispatcher service implementing IMyContract with binding netHttpBinding for instance. This would be addressable from any host in the company.
Then the implementation of IMyContract would just redirect the calls to other services, hosted on the same server in different processes.

The dispatcher service would be configured with InstanceContextMode = PerSession to have one instance created for each client proxy.

Every time a DispatcherService instance is created, it would spawn a process. That process in turn would create a WCF endpoint with binding netNamedPipesBinding. Named pipes would do fine for low overhead interprocess communication on the same machine.

Because there is only one dispatcher service instance per user session, there would also be one process per user session. This would ensure that those native C++ non re-entrant client APIs would be completely isolated from each other.

(To be continued...)

Algorithmic Trading Resources

Books

• Numerical methods in Finance with C++ very concise and programming-oriented. You have to be familiar with the methods before reading it (binomial method, finite difference, implied volatility with Black Sholes...)
• All About High Frequency Trading very unpretentious book, easy to read. Gives practical tips about the physical architecture of an automated trading platform.
• C# for Financial Markets very dense. Not much prose, many bullet points. Describes patterns for bond pricing, binomial method, finite difference method, interaction with Excel, C++/CLI... 
• Financial Instrument Pricing using C++ (review by QuantStart)
• Top 5 Essential books for algorithmic trading (QuantStart)
• Paul Willmot on quantitative finance
• Architects of Electronic Trading (SafariOnlineBooks)
• Encyclopaedia of Financial Models (SafariOnlineBooks)
• Large reading list from QuantStart

Overviews

• NumericalMethod courses the slides give a rough overview of the tools/strategies used for algorithmic trading.

Strategies

• Cheat sheet for algorithm trading (PDF) Deals with margin accounts.

Sites

• Quantopian algorithmic trading platform: web app that allows you to create algorithms and back-test them using historical data.
• Quantstart has a list of free resources here.

Open Source Projects

• Algo Trader: written in Java, there is a an open source version and full featured enterprise version. The architecture diagram is interesting.

Technology

• Trading floor architecture diagram from Cisco 
• Large file storage: HDF5
• High speed time series databases: kdb, OneTick, Reuters Velocity Analytics (formerly Vhayu)
• Big data in banking: how four financial giants crunch big data.

Courses

• Computational investing 8-week online course on coursera.

Learning Japanese

What can you do when you take the tube? Read the free version of the Evening Standard or learn Japanese with flashcards.

Same as for Chinese I use cram.com so store my vocab.

I started re-arranging the vocab list in 3 columns, separating Kanji from Romaji. like this:

You can import this format into cram.com which supports 3-sided cards.

Then you download those cards to Flashcards Deluxe where you can choose which side to use for the prompt: either English or Kanji.

One backup is never enough...

I had trouble with my backup lately when switching to a new MacBook. I tried restoring the data to the new machine as a test.

The WD Passport software restore failed (it restored a 6-month old version of my files for some reason). Since the WD Passport software doesn't have a backup tool for Mac, I tried to configure it with Timemachine instead. The WD Passport USB drive died. I reformatted it from Windows: impossible to write to it any more. Surface tests with the WD diagnostic tools failed. It probably didn't like the whole traveling thing.

The Mozy restore worked fine.

So here is the new setup:

Off-site backup

• Automatic backup to Mozy using local encryption with passphrase. 

On-site backup

• Automatic backup to a USB drive with TimeMachine and Apple encryption.
• Manual backup every now and then of sensitive files to a USB key protected with Truecrypt. It's good to have a simple copy of your important data. A simple, straight copy that doesn't require backup software. Because DVDs are out, a simple USB key should do.

The main disk is encrypted with FileVault. 

I'm still using Mozy but I'm considering moving to Backblaze that also offers local encryption with a passphrase that no one else knows and unlimited storage. Also Backblaze claims to be able to backup data from a USB drive as well without getting confused. I tried doing this with Mozy and it got mad, deleting all my files from the server as soon as I unplugged the drive.