Tools I Use to Learn Tough Languages (1)

More than a year after I started, I'm still working on Chinese and Japanese.
I'm not fluent yet, far from it... But I'm better than I was 1 year ago so here is a list of tools I use.

No SQL and Big Data Resources

Overview

Martin Fowler's overview of No-Sql databases

Databases

• HBase (used by Facebook for Messenger).
  The storage technologies behind Facebook messenger.
• MongoDb. Replication, auto-sharding, accessible over REST.
  Introduction to MongoDB 
• Cassandra. Peer-to-peer read/write replication.

Analysis

• Hadoop. used to perform distributed computations over very large files.  Overview of Hadoop and related languages and databases.

Complex Event Processing

• StreamInsight

Integrates with Linq for queries. Can be used with reactive extensions.
Streaminsight on Channel 9

• Storm

Relational streaming and Hadoop

API Design

• REST

Introduction to REST

API Design with REST

Overview Of Asynchronous Tools in C# and C++

Asynchronous programming is satisfying, it feels good to write code that doesn't block.

The tools to do it have evolved a lot over the past few years, both in C# and C++.

Adapter And Dependency Injection Without The Pretty Diagrams

How I used Adapter, Singleton and Dependency Injection (in real life, without the dry diagrams).

Confidently Assert Fallacies with Statistics, Surveys and Luck

You can be as rigorous and thorough as you like, with a bit of luck you can prove relationships between things that don't actually exist.

Java vs C++ vs C#: What Is The Best Programming Language?

From Akihabara in Tokyo how to choose a programming language?

Hide legacy C++ APIs behind WCF to simplify deployment

The problem

We are using a number of in-house C++ APIs for various systems (static info repository, Excel calculation libraries, internal market data repository, configuration repository). Those APIs were originally designed to be run in Excel spreadsheets and date back to a time when .NET didn't exist.
Some of those APIs now come with 50 or more native Windows DLLs and sometimes require installation of database drivers on the client machines.

To use them in a .NET solution you have to write C++/CLI wrappers. Interop is not enough because APIs have complex data types so you must write the marshalling code in C++/CLI yourself. Adding mixed (managed/native) libraries to .NET solutions makes builds slightly more difficult to manage.

If you are creating Winforms or WPF GUIs that happen to use one of those APIs directly you have to deploy those native DLLs or ODBC drivers to users machines. It's just painful and ClickOnce doesn't really like that.

Proposed solution

Hide those APIs along with their native DLLs and db drivers behind WCF endpoints. Deployment complexity is limited to the server. All the clients have to do is make a WCF call.

This brings other problems though:

• how to avoid building a single point of failure whereby if one API call gets stuck then all clients trying to access the service are stuck.
• how to manage user's credentials to apply proper authorisation? (because the APIs are now called from a server).
• how to manage user's session state? (an example of state is in-memory cache which is sometimes maintained to increase performance between API calls)

In our case it turns out that some of those APIs

• have state. They require to connect and disconnect (create an environment for the duration of the user's session then destroy it). 
• are not thread-safe: they were originally designed to run inside single-threaded clients such as Excel and do not have a predictable behaviour when several instances of the API are created in different threads inside the same process. 
• can be very memory-hungry with long-running operations and direct access to various databases (Oracle, SQL Server). Because it's still 32-bit code, the hunger for memory can be a problem as we've seen situations when passing arguments in bulk to an API call breaks the 2GB memory limit.

WCF allows you to have a per-session instantiation mode, which sounds good here: it makes sense to create an instance of each API for each client WCF proxy. However we really can't run all those instances inside a single process with multiple threads. The code is not thread-safe and because it's 32-bit we would quickly run out of memory.

WCF automatically manages creation of a thread pool to run services concurrently but it doesn't manage the creation of separate processes, which is what we want to do here really. The only way to ensure the native APIs instances will not create side effect is to run them in the same way they were designed and tested: one per process.

How to do that?

Let's say the contract's interface is called IMyContract.

You could create a dispatcher service implementing IMyContract with binding netHttpBinding for instance. This would be addressable from any host in the company.
Then the implementation of IMyContract would just redirect the calls to other services, hosted on the same server in different processes.

The dispatcher service would be configured with InstanceContextMode = PerSession to have one instance created for each client proxy.

Every time a DispatcherService instance is created, it would spawn a process. That process in turn would create a WCF endpoint with binding netNamedPipesBinding. Named pipes would do fine for low overhead interprocess communication on the same machine.

Because there is only one dispatcher service instance per user session, there would also be one process per user session. This would ensure that those native C++ non re-entrant client APIs would be completely isolated from each other.

(To be continued...)