Fire-proof Software #2: Designing Fault Tolerant Systems

Follow-up from part 1.

This is a few of the failure conditions we see with real production software, along with some approaches to deal with them. The general idea is to define activities.

• Each activity is a unit of work that may either complete or fail and that can be retried if necessary.
• An activity has a start and an end, as well as a status (Complete/Failed).
• An activity also has success criteria that allow a process to check the activity actually completed.

The database contains a schedule of those activities along with a MaxRunTime value and a flag indicating if it's currently running. The MaxRunTime allows processes to spot an activity that timed out.

Input data not available (Failure of one of the upstream systems)

You can’t bet on the fact that upstream systems will work, regardless of the reason why, they will fail. There is no point trying to imagine the reasons behind the potential failure, what matters is the impact of the failure on your system and on the business.

• Have a retry policy for each activity (retry period + retry window)
• Use a stale version of the data (previous day for instance) if acceptable with the business.

Notification not available

• Back-up the notification mechanism with an activity schedule: an activity should automatically start at a defined time if it hasn’t already.
• If an activity has already started following a notification, the DB flag in the activity schedule table will guarantee that it doesn't start again.

Process crash

• Have a Windows service detect the process crash and restart the process.
• If the process crashes following an unhandled exception, the currently running activity will be automatically marked as failed. When the process starts again it should attempt to schedule or start the failed activity.
• If the process is simply killed with no opportunity to mark the current activity as failed the activity will still appear as running in the ActivitySchedule. When the process starts again it will see that its activity is currently running and will simply schedule a check at StartTime + MaxRunTime. Obviously the check will fail and the process will restart the activity.

Process stalls

• Kill it with a watchdog thread: this is a pattern used by embedded systems on real-time OSs to reset a stalled CPU. Inside the app server process the main thread –the one that does all the work- should periodically reset the watchdog flag. If the watchdog flag is not reset after a defined period of time the watchdog thread kills the process after failing the current activity.

Server down or unreachable

• Use 2 servers, one primary and one backup.
• Each server runs identical processes scheduling the same activities. Both processes will attempt to start their activity at the same time however the ActivitySchedule table in the DB will allow only one process to actually start the activity.
• If one server goes down, all processes will simply attempt to reschedule their activities upon restart.
• If one server goes down while an activity is running, the activity will still appear as running in the ActivitySchedule. Upon restart processes will see the activity is marked as running and will schedule a check at the expected completion time.

Database not available

• Avoid maintaining a single DB connection for too long, that reduces the opportunities of the system to reconnect.
• If you can detect the connection failure and don’t want the complexity of implementing a retry mechanism, at least fail the current activity to take advantage of the activity’s retry policy.

Sink-Proof Software #1: Designing Fault Tolerant Systems

Middleware Enterprise apps usually do straightforward things: they pump data from here, crunch it a bit, dump it there. Complexity comes from the variety of data sources, their reliability and more importantly how much availability (up-time) the business requires.

Say you write an app that processes a daily list of financial instruments to perform a Present Value calculation. What if the input data is not available, i.e. if that list of instruments is not ready? Do you raise a critical fault and give up? Do you use a stale list of instruments? Do you wait a bit and retry? How long should you retry, should you retry forever until the list is available or is there a point where you should give up? Should the retry period be constant or should it increase exponentially to save resources?

What if one of your processes was waiting for a notification from another system and that notification was never received?

What do you do if a process crashes because of an unhandled exception, an out-of-memory exception or a third-party library bug? Is there a system in place to restart the process automatically? Does it take over its task where it left off or does it restart from scratch?

What do you do if a process stalls? Because of a blocked DB call for instance, or some synchronous API call that never comes back…

What if that process was handling client requests? Do you have a backup process to handle the requests while the primary process recovers? Can the system handle a down time?

What do you do if the entire application server goes down either because an engineer tripped on the power cable, the server room was flooded by hurricane Irene, a plane crashed on your main datacentre, the datacentre's power supply failed or there is no more network connectivity to the datacentre?

What do you do if the database is not available for some time? Do you store data in memory temporarily? Do you detect it and try and reconnect or do you just fail on all subsequent DB calls? Do you reroute the DB logging to log files so that logs remain accessible?

Do I need this config file?

Config files are all over the place with .NET: WCF, Log4Net, everything comes with an XML config file even if you didn't ask for one.

But when you think about it, why do you need config files really? Usually that's because you want to change some behaviour without having to rebuild the code. Ok. But what is so bad about rebuilding the code?

In the Enterprise world if you intend to make a change to a config file in production, you’re not going to just change the file on the production server (unless you're Jack Bauer and you have a death wish after your wife was murdered). You're going to make the change in the staging environment first then test it. If the system didn’t collapse then you'll raise a change request, have it approved by change management and eventually release it in prod. Well now compare that with rebuilding the code... mmm? Same amount of hassle, you'd have to go through the same steps anyway.

I find the main benefit of config files in Enterprise server apps is to ease the pain involved in moving from development to staging and from staging to production. This is where the config files are really useful because you have one code base for many environments. Connection strings, timeout values, environment identifiers, log file paths, simulated dates, weekend definitions, job start times, retry windows, SMTP servers, Tibco RV transport settings... They all depend on where you run the code from.

Each Technology Has Its Place

You would think software engineering is an activity where the decision to use such or such tool is based on pragmatic considerations: short term development costs and long term maintenance costs. In reality there is often a mix of religion (belief that ONE language is better than all others because it’s the language I learnt from an old bold master in a temple somewhere in Tibet) and fashion (technology perceived as cool because its name often appears in blogs and job specs).

Software libraries form a giant, ever-evolving toolset where each tool is optimal only for a narrow range of applications. This is in spite of the massive efforts by standard bodies to build all-purpose languages. No language is truly all-purpose, each one has its preferred applications.

Languages:

• C++ stack: native programming with Win32, STL, Boost… Best for situations where the CPU or the memory usage becomes a bottleneck: embedded systems, server apps handling large volumes of requests. The focus is on execution speed and low memory footprint.
• C# stack: .NET, CLR, Linq, WCF, Windows Forms… Best for situations where the performance bottleneck is in the database or the network. Need for high level of customisation while keeping development costs down: the focus is on ease of development.

GUI toolsets:

• MFC: extremely responsive front-end for consumer-grade applications (as opposed to Enterprise apps). Has been around for ages and is still actively maintained by Microsoft.
• Windows Forms: all-purpose front-end, ideal for Enterprise apps. Plenty of of 3rd party controls available. Very easy to code (when you compare to MFC for instance). Acceptably responsive. Deployment over ClickOnce easy as long as no admin rights are required and the target machines have the right .NET framework.
• WPF: suitable for front-ends with relatively simple functionality, with an accent on good looks or advanced interaction. Feels a little bit overkill for Enterprise apps in terms of ratio between programming effort / result. Model apps for WPF should be those cool-looking interfaces you see in films. You can also do boring-looking Windows apps (with a menu, drop-down lists, trees and grids…, but there is no much point, you might as well use WinForms). Deployment over ClickOnce similar to Windows Forms.
• Silverlight: suitable for extremely simple Enterprise front-ends (think IPhone apps but on the desktop), very easy Enterprise deployment via the web browser. The architecture guarantees that the client-side set of DLLs will be lean: you can’t deploy any 3rd party DLL, only Silverlight classes. Any 3rd party functionality has to run on a server, behind a WCF endpoint. In an enterprise context this type of constraint is actually a good thing because it forces you to leave complexity on servers.
• HTML5/Javascript: web apps with a large reach, required to work on any browser/platform. Would sound like a really tedious choice in a Windows-based Enterprise environment where much better tools are available. However for pure web apps HTML5/Javascript makes more sense than Flash or Silverlight.