.NET Goes Native

I am very excited about the .NET Native project. And while this is just for WinRT app store apps right now, I was encouraged by the commentary in the Channel 9 presentation. I am eager to learn more about it.

For a long time, the speed of desktop processors has spoiled us, allowing us to depend on heavier and heavier JIT workloads and dynamic utilization of multiple core libraries. With the challenges of mobile coming along, we get the benefit of heavy duty platform specific optimization in .NET Native now.

Because I often write server applications in C# that could really benefit from that extra platform optimization, I am hoping and looking forward to being able to utilize .NET Native in my work as well.

Uncle Bob on TDD

I rarely parrot blogs here but sometimes I find a post I don’t want to lose track of. Robert Martin (Uncle Bob), one of my favorites, just posted The Domain Discontinuity which asks the question: Which came first, the chicken or the road?

Two quick pull quotes.

The first has to do with the appropriate scope of TDD. Why do I include this? Because I am not a TDD fanatic. By fanatic, I mean those developers who would disagree with Uncle Bob here, even going so far as to make private and protected methods public just so their tests can access them.

“Let me stress this more. I do not create a test for every method or every class. I create tests that define behaviors, and then I create the methods and classes that implement those behaviors.

“At the start, when there are just a few tests, I might have only one simple method. But as more and more tests are added, that one simple method grows into something too large. So I extract functions and classes from it -- without changing the tests. I generally wind up with a few public methods that are called by my tests, and a large number of private methods and private classes that those public methods call; and that the tests are utterly ignorant of.

“By the way, this is an essential part of good test design. We don't want the tests coupled to the code; and so we restrict the tests to operate through a small set of public methods.”

The second pull quote is a small set of architectural principles that I completely agree with:

    • Refactoring across architectural boundaries is costly.
    • Behaviors extracted across architectural boundaries need newly rewritten tests.
    • Architecture is an up-front activity.

“The solution to that problem is to know in advance where you are going to put certain behaviors. You need to know, in advance, that there will be a boundary between the GUI, the business rules, and the database. You have to know, in advance, that the features of your system have to be broken up into those areas. In short, before you write your first test, you have to ‘dream up the [boundaries] that you wish you had’.”

I sometimes come across systems that are woefully inadequate to the task, extremely difficult to refactor, and have few, if any, tests that do not require integration across boundaries that are often, if not almost impossible, to mock or fake. Virtually all of these problems are attributable to the false notion that Agile means you rush to coding and skip up-front architectural analysis and design.

Thanks, Uncle Bob!

My Technical 2013

Technically speaking, I had a fun and productive 2013. Here are some highlights worth mentioning.

StorageClient: Client Side Load Balancing

A technology specific problem solved, bypassing server based solution with client side load balancing and fast fail retry algorithms that took us from horrible to nearly 5 nines in reliability while improving overall performance. (This was at the day job, so that's about as much as I can share about that.)

LocalCache: In Memory Cache with Async Persistence

A library that takes advantage of Concurrent Collections in .NET and SQLite to provide fast in-memory caching that persists asynchronously on local disk for rapid rehydration of in-memory cache when an application pool is recycled. This solved a big problem with service level compliance recovery on a critical service, taking complete recovery time from hours to a few minutes. (Also for the day job.)

DuoVia.Net: TCP and NamedPipes Services Library

An extension and revival of RemotingLite that makes intra-process communication easy and fast. This was my first foray into creating and sharing open source software on GitHub and publishing packages on NuGet. I enjoyed it so much, I added 8 more packages to the set. And while these projects were built on my own time, one or two of them are in regular use by one or two teams at the current day job and they have been downloaded over 1,500 times.

VersionedCollections: A Shared Idea Brought to Life

Recently I shared an idea with Ayende Rahien on his blog with respect to creating a snapshot-in-time, read-only view of a collection that is being written to constantly. I'm happy to report that it turned out to be exactly what he needed. And I am honored and appreciative to Ayende for the kudos. Sharing good ideas with community friends is almost as much fun as bringing them to life yourself.

Here's to an equally fun and productive 2014.

Doors or Rooms?

Do you build rooms with doors or doors with rooms?

Should every room be the same size with doors of different size, shape and placement?

Are some doors in the floor or in the ceiling, at floor level or in the middle of the wall?

Do your doors open in our out? Are they too small? Too big?

Rooms are code that does stuff. Does work. Holds stuff. Creates stuff.

Doors are code that gets stuff into and out of rooms.

Which one do you spend more time working on?

Making Distributed Computing Relevant and Accessible

First, let us assume that distributed computing is generally that area of developing and running software designed to process large numbers of long running tasks on servers that are optimally proximal to the data being processed.

Second, let us agree, if for this discussion only, that distributed computing is NOT your collection of services on back end servers that support your service oriented architecture (SOA) for your web and mobile apps.

Third, let us presume that you are NOT already blessed with a job where you write distributed computing software.

How then can distributed computing be relevant to you? And how can you take advantage of distributed computing without becoming an expert in one of the several well known distributed computing platforms on the market today?

Both are excellent questions. Thank you for asking. Let’s try a practical approach.

Imagine you are at your desk and your boss comes to you and ask how fast your web servers respond to the customer. Of course, your first instinct is to write this program to find out:

private static void DoTenUrlsInParallel()
{
   Console.WriteLine("Do 10 urls in parallel");
   var sw = Stopwatch.StartNew();
   ISpeedTest test = new SpeedTest();
   Parallel.ForEach(TestUrls, (url) =>
   {
      var result = test.GetSpeed(url);
      Console.WriteLine("r:{0}, s:{1}, b:{2}, u:{3}",
         result.ResponseTimeMs, result.ReadStreamTimeMs, 
         result.ResponseLength, result.Url);
   });
   sw.Stop();
   Console.WriteLine("Total elapsed time: {0}", 
      sw.ElapsedMilliseconds);
   Console.WriteLine(string.Empty);
}

You take him the results and he says, “But isn’t this from your desk? I want to know what these numbers look like from all around the world. East and west U.S. North and west Europe. And east and south east Asia. And I want a regular stream of these numbers fed into a spreadsheet for me every day.”

Do you say, “No problem.” You do if you have a Windows Azure account and you know about the distributed task parallel library from DuoVia called DuoVia.Net.Distributed. You go back to your desk and modify the code to look like this:

private static void DoTenUrlsThreeTimesEachAroundTheWorldInParallel(bool runLocal = false)
{
   var serverEndpoints = new IPEndPoint[0];
   if (runLocal)
   {
      serverEndpoints = new IPEndPoint[] { new IPEndPoint(IPAddress.Parse("127.0.0.1"), 9096) };
   }
   else
   {
      //these server names are temporary - to run this test use your own
      var servers = new string[]
      {
         "myaz-westus.cloudapp.net",
         "myaz-eastus.cloudapp.net",
         "myaz-northeu.cloudapp.net",
         "myaz-westeu.cloudapp.net",
         "myaz-soeastasia.cloudapp.net",
         "myaz-eastasia.cloudapp.net"
      };

      serverEndpoints = new IPEndPoint[servers.Length];
      for (int i = 0; i < servers.Length; i++)
      {
         var host = Dns.GetHostAddresses(servers[i]);
         var ip = (from n in host 
                   where n.AddressFamily == AddressFamily.InterNetwork 
                   select n).First();
         serverEndpoints[i] = new IPEndPoint(ip, 9096);
      }
   }

   float subscriptionRate = 2.0f; //oversubscribed 
   int logPollingIntervalSeconds = 2;
   using (DistributedClient<ISpeedTest> client = 
          Distributor.Connect<ISpeedTest>(typeof(SpeedTest),
          subscriptionRate,
          logPollingIntervalSeconds,
          LogLevel.Debug,
          serverEndpoints))
   {
      for (int i = 0; i < 3; i++)
      {
         var sw = Stopwatch.StartNew();
         Console.WriteLine(@"round:{0}", i + 1);
         var loopResult = client.ForEach(TestUrls, (url, proxy) => proxy.GetSpeed(url));
         foreach (var result in loopResult.Results)
         {
            Console.WriteLine(@"r:{0}, s:{1}, b:{2}, on: {3}, u:{4}",
               result.ResponseTimeMs, result.ReadStreamTimeMs, 
			   result.ResponseLength, result.MachineName, result.Url);
         }
         sw.Stop();
         Console.WriteLine("Total elapsed time: {0}", sw.ElapsedMilliseconds);
         Console.WriteLine(string.Empty);
      }
   }
}

And you and your boss are happy.

Sometimes distributed computing is more about location and proximity to data or infrastructure than it is to getting massive amounts of data processed in as little time as possible.

You can find the full demo source code here.

Diversions in the D Programming Language

I am not a systems programmer, meaning I do not write operating system device drivers or file systems or operating system modules, etc, all written in a language that will compile down to raw machine code. I write in C# primarily which is arguably an applications programming language, running in the much loved .NET Common Language Runtime.

The vast majority of systems programming is done in C and C++. And for some reason, C++ has always been a daunting mess of libraries, odd syntax and pointer and memory allocation madness to me. Even setting up an environment to get the right build libraries, the right compiler and linker, etc., have always led me to fits of impatience. And for that reason and many others, I have stuck to C# and applications development.

But every once in a while, I look in on systems programming to see if anyone has really solved the problems I love to hate with respect to C and C++. And for a few years I’ve read a little about the D programming language here and there. A week ago, over the weekend, I decided to give it a try and really see what I could learn.

I have to say, I have been impressed. The D programming language offers a few things that I would dearly love to see in C#.

1. Exception Safety – the scope keyword

void abc() 
{ 
  auto resource = getresource();  // acquire some resource 
  scope(exit) resource.close();   // close the resource 
  doSomeProcessing();             // do processing
}

As C# programmers, we’re used to the try..catch..finally blocks. And we clean up a resource in the finally block. The trouble with that is many lines of code can end up separating your resource acquisition code from your resource cleanup code. Yes, with vigilance and well written tests, this is okay. But wouldn’t it be cool to be able to tell the compiler, “Hey, when I’m done with this thing I just now created, clean it up for me, no matter what code comes after this in this method.” I would love to see the scope keyword added to C#.

2. Concurrency approach

int perThread;
shared int PerProcess;

In C#, when you declare a class level variable, it is automatically shared between threads. You can use the [ThreadStatic] attribute to get a per thread instance of a given value or object. But it then has to be static. With the D programming language, you get thread safety in class variables. To override that safety, you have to explicitly tell the compiler you want the value shared. While I’m not advocating a change to C# in this regard, I would love to have a way to assure that a variable cannot be modified across thread boundaries.

3. Message based threads

import std.concurrency, std.stdio;
void main() {
   auto low = 0, high = 100;
   auto tid = spawn(&writer);
   foreach (i; low .. high) {
      writeln("Main thread: ", i);
      tid.send(thisTid, i);
      enforce(receiveOnly!Tid() == tid);
   }
}

void writer() {
   for (;;) {
      auto msg = receiveOnly!(Tid, int)();
      writeln("Secondary thread: ", msg[1]);
      msg[0].send(thisTid);
   }
}

For me, this is perhaps the coolest part of the D programming language’s base class library which they call Phobos. Note that main spawns a thread calling writer. The loop in main then sends a message to writer and the loop in writer receives the messages and operates on them and then sends a message back to the original thread.

You can learn a lot about D on www.dlang.org and read more about D concurrency on Informit. And if you want to play with D in Visual Studio, hop on over to see VisualD on dsource.org.

Choosing a Content Management System

Growth in the content management system (CMS) market is strong as more and more companies look for software to help them manage their burgeoning coffers of content. This growth is partly fed by the growing temptation to use these systems for purposes far beyond managing content with CMS vendors tacking on every new feature they can dream up to feed this frenzy. The result is often a bloated, unwieldy, unmanageable, complexity induced bug-infested mess.

I have a relatively high level of interest in this space given my brief professional experience in this market. So I read with interest a post today on the subject by one of my favorite software development authors, Martin Fowler. In his post entitled Editing Publishing Separation Fowler discusses the advantages of separating the editing data and experience from the publishing experience and published data. I invite you to read his article as he makes a very cogent argument which should be taken into account as you consider choosing a CMS.

Before he makes this point, he sets up his theme with this preamble:

"...a regular theme has been the growing impact of content management systems (CMS). They aren't usually seen as helpful, indeed there is a clear sign that they are becoming a worryingly invasive tool - used for more than their core purpose in such a manner that they hinder overall development."

It is precisely this thought which accurately represents my own limited experience in the CMS market. So here are my recommendations for consideration when choosing a CMS for your content management purposes.

  • As Martin suggests, look for a separate editing and publishing experience including separation of data and the advantages that can come from that.
  • And pay close attention to Martin’s opening quoted above. When looking at a feature, such as e-Commerce, consider whether you should be looking at a full featured e-Commerce platform rather than a feature that may have been cobbled onto the product as an afterthought. Remember that your primary purpose should be the management of content.
  • Whether you choose open or closed, open source or commercial, be sure that your own development team can understand and use the system with alacrity.
  • If you have a PHP team, don’t buy a .NET based CMS. And vice versa.
  • Take your time in evaluation. A quick, under-informed decision can cost you far more than you can imagine in future headaches.
  • Give your development team time to find the real problems with the CMS being evaluated and determine whether and how they can live with the problems they encounter.
  • Be sure to test the support and services teams of the vendor if you are going commercial over open source. Or if you are going open source, look for a proven and reliable partner who can guide you through the rough patches. Because no matter which CMS you choose, there will be some of those.
  • Consider licensing and support costs but don’t fall into the trap that open source is free. The efficiency of your own development and content management team using the CMS over the long haul will have a far larger impact on your overall cost of ownership.
  • Ask someone from Missouri to join your evaluation team. In other words, NEVER just accept the word of a sales representative that the CMS she is selling has the feature that you want and that it works as advertised. Like all software, there is a HUGE amount of advertising fluff and endless feature lists. Make sure that you have the sales team “show” you and then make sure that your own development team can make it work without their help and without having to contact support before you accept the claim, “Oh yes, our CMS does that.”
  • And last but not least, do your own research of satisfied and dissatisfied customers. Make some calls. Ask to speak to someone on the evaluation team where another product was chosen over the one you are considering. Ask if they evaluated the product you have your sights set on. Chances are that a few phone calls will land you a gold mine in helpful information.

Of course, you can apply these same principles to purchasing any enterprise software. And you should.

ASP.NET MVC 4 Beta Released

I’m excited to watch the rollout of ASP.NET MVC 4 Beta and looking forward to finding the time and a good solid project to take advantage of the new mobile support and Azure SDK.

Here’s a basic list of what’s new.

  • ASP.NET Web API
  • ASP.NET Single Page Application
  • Enhancements to Default Project Templates
  • Mobile Project Template
  • Display Modes
  • jQuery Mobile, the View Switcher, and Browser Overriding
  • Recipes for Code Generation in Visual Studio
  • Task Support for Asynchronous Controllers
  • Azure SDK

I want to hear your ASP.NET MVC 4 stories. Let me know what you think about it. You may get a chance to deep dive before I do, given my current project priorities.