How System.Net.Http 4.3.0 Ruined Everyone's Day

I have not updated the MessageWire library for about a year now. But it still works just fine. Mostly. New Year's resolution #1: Update MessageWire.

trustme

Dependency Hell

I'm cleverly using MessageWire to create a customized shared session service for a hybrid set of web applications that include old ASP.NET Web Forms running in .NET Framework 4.6.1 and newer, upcoming, web sites built on ASP.NET Core 2.0. So first it had to work with the older Web Forms site. And it did work. Very well. I'll share some of that fun code on another day in another post.

What did not go well is that my Web Forms site using a library that uses HttpClient in System.Net.Http to call an internal web service suddenly failed.

Here's the HTTP request before pulling MessageWire into the project.

POST http://localhost:53739/api/ProfileDetail HTTP/1.1
Accept: application/json
Content-Type: application/json; charset=utf-8
Host: localhost:53739
Content-Length: 592

{"Id": "…removed actual data…"}

And here's the same request after pulling MessageWire into the project with no other code changes.

POST http://localhost:53739/api/ProfileDetail HTTP/1.1
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
Accept: application/json
Accept-Encoding: gzip, deflate
Host: localhost:53739
 
250
{"Id": "…removed actual data…"}
0

See the problem? Yikes, my web service was unable to deserialize the second request created with HttpClient v4.3.0, so of course, that led to a huge failure.

Everyone's Worst Nightmare

What's worst about this is that I did not know about this side effect and this code made it into production causing a nightmare during a critical moment of the day and the month. Of course we rolled it back to stop the bleeding, but damage was done and folks were not happy with me in the least. It took me the rest of the day to find the symptoms of the cause which precipitated the failure.

The fact that it took me so long felt like another failure. Even with the final discovery that the HTTP request was borked, I was clueless as to its exact cause. I even engaged several team members to pour through my code to find the problem with the code I had committed. They could not find it.

So today, with less pressure on, I began methodically thinking of it and tracing back the code that generates the HTTP request. Ultimately it came down to HttpClient in System.Net.Http. This led me to discover that pulling in MessageWire into my ASP.NET Web Forms application on .NET Framework 4.6.1 also pulled in the flawed System.Net.Http version 4.3.0 package.

Light bulb moment! Check for updates. Sure enough, there was a System.Net.Http 4.3.3 available. And boom goes the dynamite! The request was back to looking normal and working just fine with the service being called.

MS OSS Buyer Beware!

When you pull in any new NuGet package which brings with it some dependencies, be sure you check out any updates to those dependencies. Shocking as it may seem, even Microsoft's packages can contain nasty little bugs. And test your assumptions.

Announcing MessageWire

During the holidays I began playing with a port of the Zero Knowledge code in ServiceWire but using NetMQ, the .NET native port of ZeroMQ. I’ve named this fun little project MessageWire and I’m happy to announce the first release and show off this new logo.

MessageWireLogo

MessageWire is a Zero Knowledge authentication and encryption wrapper for a NetMQ Dealer socket (client) Router socket (server) combination.

Get the code here. Get the NuGet package here. I’ll be blogging more about it as the code evolves. It’s early days so use at your own risk.

ServiceWire 5.0 with .NET Core Support Released

I’m happy to announce the release of ServiceWire 5.0 with .NET Core support published to NuGet today. The final trick was preparation of the NuGet package which was greatly helped by Armen Shimoon of dotnetliberty on using project.json for NuGet package metadata. The post was written in January 2016, so it was a bit out of date. The project.json file ended up like this:

{
  "name": "ServiceWire",
  "title": "ServiceWire",
  "authors": [ "Tyler Jensen" ],
  "description": "ServiceWire is a very fast...",
  "projectUrl": "https://github.com/tylerjensen/ServiceWire",
  "packOptions": {
    "iconUrl": "http://www.tsjensen.com/blog/image.axd?picture=2014/11/swlogo_sm.png",
    "licenseUrl": "http://www.apache.org/licenses/LICENSE-2.0",
    "copyright": "Tyler Jensen 2013-2016",
    "owners": [ "Tyler Jense" ],
    "summary": "ServiceWire is a fast and easy RPC library...",
    "releaseNotes": "BREAKING CHANGES: Ported to .NET Core netstandard1.6...",
    "tags": [ "WCF", "Services", "Host", "Client", "..." ]
  },
  "version": "5.0.0-*",

  "dependencies": {
    "NETStandard.Library": "1.6.0",
    "Newtonsoft.Json": "9.0.1"
  },

  "frameworks": {
    "netstandard1.6": {
      "imports": "dnxcore50",
      "dependencies": {
        "System.Reflection.Emit": "4.0.1",
        "System.Threading.Thread": "4.0.0"
      }
    },
    "net45": {
      "dependencies": {
        "System.Reflection.Emit": "4.0.1"
      },
      "frameworkAssemblies": {
        "System.Management": "4.0.0.0"
      }
    },
    "net461": {
      "dependencies": {
        "System.IO.Pipes": "4.0.0",
        "System.Reflection.Emit": "4.0.1",
        "System.Threading.Thread": "4.0.0"
      },
      "frameworkAssemblies": {
        "System.Management": "4.0.0.0"
      }
    },
    "net40": {
      "frameworkAssemblies": {
        "System.Management": "4.0.0.0"
      }
    },
    "net35": {
      "dependencies": {
        "TaskParallelLibrary": "1.0.2856"
      }
    }
  }
}

Then packaging up the NuGet package was easy with this command:

dotnet pack ServiceWire.Core -c Release -o D:\NugetPackages

Then just push it to NuGet. Simple as that.

And this concludes the porting of ServiceWire to .NET Core. But there’s more. There is always more. Some features were disabled in the NET Stardard 1.6 build and so a bit of utility work is still required. And there is some fun work ahead to make the library even easier to use. And then there is some performance work that out to be done along with some new unit tests using xUnit.

If you enjoy using ServiceWire, I’d love to hear from you. And I would love to have more pull requests from those who find bugs or ways to improve it.

.NET Core on Linux with ServiceWire

After a BIOS update, I was able to get Hyper-V working on my Windows 10 Pro machine and spun up an Ubuntu 16.04 instance. Once it was up and running I just followed the .NET Core Linux install instructions. No other changes to the VM.

Using the Portable Application instructions on the new docs.microsoft.com site for .NET Core, I executed the following commands to prep the CoreTestClient1 project for deployment to the Linux VM with my command prompt in the root directory of the project.

dotnet restore

and then

dotnet publish -f netcoreapp1.0 -c release

From there it was just a matter of copying the files in the \bin\Release\netcoreapp1.0\publish folder to a new folder called test1 on the Linux VM using an smb local file share.

After spinning up a debug instance of CoreTestHost on my Windows machine in Visual Studio, I tried to run CoreTestClient1 in the Ubuntu VM first without the extension. Oops. Then got it right and then tried it again just our of sheer delight.

linuxdotnet

The results are not particularly impressive as I was running on a VM with a single core, but the fact that it ran and without a hitch was enough to make me very happy. With a single day of porting to .NET Core and one more to remember how to install and use Linux on a Windows Hyper-V virtual machine and not a single bug that had to be fixed, I was able to complete the following essential requirements using only .NET Core, C# and ServiceWire

Requirements

  • Write an interface in a class library project called Common.
  • Write an implementation for that interface in class library project called Impl.
  • Write a console app that will host the implementation in a project called Host.
    • This project may reference Common and Impl.
    • Using ServiceWire allow remote calls to the implementation of the interface.
  • Write a console app called project Client.
    • This project may only reference Common.
    • Using ServiceWire, connect to Host and call the methods on the interface.
    • Write the results to the console.
  • Write all of this code on a single Windows machine with Visual Studio.
  • Run the Host project on the Windows machine.
  • Deploy the Client project to a Linux machine and run it from the Linux shell.
  • No tricks. You may not use Mono.

For examples of the code that you might find in the Host and Client, have a look at my previous post.

What’s Next?

Next I will be working on producing a NuGet package for ServiceWire 5.0.

Final Confession
It wasn’t quite all that smooth as I had not touched Linux or Hyper-V in years, so there were a few bumps, including a hosed up virtual network adapter that left my first Ubuntu install crippled. I also tried building and deploying a self contained app which required changes to the project.json file, but I gave it up as folly since I had already installed .NET Core on the Linux VM. And after deploying the portable app the first time, I was trying to run the app with “dotnet run” rather than “dotnet {assemblyFileName}” and of course that did not work. All told, I spent about 6 to 8 hours on all of this but that was broken up by multiple distractions so it required two calendar days. Now that the learning curve has been climbed, the next time out of the box should be much easier. And I hope this post will help you. I know it will serve well as a bookmark for me the next time I climb this curve. And that should be soon.

ServiceWire and .NET Core Integration Tests

I’m very happy with the quick and dirty .NET Core integration test code I’ve just committed to GitHub for ServiceWire. The only changes to existing integration test code were these:

Removal of Named Pipes from the test since .NET Core does not support Named Pipes (so far as I can learn) as Named Pipes is a Windows only thing (again, so far as I have learned).

Modified code to get IP address and port from command line args rather than configuration along with some defaults.

So here’s the primary host host code. Note how easy it is to host implementations for multiple interfaces.

class Program
{
   static void Main(string[] args)
   {
      var logger = new Logger(logLevel: LogLevel.Debug);
      var stats = new Stats();

      var addr = new[] { "127.0.0.1", "8098" }; //defaults
      if (null != args && args.Length > 0)
      {
         var parts = args[0].Split(':');
         if (parts.Length > 1) addr[1] = parts[1];
         addr[0] = parts[0];
      }

      var ip = addr[0];
      var port = Convert.ToInt32(addr[1]);
      var ipEndpoint = new IPEndPoint(IPAddress.Any, port);

      var useCompression = false;
      var compressionThreshold = 131072; //128KB

      var tester = new NetTester();
      var mytester = new MyTester();

      var tcphost = new TcpHost(ipEndpoint, logger, stats);
      tcphost.UseCompression = useCompression;
      tcphost.CompressionThreshold = compressionThreshold;
      tcphost.AddService<INetTester>(tester);
      tcphost.AddService<IMyTester>(mytester);

      var valTypes = new ValTypes();
      tcphost.AddService<IValTypes>(valTypes);

      tcphost.Open();

      Console.WriteLine("Press Enter to stop the dual host test.");
      Console.ReadLine();

      tcphost.Close();

      Console.WriteLine("Press Enter to quit.");
      Console.ReadLine();
   }
}

And here’s the client code.

class Program
{
   private static void Main(string[] args)
   {
      var addr = new[] { "127.0.0.1", "8098" }; //defaults
      if (null != args && args.Length > 0)
      {
         var parts = args[0].Split(':');
         if (parts.Length > 1) addr[1] = parts[1];
         addr[0] = parts[0];
      }

      var ip = addr[0];
      var port = Convert.ToInt32(addr[1]);
      var ipEndpoint = new IPEndPoint(IPAddress.Parse(ip), port);
      for (int i = 0; i < 1; i++) RunTest(ipEndpoint, ip);

      Console.ReadLine();
   }

   private static void RunTest(IPEndPoint ipEndpoint, string ip)
   {
      using (var client = new TcpClient<IValTypes>(ipEndpoint))
      {
         decimal abc = client.Proxy.GetDecimal(4.5m);
         bool result = client.Proxy.OutDecimal(abc);
      }

      using (var client = new NetTcpTesterProxy(ipEndpoint))
      {
         var id = client.GetId("test1", 3.314, 42, DateTime.Now);
         long q = 3;
         var response = client.Get(id, "mirror", 4.123, out q);
         var list = client.GetItems(id);
      }
      using (var client = new NetTcpMyTesterProxy(ipEndpoint))
      {
         var id = client.GetId("test1", 3.314, 42);
         int q2 = 4;
         var response = client.Get(id, "mirror", 4.123, out q2);
         var list = client.GetItems(id, new int[] { 3, 6, 9 });
      }

      var sw = Stopwatch.StartNew();
      var from = 0;
      var to = 400;
      Parallel.For(from, to, index =>
      {
         using (var client = new NetTcpTesterProxy(ipEndpoint))
         {
            for (int i = 0; i < 10; i++)
            {
               var id = client.GetId("test1", 3.314, 42, DateTime.Now);
               long q = 2;
               var response = client.Get(id, "mirror", 4.123, out q);
               var list = client.GetItems(id);
            }
         }

         using (var client = new NetTcpMyTesterProxy(ipEndpoint))
         {
            for (int i = 0; i < 10; i++)
            {
               var id = client.GetId("test1", 3.314, 42);
               int q2 = 6;
               var response = client.Get(id, "mirror", 4.123, out q2);
               var list = client.GetItems(id, new int[] { 3, 6, 9 });
            }
         }
      });
      sw.Stop();
      var msperop = sw.ElapsedMilliseconds / 24000.0;
      Console.WriteLine("tcp: {0}, {1}", sw.ElapsedMilliseconds, msperop);
   }
}

It’s quite simple, but I’m going to be working on both host and client code to make it easier to code up both of them with some convention assumptions that will also not break existing host and client code. After that, I’ll publish a new NuGet package with a major version rev to 5.0.

ServiceWire on .NET Core

With a few days of free time on my hands, I’ve picked up an old open source project of mine and kicked off a .NET Core port of ServiceWire which still needs testing and much more refactoring, I'm sure. Just getting everything to build was a challenge. Especially in the area of Reflection. Here’s the bridge code I wrote to avoid making to many IFDEF blocks in the code.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using System.Reflection.Emit;
using System.Threading.Tasks;

namespace ServiceWire
{
   public static class TypeExtensions
   {
      public static Type BaseType(this Type t)
      {
#if NETSTANDARD1_6
         return t.GetTypeInfo().BaseType;
#else
         return t.BaseType;
#endif
      }

      public static bool IsInterface(this Type t)
      {
#if NETSTANDARD1_6
         return t.GetTypeInfo().IsInterface;
#else
         return t.IsInterface;
#endif
      }

#if NETSTANDARD1_6
      public static MethodInfo[] GetMethods(this Type t)
      {
         return t.GetTypeInfo().GetMethods();
      }
#endif

#if NETSTANDARD1_6
      public static Type[] GetInterfaces(this Type t)
      {
         return t.GetTypeInfo().GetInterfaces();
      }
#endif

#if NETSTANDARD1_6
      public static Type CreateType(this TypeBuilder b)
      {
         return b.CreateTypeInfo().AsType();
      }
#endif

#if NETSTANDARD1_6
      public static ConstructorInfo GetConstructor(this Type t, Type[] ctorArgTypes)
      {
         return t.GetTypeInfo().GetConstructor(ctorArgTypes);
      }
#endif

#if NETSTANDARD1_6
      public static MethodInfo GetMethod(this Type t, string invokeMethod, BindingFlags flags)
      {
         return t.GetTypeInfo().GetMethod(invokeMethod, flags);
      }

      public static MethodInfo[] GetMethods(this Type t, BindingFlags flags)
      {
         return t.GetTypeInfo().GetMethods(flags);
      }

      public static PropertyInfo[] GetProperties(this Type t, BindingFlags flags)
      {
         return t.GetTypeInfo().GetProperties(flags);
      }

#endif

      public static bool IsValueType(this Type t)
      {
#if NETSTANDARD1_6
         return t.GetTypeInfo().IsValueType;
#else
         return t.IsValueType;
#endif
      }

      public static Type[] GetGenericArguments(this PropertyInfo pi)
      {
#if NETSTANDARD1_6
         return pi.PropertyType.GetTypeInfo().GetGenericArguments();
#else
         return pi.PropertyType.GetGenericArguments();
#endif
      }

      public static bool IsGenericType(this Type t)
      {
#if NETSTANDARD1_6
         return t.GetTypeInfo().IsGenericType;
#else
         return t.IsGenericType;
#endif
      }

      public static bool IsPrimitive(this Type t)
      {
#if NETSTANDARD1_6
         return t.GetTypeInfo().IsPrimitive;
#else
         return t.IsPrimitive;
#endif
      }

   }
}

It’s a quick and dirty implementation that needs more work. The split of Type into Type and TypeInfo classes in .NET Core is a pain but I can see why they would want to eliminate Reflection except where it is absolutely necessary.

Token Authentication with ASP.NET Core

I’ve spent a lot of time over the last 16 months or so working with ASP.NET Web API and Microsoft.Owin’s UseOAuthAuthorizationServer middleware extending it with custom OAuthAuthorizationServerOptions, ISecureDataFormat<T>, OAuthAuthorizationServerProvider and IAuthenticationTokenProvider implementations.

This included the use of the System.IdentityModel.Tokens InMemorySymmetricSecurityKey used in the SigningCredentials constructor to be used in signing the JWT token. And when I went to port all of this to ASP.NET Core, I learned to my surprise that there is no equivalent to UseOAuthAuthorizationServer middleware and there is no InMemorySymmetricSecurityKey. Instead, you’re on your own for creating your own authentication/authorization middleware and signing a JWT is done using a SymmetricSecurityKey (a class that used to be abstract but now is not).

Here’s my first attempt:

salt = salt ?? "0a987sdf7asdg896asdf6as9df7a7sdf8asd";
var keyBytes = Encoding.UTF8.GetBytes(
    Convert.ToBase64String(Encoding.UTF8.GetBytes(SecKey + salt)));
_symmetricSecurityKey = new SymmetricSecurityKey(keyBytes);
_tokenValidationParameters = new TokenValidationParameters
{
    CryptoProviderFactory = _symmetricSecurityKey.CryptoProviderFactory,
    ValidateIssuerSigningKey = false, //true,
    IssuerSigningKey = _symmetricSecurityKey,
    ValidateIssuer = true,
    ValidIssuer = "issuername",
    ValidateAudience = true,
    ValidAudience = "all",
    ValidateLifetime = true,
    ClockSkew = TimeSpan.Zero
};
_signingCredentials = new SigningCredentials(_symmetricSecurityKey, "HS256");

A very good blog post on this topic can be found on Stormpath’s blog. I highly recommend you read that blog post. There are any number of ways to code up your “token” endpoint. I like how easy it is to write a custom authentication and authorization token endpoint using ASP.NET Core. More fun with .NET Core to follow.