Code Outside of the Box

Hydrogen Extensions for ASP.NET MVC 5 Alpha Release

2017-03-20T00:00:00+00:00

I’m happy to share the first alpha release of my extensions library for ASP.NET MVC 5 which adds support for async filters.

You can download the library from NuGet and the source is available on GitHub.

You can read about my journey to get here:

Getting started

First we need to swap out the default implementation IAsyncActionInvoker with Hydrogen.Extensions.Mvc5.Async.ControllerActionInvokerEx. There are two ways to do this:

1) Set the ActionInvoker property of a controller when it’s constructed:

public class HomeController : Controller
{
    public HomeController()
    {
        ActionInvoker = ControllerActionInvokerEx.Instance;
    }
}

2) Using your DI container of choice, register Hydrogen.Extensions.Mvc5.Async.ControllerActionInvokerEx as an implementation of IAsyncActionInvoker.

For example (using Autofac):

builder.RegisterInstance(ControllerActionInvokerEx.Instance)
        .As<IAsyncActionInvoker>()
        .SingleInstance();

Note that ControllerActionInvokerEx can be used as a singleton. It doesn’t maintain any state.

Creating an async filter

You may create a subclass of AsyncActionFilterAttribute or AsyncExceptionFilterAttribute and override the async methods.

For example:

public class MyAsyncActionFilterAttribute : AsyncActionFilterAttribute
{
    public override async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
    {
        // Execute code before the action is invoked.
        // You can "short-circuit" the action by setting 'context.Result'
        // and returning before calling 'next()'.
        
        // The implementation is responsible for calling 'next()'.
        var actionExecutedContext = await next().ConfigureAwait(false);

        // Execute code after the action is invoked
    }

    public override async Task OnResultExecutionAsync(ResultExecutingContext context, ResultExecutionDelegate next)
    {
        // Execute code before the result is invoked.
        // You can "short-circuit" the action by setting 'context.Canceled = true'
        // and returning before calling 'next()'.
        
        // The implementation is responsible for calling 'next()'.
        var resultExecutedContext = await next().ConfigureAwait(false);

        // Execute code after the result is invoked
    }
}

Alternately you may implement any of the async filters:

IAsyncActionFilter
IAsyncResultFilter
IAsyncExceptionFilter
IAsyncAuthorizationFilter

Unfortunately, for compatibility reasons, you must also implement the non-async version of these filters (e.g. void OnActionExecuting(ActionExecutingContext filterContext)). However, these methods can be a NOOP since the code is never executed by ControllerActionInvokerEx.

This library, currently, does not implement as async version of IAuthenticationFilter.

This is an alpha release

I fully expect there to be bugs, however I hope to use it in real production code soon.

Maybe one day this code, or some derivation there of, can make it into an official release.

I hope someone else might find this library useful. And I’d appreciate feedback, comments, or contributions (including bug reports)!

Improving Async Support in ASP.NET MVC - Part 3 - Async Filter API

2017-03-18T00:00:00+00:00

The obvious solution is to just copy the API from ASP.NET MVC Core.

public interface IAsyncAuthorizationFilter : IAuthorizationFilter
{
    Task OnAuthorizationAsync(AuthorizationContext context);
}

public interface IAsyncExceptionFilter : IExceptionFilter
{
    Task OnExceptionAsync(ExceptionContext context);
}

public interface IAsyncActionFilter : IActionFilter
{
    Task OnActionExecutionAsync(ActionExecutingContext context
        , ActionExecutionDelegate next);
}

public delegate Task<ActionExecutedContext> ActionExecutionDelegate();

public interface IAsyncResultFilter : IResultFilter
{
    Task OnResultExecutionAsync(ResultExecutingContext context, ResultExecutionDelegate next);
}

public delegate Task<ResultExecutedContext> ResultExecutionDelegate();

And that’s basically it.

But you’ll notice that these async interfaces must also implement their non-async counterparts. Unfortunately MVC5 doesn’t have a shared marker interface as exists in MVC Core. And so the framework (and many 3rd party extensions) directly reference the filter types for various reasons. This means that implementations of the async interfaces will just need to implement the non-async methods as a NOOP - annoying but not the end of the world.

Here’s an example implementation based on FilterAttribute:

public class AsyncActionFilterAttribute : FilterAttribute
    , IAsyncActionFilter
    , IAsyncResultFilter {

    public virtual void OnActionExecuting(ActionExecutingContext filterContext) {
    }

    public virtual void OnActionExecuted(ActionExecutedContext filterContext) {
    }

    public virtual async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next) {
        OnActionExecuting(context);
        if (context.Result == null) {
            OnActionExecuted(await next().ConfigureAwait(false));
        }
    }

    public virtual void OnResultExecuting(ResultExecutingContext filterContext) {
    }

    public virtual void OnResultExecuted(ResultExecutedContext filterContext) {
    }

    public virtual async Task OnResultExecutionAsync(ResultExecutingContext context, ResultExecutionDelegate next) {
        OnResultExecuting(context);
        if (!context.Cancel) {
            OnResultExecuted(await next().ConfigureAwait(false));
        }
    }
}

Next time, putting everything together.

Improving Async Support in ASP.NET MVC - Part 2 - Refactoring AsyncControllerActionInvoker

2017-03-13T00:00:00+00:00

Last time I began by discussing several of the shortcomings of the current ASP.NET MVC async implementation. Specifically the lack of async filters (action filters, etc.).

I’ve decided that the most straight-forward approach is implement a refactored AsyncControllerActionInvoker. This is the component within the MVC pipeline responsible for invoking filters.

I’ve started a new project, MvcAsync, for the refactored code.

Refactoring APM to TAP

As I mentioned, most of the MVC pipeline is written using the classic Asynchronous Programming Model (APM) which requires Begin/End method pairs. This results in very convoluted code that is difficult to reason about.

Fortunately it is possible to wrap APM using the Task-based Asynchronous Pattern (TAP). Stephen Cleary has written and great deal about the topic and has some wonderful resources.

You can compare my implementation to the original to see just how stark the difference is.

And because I wrapped the incoming and outgoing APM, I was largely able to re-used the existing unit tests to assert that my implementation is correct (for now).

Performance

By wrapping both in the incoming and outgoing APM methods in Tasks, I was originally worried about performance. So of course I decided to create a benchmark (using the fantastic BenchmarkDotNet library).

I was pleasantly surprised by the results. Performance is nearly the same, and sometimes better in my Task-based implementation.

                                                        Method |       Mean |    StdDev |
-------------------------------------------------------------- |----------- |---------- |
 AsyncControllerActionInvokerEx_BeginInvokeAction_NormalAction | 17.5357 us | 0.5481 us |
      AsyncControllerActionInvokerEx_InvokeAction_NormalAction | 16.9045 us | 0.2451 us |
   AsyncControllerActionInvoker_BeginInvokeAction_NormalAction | 17.1643 us | 0.1946 us |

Of course I’m assuming there would be an even greater improvement in performance if the entire pipeline could be refactored to use TAP. Unfortunately that would require a lot of changes, many of them breaking changes.

Using `AsyncControllerActionInvokerEx` at Runtime

Fortunately it’s pretty easy to swap in a custom ActionInvoker at runtime. As a developer you have a few options:

You can create an implementation of IAsyncActionInvokerFactory or IActionInvokerFactory and register with your DI container.
You can directly register an IAsyncActionInvoker or IActionInvoker with your DI container.
You can simply assign the ActionInvoker property on Controller in a constructor.

Next Time

Next time I hope to start planning out the API for new async filters.

Improving Async Support in ASP.NET MVC - Part 1

2017-03-12T00:00:00+00:00

The Current Landscape

ASP.NET MVC (the non-core version) has enjoyed decent support for asynchronous actions. The older Asynchronous Programming Model (APM) approach required inheriting from AsyncController and involved creating ActionAsync() and ActionCompleted() method pairs. The newer Task-based Asynchronous Pattern (TAP), or async/await, approach simplified development significantly by allowing action methods on normal controllers to return Task<ActionResult>. On the one hand, the implementation of async inside ASP.NET MVC has been kludgey at best. On the other hand, Microsoft did a good job of tying in async support without any major breaking changes.

Despite this, there are a number of use cases that are simply not supported making it really challenging to create async-only business libraries.

Asynchronous filters (action filters, authentication filters, etc.)
Asynchronous action results (such as rendering Razor views)
Asynchronous child actions (resulting from lack of async action results)

Granted these problems and more are solved in ASP.NET Core. But a lot of world runs on a mix of legacy code, including web forms and old ASP.NET web services. It would have been nice to have some of these features back ported to the ASP.NET MVC 5.x trunk.

And so I am attempting to just that. I’m also planning to document the process. Whether or not these changes ever make it into an “official” release, I think this is a worth-while exercise that hopefully someone finds useful.

The ASP.NET MVC Async Pipeline

Supporting asynchronous action filters is right at the top of my list. All of the code today used to invoke action filters lives in AsyncControllerActionInvoker, so I figured it wouldn’t be too hard. Unfortunately that’s where discovered that the MVC pipeline isn’t built around async/await, but rather classic APM.

Let’s look at how the pipeline is layered today:

// This is the main entry point after a request is matched to a route
// (also called when rendering a child action)
class MvcHandler : IHttpAsyncHandler {
    IAsyncResult BeginProcessRequest(...)
    void EndProcessRequest(IAsyncResult)
}

// Called from MvcHandler
class Controller : IAsyncController {
    IAsyncResult BeginExecute(...)
    void EndExecute(IAsyncResult)
}

// Called from Controller
// Invokes MVC filters and executes ActionDescriptor
class AsyncControllerActionInvoker : IAsyncActionInvoker {
    IAsyncResult BeginInvokeAction(...)
    bool EndInvokeAction(IAsyncResult)
}

// Called from AsyncControllerActionInvoker
// Different concrete implementation based on whether it's a TPL or APM action
abstract class AsyncActionDescriptor {
    IAsyncResult BeginExecute(...)
    object EndExecute(IAsyncResult)
}

I want to rewrite AsyncControllerActionInvoker so that its internal logic is based on async/await, which will hopefully clean up the code quite a bit. But I suspect that wrapping Task/APM at both the entry point and when calling to AsyncActionDescriptor will result in some performance loss. Which means I might as well rewrite the whole stack. But then this is a breaking change (like ASP.NET Core).

Getting Started

The source code today is still hosted on https://aspnetwebstack.codeplex.com/. I decided to fork the repository and push my work to a branch on GitHub: https://github.com/jdaigle/aspnetwebstack/tree/async-pipeline-cleanup.

I’m not sure yet if I should modifying the code in System.Web.Mvc and recompile, or if I should try implementing the stack in an separate assembly and replace at runtime (introducing breaking changes will obviously have an effect on that).

Next time I hope to have a working build of the revised async pipeline (with all existing unit tests passing). And then I’ll take on designing an API for the async filters.

But in the meantime if anyone has any comments or feedback please feel free to leave them below or contact me.

Update: 2017-03-13

It seems that Microsoft is, in fact, working on a 5.2.4 release. And there will be a new home for AspNetWebStack on GitHub eventually. In fact, you can find the nightlys of 5.2.4 on MyGet. So maybe I could get some of my changes and ideas accepted once development is open again.

Honestly I’m not sure why Microsoft isn’t doing 5.2.4 in the open already - it’s a bit disheartening.

Update 2017-05-07

The AspNetWebStack repository is finally open.

"Code-First" Migrations with Horton

2016-09-12T00:00:00+00:00

Ask any MS SQL Server database expert about Entity Framework, particularly “code-first” migrations, and you’ll get responses like this:

@JosephDaigle "Code First" = Database Last.
— Geoff Hiten (@SQLCraftsman) August 10, 2016

It’s true. The entire concept of “code-first” can be distilled into one simple description; your application code generates your database schema. If you’re using EF migrations, then the typical workflow is to run add-migration which compares your DbContext’s current model with what it thinks is the previous model (based on the last migration applied to the database) and generates a new migration from the difference.

There are two fundamental problems with this “code-first” approach. First, EF generates awful defaults. For example, it’ll create an index for each foreign key of a table (whether you need it or not!). It also generates FK constraints with ON DELETE CASCADE, something I rarely want in my schema. Developers will almost never change these defaults which can result in sub-optimal schemas.

This directly leads to the other fundamental problem; auto-generated migrations allow developers to further distance themselves from the database. It’s easy to blindly trust the schema that EF generates. And ultimately I think this does actual harm by abstracting something (the database schema) which shouldn’t be abstracted in the first place!

There a lot of other problems with EF migrations in general (such as poor tooling, lack of idempotency, and the requirement to implement “Down” migrations), but that’s content for another article.

Scaffolding Migrations with Horton

Historically I’ve frowned upon scaffolding migrations. Despite this, I still often find myself copying/pasting existing migrations and changing names and identifiers - it’s so much easier to start from something rather than nothing. Also, teams that use EF “code-first” migrations are accustomed to always scaffolding (in fact, there is no other way to add a migration).

And so I think there might be a gap in a lot of SQL-migration-based tooling: a way to scaffold a new migration to reduce time and errors. But the keyword is “scaffold”: the migration should always be reviewed and cleaned up before committing.

To this end, I’ve added an experimental feature to Horton to scaffold new database migrations.

The code lives in a branch, with the most important class being the EF DiffTool.

This feature uses EF’s “Metadata Workspace” to analyze the mapping entity model and compare it to a physical database schema. Specifically, the tool looks for new tables and schemas, new or altered columns, and new possible FK constraints.

Why not dropped objects? Well, just because something was removed from the code doesn’t mean it should be removed from the database. This I think is another fundamental problem with EF’s “code-first” migrations.

How does it work? Horton has learned a new command named ADD-MIGRATION. You simply execute this command, and if any changes are detected then it will prompt for the “name” of the new migration to write. The migration is written to your migration directory and given the next sequence number.

There are a couple of other minor points:

The command works from a strict set of conventions:
Your concrete DbContext implementation must have a constructor which accepts a connection string or connection string name.
The path to the assembly containing the DbContext is defined in a dbcontext.path file in the root of your migrations directory. Relative paths are allowed, and encouraged.

Since there is a dependency on EF, I didn’t want to add this feature to the core Horton.exe engine. And so I quickly invented a command plugin system. Commands are dynamically added at runtime via reflection. Right now it’s really simple: Horton simply scans its app domain directory for .NET assemblies and registers any new command it finds.

A final word: this is an experiment and has not been used for production purposes yet. There are still some issues to work out, particularly regarding “legacy” schema and models.

Code Organization, CQRS Style

2016-06-27T00:00:00+00:00

I am a huge believer in the Command/Query pattern for my applications’ internal services and APIs, particularly for those services that touch a database.

But I’ve been constantly challenged with code organization. What is the right approach for a particular project and team?

I think you might find a lot of .NET developers, especially those that use ASP.NET MVC, lean towards code organization by function. What does this mean?

So in ASP.NET MVC, we have separate folders (and therefore namespaces) for Controllers, Views, and pretty much anything else like our Model, ActionFilters, Helpers, etc. This kind of insanity often manifests itself in other parts of the code base. Therefore we may end up with something like this:

MyServiceProject/
├── Commands/
│   ├── SomeFeature/
│   │   ├── FeatureUseCaseCommand.cs
├── CommandHandlers/
│   ├── SomeFeature/
│   │   ├── FeatureUseCaseCommandHandler.cs
├── Queries/
│   ├── SomeFeature/
│   │   ├── FeatureUseCaseQuery.cs
├── QueryHandlers/
│   ├── SomeFeature/
│   │   ├── FeatureUseCaseQueryHandler.cs
│ ... etc...

Obviously I would prefer to invert the hierarchy and organize the code by feature. For example:

MyServiceProject/
├── SomeFeature/
│   ├── Commands/
│   │   ├── FeatureUseCaseCommand.cs
│   ├── CommandHandlers/
│   │   ├── FeatureUseCaseCommandHandler.cs
│   ├── Queries/
│   │   ├── FeatureUseCaseQuery.cs
│   ├── QueryHandlers/
│   │   ├── FeatureUseCaseQueryHandler.cs
│ ... etc...

That’s a bit better. I’ve also tried flattening the hierarchy:

MyServiceProject/
├── SomeFeature/
│   ├── FeatureUseCaseCommand.cs
│   ├── FeatureUseCaseCommandHandler.cs
│   ├── FeatureUseCaseQuery.cs
│   ├── FeatureUseCaseQueryHandler.cs
│ ... etc...

While this has less ceremony, the code itself can really start to feel cluttered.

But one day I had a thought: why does the Command and CommandHandler need to be in separate files? They’re likely to change together, so it’s not a violation of the Single Responsibility Principle. and you can easily define two classes in the same file. So now we have this:

MyServiceProject/
├── SomeFeature/
│   ├── FeatureUseCaseCommand.cs
│   ├── FeatureUseCaseQuery.cs
│ ... etc...

That’s pretty clean. I’m also experimenting with nested types:

public static class FeatureUseCase {

    public sealed class Command : ICommand {
        public string Data { get; set; }
    }

    public sealed class Handler : ICommandHandler<Command> {
        public void Handle(Command command) {
            // handle command
        }
    } 
}

This, I think, is very cool. Instead of having FooCommand and FooCommandHandler, I simply have a class named Foo with Command and Handler as nested types. Functionally it’s no different from the former design. But maybe this reads better. Additionally, if I later decide to add something like validation, I can simply add whatever classes/code I need as a new nested types. The related code lives together in a single code file, and my project is clean.

Command Pattern Therapy

2016-05-17T00:00:00+00:00

I was planning to write a lengthy article arguing about the merits of the command pattern and how it relates to “clean” web application design. But while researching for this article I rediscovered a series by Jimmy Bogard entitled Put your controllers on a diet:

Take 15 minutes and read them. Seriously, I’ll wait. By and large these articles cover pretty much every argument I would make for restructuring your web application and business logic API into sets of commands and queries.

Here’s a quick recap of the the most important take-aways from his series:

The controller is a “lousy” place for business logic. And terrible for unit testing. We want them to be slim and basically only responsible for converting data to and from requests and responses.
Don’t further abstract existing abstractions. I.e., don’t bother creating wrappers around your ORM. And please don’t create an IRepository<T> or IUnitOfWork.
GET/POST should conform to semantics. A POST is a “command” and a GET is a “query”. Queries should have no side effects (other than logging, etc.) and return data. A command may mutate state, and returns a status code and sometimes a resource identifier.
Design your business logic API using objects representing “command” and “query” operations. These operations are handled by a mediator with a single interface.

I want to focus on that last point: building a single service interface. Far too many applications I’ve seen suffer from a new type of “dependency hell”. It’s not uncommon to see a Controller with a constructor that has literally dozens of “service” and “repository” interface parameters.

Interfaces are terrible for designing a constantly evolving businesss API.

Interfaces have a number of problems. When I’m adding a new operation, should I create a new interface or should I add it to an existing interface? If so, which one?

Adding a member to an interface is a breaking change. Granted, in most applications there is only a single implementation (which begs the question, why do we even need the interface?). But often we create mock/stub implementations for unit tests. And all of those tests need to be updated even if we’re not touching those features!

It’s usually quite difficult to implement cross cutting concerns with interface-based services. Such as database transactions (you are using explicit database transactions right?) and auditing/logging. There are some neat proxy frameworks out there that generate runtime code which proxy the interface calls and allow you to implement these cross cutting concerns. But it doesn’t feel right to me.

In my view, the command pattern is ideal as it clearly represents a pipeline:

HTTP POST → Controller Action → Create Command → Execute → [stuff] → Handler → Return

The POST is handled by a controller action. The controller action is responsible for a) creating a command and b) passing it to the command processing engine. The command processing engine executes a pre-configured pipeline of “behaviors”. You can think of these as not unlike ActionFilters in ASP.NET MVC. These can be anything from starting a database transaction/unit of work, to security checks, to logging and auditing. Ultimately the associated command handler method is called, passing the command object as a parameter.

In fact, you could go so far as to allow ASP.NET MVC to create the command object and bind the properties:

[HttpPost]
public ActionResult CreateCustomer(CreateCustomerCommand command) {
    command.Validate(); // ensure the command properties are all valid from the POST
    CommandProcessor.Execute(command);
    return OK;
}

How awesome is that? And the best part is that adding new features typically doesn’t require modifying existing code. Instead we’re just adding new classes.

In fact, we can take it further and get rid of our dependency on a dependency injection framework. But that’s an argument best left for another article.

I wouldn’t necessarily always design my application this way. It can be overkill for small apps that rarely change. However, for larger applications that are constantly changing, the command pattern is invaluable. Software architecture is all about managing dependencies and churn. How easily can you add new features without touching existing code or introducing new dependencies?

I strongly recommend that you consider using a single command/query framework for your business logic services, such as MediatR. Or write your own, it’s not that hard and can be fun.

Evolving Command Handler Design

2016-05-12T00:00:00+00:00

It’s no surprise that I’m a big fan of the command and mediator patterns when building business logic and service APIs. Particularly when compared to the traditional approach to abstracting services: C# interfaces with methods.

It’s awesome to be able to reduce all of your service interfaces down to just one:

public interface IServiceAPI
{
    TResult Execute<TCommand>(TCommand command);
}

But this article is not about the merits of the pattern, or how to design your API layer. This article is going focus on my evolving ideas around implementing the actual command “handler” logic.

In the beginning

My very first technique is borrowed directly from NServiceBus when I first introduced to it seven years ago.

public interface ICommandHandler<TCommand, TResult>
{
    TResult Execute(TCommand command);
}

public class MyCommandHandler : ICommandHander<MyCommand, MyCommandResult>
{
    public MyCommandHandler(DbContext ... etc)
    {
        // ... set instance variables
    }
    
    public MyCommandResult Execute(MyCommand command)
    {
        // ... do stuff, access instance variables/etc.
    }
}

It’s very simple: each command is handled by a class that implements ICommandHandler<TCommand>. A particular class can implement multiple command handlers, however in practice Single Responsibility Principal (SRP) kicks in and you would typically create a separate class per command.

Dependency injection in very straightforward too. Dependencies are supplied to the class’ constructor at runtime; often the command handler itself is constructed and managed by some IoC framework.

During application startup the bootstrapping system will scan for non-abstract types that implement the command handler interface and register them against the command types. At runtime, the class is dynamically created and the method invoked. It’s no surprise that this approach is quite similar to how ASP.NET MVC controllers and actions work. In fact I’ve borrowed techniques from its code in my own frameworks.

So while the code is simple to understand, it is a lot of ceremony and boilerplate.

Enter functions

Consider this: an interface that defines only a single method is not an interface; it is a delegate. Why do we need to instantiate a class just to call our command handler function? What if we designed it just as a static method?

public static MyCommandResult Execute(MyCommand command)
{
    // do stuff
}

We can simply declare, somewhere, a static method that matches our “interface”, or delegate: public delegate TResult Execute<TCommand>(command);. Now it’s trivial to wire up code that executes appropriate delegate when a command of type TCommand is supplied.

That’s great, but what about my dependencies? Functional programming has solved this with currying.

public static MyCommandResult Execute(MyCommand command, IDependency foo, IOtherDependency bar)
{
    // do stuff
}

The idea is that our framework will, at runtime, compile a new expression that calls our command handler method with resolved dependencies. That compiled expression will match the common delegate. Again, this is nothing new - ASP.NET MVC does this with model binding of action parameters. And we have one fewer allocation since we’re not creating an instance of a class just to call a method.

As far as automatically registering command handlers, there are a few techniques I’ve used the past. But the simplest has been to decorate each command handler method with an attribute, such as [CommandHandlerAttribute]. This makes it super easy, at runtime, to scan and find the matching methods.

I’ve experimented with static registration techniques, but they’re not pretty. I think this is mostly due to limitations in the expressiveness of C#. You can find an example here.

Now we have lightweight static methods that are dynamically invoked. But where do we put them? Unfortunately for .NET, all methods including static methods must be declared on a class. Do we create one static class per handler? Do we put them all in one giant class? How do we group them? How should we name the static class? I suppose this is a really minor concern, but it annoys me because naming and organization can be difficult.

The next generation

While thinking about the “problem” of organizing my command handlers, I was recently made a connection to how frameworks like NancyFx define modules. If you look at a typical Nancy module you might notice that, in a way, it’s just defining and mapping command handlers (pedantically speaking they’re “HTTP request” handlers, but conceptually the same).

What if instead of a “NancyModule” we could build a “CommandModule”?

Caveat: this is all experimental. I’ve not tried this technique in production code.

public class ProductModule : CommandModule
{
    public ProductModule(IFoo foo, IBar bar)
    {
        Handle<AddProductModel>(command =>
        {
            foo.Something();
            return 1;
        });

        Handle<AddProductReview>(command =>
        {
            foo.Something();
            bar.SomethingElse();
        });
    }
}

The way this, like NancyFx, works: on startup “modules” are discovered and initially constructed. This first pass results in a registry that maps the “request” (command type in my case) to a particular module and lambda. At runtime, when a “request” (i.e. command) is executed, the appropriate module is constructed and the corresponding lambda is executed.

There are some interesting properties about this design:

The command handler lambda can compile with a closure around any referenced dependencies, which are passed via the module constructor.
When the command is executed, we do instantiate an instance of the module (one per “request”).
If the lambda was compiled with a closure, then another class is instantiated (one per “request”) with the captured references.

That is extra overhead, but maybe it’s a good tradeoff in order to have more expressive code? This seems like the perfect way to organize this code. Each module can be built with a clear and single purpose. I can use existing IoC frameworks to handle module construction and dependency injection. The framework itself becomes a lot simpler.

Some brainstorming: Maybe there is a way to compile an expression which skips the module construction at runtime. We know about the about the MethodInfo of the lambda. So we know about the anonymous DeclaringType. If the DeclaringType is for a closure, we can inspect it’s fields. At runtime, we can construct that object and populate the fields from an IoC container. Then use that call the method. So that’s one fewer allocation.

Wrapping up

Over the years my ideas and design for command handlers has evolved.

First were full-blown classes implementing a interface.
Then I reduced those down to simple static methods that live somewhere.
Finally, I’ve experimented with “modules” that declare command handlers as lambdas.

Each technique has pros and cons. Different applications, or contexts within a single application, may be suited to one technique over another. Some teams might be comfortable with one technique over another. But I’m curious about you, dear reader, and what technique you like and why?

The Modern Joel Test

2016-04-13T00:00:00+00:00

The Joel Test is a classic article written Joel Spolsky that describes 12 criteria often cited as useful to prospective employees when looking for a new software job.

The Joel Test

Do you use source control?

Can you make a build in one step?

Do you make daily builds?

Do you have a bug database?

Do you fix bugs before writing new code?

Do you have an up-to-date schedule?

Do you have a spec?

Do programmers have quiet working conditions?

Do you use the best tools money can buy?

Do you have testers?

Do new candidates write code during their interview?

Do you do hallway usability testing?

It was written nearly 16 years ago and is still applicable today. For instance, you’d be surprised how many software shops still don’t use source control.

But for contemporary software development, I think the test could use a few updates.

1. Do you use modern source control?

These days it’s not always enough to use source control. Developers often expect or demand to work with the best tools available. Such as Git or Mercurial (if anyone still uses that). You’ll find a lot of shops still using SVN or TFSVC. And that’s fine, but I think there are huge productivity gains to be had by adopting Git or Hg.

2. Can you make a build in one step?

Nothing new here. Except that it’s common for builds to be scripted using something like make/rake/psake or some other task runner. Those scripts should be committed and versioned alongside the source code. XML build configurations are out, builds scripts are in.

3. Do you use Continuous Integration with automated builds?

Daily builds are a thing of the past. Automated CI builds are basically want you need to develop quality software.

And in the case of SaaS, I would take it a step further: do you automate deployment into your qa/staging/prod environments?

4. Do you have a bug database?

As most shops are practicing some form of “agile” development whereby requirements are represented as user stories or tasks in a backlog, bug tracking is usually integrated into that workflow. There are countless tools available: TFS, Jira, Pivotal Tracker, etc.

However I think it’s important that the development teams control these tools - there’s nothing worse than having a workflow dictated by management. Developers hate paperwork and documentation, and neither directly result in higher quality software.

5. Do you fix bugs before writing new code?

Nothing to do add here.

6. Do you have an up-to-date schedule?

What are you iterations like? Is it more SCRUM with strict time-boxed sprints, or is it a looser kanban approach? When and how are planning/retrospective meetings held? Standups?

How do you plan/schedule work on technical debt and other non-user-story tasks?

7. Do you have a spec?

How is the backlog managed? Who are the product owners and business analysts? Are user stories well written and include screenshots and acceptance criteria?

8. Do programmers have quiet working conditions?

Said differently, do you have an open office plan? This is probably quite subjective, but I personally hate open or shared offices.

How are meetings scheduled? As a developer, can I block off my morning and/or afternoon such that no meetings are allowed?

Can I work remotely? Are there communication and collaboration tools in place to facilitate remote workers?

9. Do you use the best tools money can buy?

Do I get to control what’s installed on my machine? Do you use a proxy server or otherwise limit my Internet access? In the “enterprise” world it’s very common to find restrictions which actively prevent a software developer from doing his or her job.

10. Do you have testers?

Do you do automated testing?

11. Do new candidates write code during their interview?

I’m not going to open this can of worms today…

12. Do you do hallway usability testing?

I think some take this rule too literally - especially regarding remote teams. Everybody can do screen sharing these days with ease.

It’s definitely important, but there’s a balance to find and it’s hard. You don’t want to just interrupt other people and drag them into your space to look at something, but you also don’t want to burden folks with meetings and scheduling.

It's All About the Pentiums

2016-04-12T00:00:00+00:00

A couple of months ago I wrote about searching for the ultimate dev laptop. Today there are some updates, but I’m going to focus on the processors. In this case, I’m looking at the top-of-the-line model each:

Model	Family	Number	Speed/Cores
Apple MacBook Pro 13" (early-2015)	"Broadwell-U" (5th gen)	5557U	2 x 3.1GHz
Apple MacBook Pro 15" (mid-2015)	"Haswell-H" (4th gen)	4980HQ	4 x 2.8GHz
Dell Latitude E5470	"Skylake-H" (6th gen)	6820HQ	4 x 2.7GHz
Dell XPS 13	"Skylake-U" (6th gen)	6560U	2 x 2.7GHz
Lenovo T460s	"Skylake-U" (6th gen)	6600U	2 x 2.6GHz

Some notes:

Apple has still not released or announced updated MacBook Pros. Hence, they are a generation behind.
Dell has updated their Latitudes to latest processors. But the 6820HQ is technically the older Skylake-H.
I’m now also comparing the Dell XPS 13. It’s even smaller than the Latitude. It has Thunderbolt instead of HDMI. It’s also touch, and perhaps a bit under-powered. But not severely. The biggest difference is that you can upgrade the HD and some other components.
Lenovo has also updated their T series. The Skylake-U is the “low power” processor.

So I’m still torn. I’m definitely going to wait and see what Apple does this summer.

Speaking at SQLSaturday Atlanta in May

2016-04-07T00:00:00+00:00

I’ll be speaking at SQLSaturday in Atlanta on May 21, 2016. This is really exciting since it’s my first professional public speaking engagement.

My session is called Data Access for Performance Junkies. Subtitled “or: How I Learned to Stop Worrying and Love the ORM.”

ORM frameworks get a bad rap for their performance characteristics and complexity. But like any tool, it’s all about learning to use correctly and intelligently. In this session we’ll look at how application developers often use (and misuse!) these tools, and identify and correct problematic usage patterns. We’ll also discover when it’s appropriate to use an ORM, when it’s not, and how incorporate “raw” SQL where it makes sense.

Introducing Horton

2016-04-03T00:00:00+00:00

What is Horton?

Horton is the simple database migration utility. It’s a little program that does one thing: enables versioning of database schema through SQL based migration scripts.

You’ll find a “getting started guide” in the project’s GitHub repository. Horton is also released through GitHub. You can find and download the latest release from the releases tab.

What does it do?

It may help to read this first: I recently wrote about database versioning with migrations.

Horton handles migrations for schema objects and desired state for “code” like objects (stored procedures, views, etc.). It was written to help apply database change scripts to production servers in a fully predictable and automated fashion (i.e. continuous deployment). It’s grown a number of features in recent years, but has never strayed from its very focused purpose.

Why am I “Introducing” it now at version 4.0?

Horton is actually the latest incarnation of a SQL migration tool I wrote almost seven years ago. The first commit was in September 2009! And even though it was published on GitHub, it was always just an internal tool and I never treated it as a real open source project. But I’m trying to change that.

Versions 1-3 did not strictly follow semantic versioning either. There were few, if any, breaking changes over its lifetime.

For version 4, I decided to rewrite it.

I rewrote the command line parser. Actually, it uses NDesk.Options which is the best .NET command line parser I know of. The command line arguments are similar, but different. Notably you can execute different COMMANDs from the same tool.
I redesigned the schema_info table. This is biggest breaking change as it renders the tool entirely incompatible with previous version. There’s an open GitHub issue suggesting how solve this if it becomes a problem (though highly unlikely because not many projects used the old tool!).
There’s cleaner separation between database specific code and the general horton program code. Data access is all based on ADO.NET, so in theory I should be able to support any provider.

I’d love for folks to give it a try and provide feedback. Enjoy!

The Database Versioning Showdown

2016-04-02T00:00:00+00:00

Versioning of your database schema is an important component of mature software development. Depending on your database system there are number of tools available to version your schema. While most are adequate for of version control, many are not necessarily well designed for the deployment of schema changes.

A lot of dev shops these days are practicing some form of continuous deployment (CD). This is particular useful to those who are building software-as-a-service (SaaS). The ability to go from source to a working application, including the database, is critical.

While focusing on relational databases, most tools fall into one of two camps: 1) model based or 2) migration based.

Model Based Version Control

What I’m calling model based version control refers to maintaining each object in the database (table, view, stored procedure, etc.) as a separate file in your version control system. Most often it takes the form of a SQL script containing the necessary CREATE commands for the corresponding database object.

Perhaps the two most popular tools, particularly in the .NET community are Redgate’s SQL Source Control and SQL Server Data Tools (SSDT) database projects.

It’s easy to see how this approach provides excellent visibility into changes that are made to the schema. E.g. you can easily see the history of an individual object and also see line-by-line annotations.

When it comes time to deploy, either upgrading an existing database instance or setting up a brand new one, the model represents a desired state. Generally some tool is responsible for comparing the model versus the existing state of a target database and generating an upgrade script. Some tools do a better job than others when it comes to determining the most efficient and correct script to get to the desired state. But regardless, you’re relying on some algorithm to figure it out.

The biggest danger of desired state upgrades is the potential for data loss. A schema comparison tool will do it’s best to determine how to alter the schema correctly without data loss, for example renaming a column rather than dropping/adding, but it’s bound to make a mistake.

Migration Based Version Control

Rather that representing the database as a model in your version control system, migrations are a set of sequentially executable scripts that iteratively alter a database’s schema. Often just SQL scripts, and using a naming convention that may include a numeric prefix for ordering, migration scripts are usually very simple to reason about.

They’re often written “by hand” to execute specific changes in a precise order. As a result there’s far less risk of data loss as it’s easier to predict the script’s affects.

Migrations are also efficient. Instead of comparing and generating a change script, we simply execute the migrations that haven’t been run yet in the correct order. Often a special database table is used to maintain information about the migration scripts that have been executed.

I think database migrations first became really popular as part of the ActiveRecord ORM in Rails (i.e. Ruby on Rails) as far back as 2005-2006. You wrote “up” and “down” migrations in Ruby, and a tool that shipped with Rails would upgrade/downgrade the database accordingly. When building later versions of Entity Framework, Microsoft adopted a similar pattern for database migrations as part of their “code-first” database strategy. In a future article I’ll discuss why you don’t need another domain specific language (DSL) on top of SQL, and why it might actually be harmful.

But migrations aren’t perfect. They require discipline. They usually require that all database changes execute as a migration so that the schema is never “out of sync”. But I argue that this discipline is a good thing as all changes are consistently version controlled and tracked. However, if it’s possible or it’s likely that the schema will be modified outside of migrations then the “desired state” model may in fact work better.

Why not both?

wanted: DB migrations for schema, DB projects for sprocs/functions
— Jimmy Bogard (@jbogard) March 16, 2016

Clearly migrations are better suited for schema changes in which data preservation is critical. They naturally map to the how a DBA or developer may make changes: as a set of iterative commands performed step-by-step.

However some database objects, such as stored procedures, views, and functions, can exist independently of the underlying schema. They’re also not really tied to data per se. Instead they’re more of an abstraction over the relational model. Stored procedures and functions, in particular, are almost “code” by their nature. So it does make sense to treat these like “classes” or code modules that are modified over time.

Ideally we would have a tool that mixes both migrations and model/desired state. Fortunately these tools exist! In fact I wrote such a tool almost seven years ago (!!) which I now call Horton (https://github.com/jdaigle/Horton). While researching for this article I stumbled across some other .NET based tools. One in particular that stoop out is RoundHousE (https://github.com/chucknorris/roundhouse). It’s not quite as old as Horton, but it’s been around long enough and seems has a well defined community, so it warrants attention.

I suggest checking these out and figuring out how they might work in your continuous deployment strategy.

Integrating ASP.NET MVC and WebAPI

2016-02-25T00:00:00+00:00

The current versions of ASP.NET MVC and WebAPI (5.x) exists as two separate frameworks. Each has it’s own routing conventions, model binding, action filters, dependency resolver, controller factory, configuration patterns, etc. And if you dig into the source code you’ll find A LOT over overlapping code; it’s practically copy/paste between the namespaces.

But this does not mean you cannot mix the two together.

Scott Allen has an excellent article in which he describes when to use each framework and when it might be appropriate to use both in the same project.

I’ve gotten more than a few questions over the last year on how to use the ASP.NET MVC framework and the Web API framework together. Do they work together? Should they work together? When should you use one or the other?

Here’s some general rules of thumb I use.

Because each framework has it’s own separate processing pipeline, you need to consider carefully whether you really need both frameworks in the same project. Especially if you do important things with routing and action filters, such as authentication or authorization - it’ll take extra work to share code and algorithms between the two.

Fortunately the future is bright. ASP.NET Core promises to finally unify the frameworks.

Aftermarket ASP.NET MVC - Part 5 - Feature Folders

2016-02-19T00:00:00+00:00

This is part 5 of a multi-part series on “fixing” some of the inherit design problems with ASP.NET MVC.

Part 1 - Introduction
Part 2 - The View Engine
Part 3 - View Model Conventions
Part 4 - Routing and URL Generation
Part 5 - Feature Folders
Part 6 - ActionResults and Content Negotiation

I’ve also setup a repository on GitHub that includes many of these experiments and implementations.

Expanding on the theme from last time:

Don’t let your routing solution dictate how to organize your code.

ASP.NET MVC establishes the convention of putting all of your controllers inside of a Controllers\ directory and all of your views instead of a Views\ directory. This is, no surprise, extremely similar to the convention established by Rails. Remember that Rails was the new-hotness when ASP.NET MVC was first developed around 2007-2008.

A prototypical ASP.NET MVC source code layout:

MyApp/
├── assets/
│   ├── site.css
├── Controllers/
│   ├── HomeController.cs
│   ├── ProductsController.cs
├── Views/
│   ├── Home/
│   │   ├── HomePage.cshtml
│   │   ├── AboutPage.cshtml
│   ├── Products/
│   │   ├── Index.cshtml
│   │   ├── Detail.cshtml
│   ├── Shared/
│   │   ├── _Layout.cshtml
│   ├── _ViewStart.cshtml
│   ├── Web.config

Yuck!

A lot of people have written about how this is just a terrible way to organize your code. At the top level, you don’t see your application. You see an MVC framework details leaking through.

I should be able to organization my code based on feature instead of what type of class it is.

MyApp/
├── assets/
│   ├── site.css
├── Home/
│   ├── HomeController.cs
│   ├── HomePage.cshtml
│   ├── HomePageViewModel.cs
│   ├── About.cshtml
│   ├── AboutViewModel.cs
├── Products/
│   ├── ProductsController.cs
│   ├── Index.cshtml
│   ├── IndexViewModel.cs
│   ├── Detail.cshtml
│   ├── DetailViewModel.cs
├── SharedViews/
│   ├── _Layout.cshtml
├── _ViewStart.cshtml
├── Web.config

Now you can look at the project structure and get a sense of what the app is! Additionally, all of the code related to a feature is in close proximity. In my opinion this makes development easier, possibly even faster.

A Quick Word About Areas

In theory, the Areas feature can get you pretty close to this model out of the box. However I’m not a fan of Areas. They were originally designed to work sort of like “sub-projects”. In theory they can physically live in their own assemblies, but this causes problems. But at the end of the day Areas still suffer, though internally, from the same organizational problems.

I prefer to avoid areas.

Making Feature Folders Work

Unfortunately Feature Folders won’t necessarily work out-of-the-box. There are a few things we need to tweak.

First, as a mentioned in a previous article, assigning routes to individual actions is important.

Second, the default view engine only knows to look for views inside of Views\. But we can subclass the default engine using the technique I wrote about here to make it work.

Inside the traditional Views\ directory lives a special Web.config. This does two things: 1) it makes the Razor tooling work and 2) it prevents access to the cshtml files in the directory tree as “web pages” (as the views are commonly deployed, and then compiled at runtime). We may move some of the config to root the Web.config of our project.

First, we need to register and include the RazorWebSectionGroup.

<configSections>
    <sectionGroup name="system.web.webPages.razor"
                  type="System.Web.WebPages.Razor.Configuration.RazorWebSectionGroup, 
                        System.Web.WebPages.Razor, Version=3.0.0.0,
                        Culture=neutral, PublicKeyToken=31BF3856AD364E35">
        <section name="host" 
                 type="System.Web.WebPages.Razor.Configuration.HostSection,
                       System.Web.WebPages.Razor, Version=3.0.0.0,
                       Culture=neutral, PublicKeyToken=31BF3856AD364E35"
                 requirePermission="false" />
        <section name="pages"
                 type="System.Web.WebPages.Razor.Configuration.RazorPagesSection,
                       System.Web.WebPages.Razor, Version=3.0.0.0,
                       Culture=neutral, PublicKeyToken=31BF3856AD364E35"
                 requirePermission="false" />
    </sectionGroup>
</configSections>

<system.web.webPages.razor>
    <host factoryType="System.Web.Mvc.MvcWebRazorHostFactory,
                       System.Web.Mvc, Version=5.2.3.0,
                       Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
    <pages pageBaseType="System.Web.Mvc.WebViewPage">
    <namespaces>
        <add namespace="System.Web.Mvc" />
        <add namespace="System.Web.Mvc.Ajax" />
        <add namespace="System.Web.Mvc.Html" />
        <add namespace="System.Web.Routing" />
    </namespaces>
    </pages>
</system.web.webPages.razor>

This essentially makes the Razor tooling and compilation work.

Next, under <handlers> in <system.webServer> we need to add:

<add name="BlockViewHandler" 
     path="*.cshtml" 
     verb="*" 
     preCondition="integratedMode" 
     type="System.Web.HttpNotFoundHandler" />

In the default Views\Web.config, this handler is registered with the path="*". But we don’t want to block access to all files, just our cshtml files. Finally we can add an appSetting to turn off “webpages”:

<add key="webpages:Enabled" value="false" />

Unless you want normal web pages to work… and why would you at this point? In which case you’ll need to leave them enabled and perhaps tweak the handler to not exclude the pages you want.

Alternatively you can copy the default Views\Web.config into each of your feature folders. But that’s ugly and possibly un-maintainable.

Another interesting problem man run into: if you have a folder named Foo\, and a URL such as ~\Foo\ that matches that directory name, by default System.Web.Routing won’t match that URL because it’s a physical folder on disk. In order to force System.Web.Routing to match that URL, we need to set RouteTable.Routes.RouteExistingFiles = true;. This setting changes the behavior so that it does not first check that a file exists before attempting to find a route.

I also encourage setting up routes that explicitly ignore your static assets (CSS/JS, images, HTML files, etc.) to help with performance.

Things Might Get Better with ASP.NET Core

Feature Folders almost just work. I think some of the decisions made in ASP.NET MVC are partially due to adopting the conventions of Rails-like frameworks, but also partly to workaround the inherit properties of ASP.NET.

In ASP.NET, traditionally, you deploy everything as a file. You’re static content and your dynamic content (ASPX pages) are mixed. There’s no routing, and URLs are reflective of the physical layout of files on disk.

But in almost any modern web application, except for static content, URLs are an abstraction over dynamic content.

Luckily ASP.NET Core throws away everything and starts from scratch. We may, for instance, have a project structure which looks like this:

MyApp/
├── wwwroot/
│   ├── assets/
│   │   ├── site.css
├── Home/
│   ├── HomeController.cs
│   ├── HomePage.cshtml
│   ├── HomePageViewModel.cs
│   ├── About.cshtml
│   ├── AboutViewModel.cs
├── Products/
│   ├── ProductsController.cs
│   ├── Index.cshtml
│   ├── IndexViewModel.cs
│   ├── Detail.cshtml
│   ├── DetailViewModel.cs
├── SharedViews/
│   ├── _Layout.cshtml
├── _ViewStart.cshtml

As you can see, they’ve introduced a wwwroot directory. This is a special folder which contains all of the static content that will be deployed too the root of the web application on the server. Everything else in the project is just code.

Fun fact: while you may compile this before deployment, it should be noted that ASP.NET Core has a new build model where it will compile your code at runtime and detect changes recompile and relaunch.

High Performance Message Broker Design

2016-02-01T00:00:00+00:00

In my previous post I talked of beginning research into designing and implementing an AMQP-based message broker.

Implementing the network layer and AMQP protocol is relatively easy. But the actual message broker implementation (queuing, dispatch, and persistence) is an interesting problem.

Here are some of the requirements:

I should be able to concurrently enqueue messages.
Enqueued messages may have a TTL.
Instead of de-queuing from the head of the queue, I need to be able to ACQUIRE a message.
An ACQUIRED message will be delivered. If accepted, the message will be ARCHIVED. If released the message becomes AVAILABLE for delivery again from the same position (i.e. the head).
I should be able to spontaneously release ACQUIRED messages after an expiry.
Consumers should be notified of new messages for delivery.
Consumers may filter messages.
Consumers should implement flow control to restrict the number of delivered messages.

I took a lot of inspiration from other open source projects such as Apache QPID, Apache ActiveMQ (specifically ActiveMQ Apollo), RabbitMQ, and Azure Service Bus. And the AMQP Spec itself lends some guidance from the protocol level of how a broker might behave.

Concurrent Linked List Queue

What I’ve discovered is that a traditional queue, such as .NET’s *ConcurrentQueue* is insufficient. Particularly due to acquiring and releasing messages. But I can use a concurrent linked list, with some modifications.

[HEAD] -> [ ] -> [ ] -> [ ] -> [ ] -> [ ] -> ... -> [ ] -> [TAIL] -> NULL

Using a linked list as a queue requires maintaining HEAD and TAIL pointers. Entries are enqueued by a) creating a new entry b) setting the next-> pointer of the current TAIL to the new node and c) and setting TAIL to point to the new node. To dequeue a) take the HEAD and if next-> is not NULL b) set HEAD to point to the next node.

To achieve concurrency without blocking or locking, we can use compare-and-swap instructions. Essentially we loop until the system can atomically swap pointers from a known state. Here’s the implementation for Enqueue():

We don’t actually want to “dequeue” in the sense of removing the node. Instead we want to acquire. We can accomplish this by maintaining some additional state in the queue entry itself. Consider a state field with three values: AVAILALBE, ACQUIRED, and ARCHIVED.

At any point we can get the next available node by starting at the HEAD and walking the list until we reach an AVAILABLE entry, or NULL. Then acquire the entry by changing the state. The entry can be released, causing re-delivery, or ARCHIVED. Again, we can use an algorithm that makes use of compare-and-swap to do this concurrently without blocking or locking.

When an entry is ARCHIVED, it’s not immediately removed the linked list. Instead we occasionally run a tweak-able scavenging algorithm. This algorithm walks the linked list and removes ARCHIVED entries.

Queue Consumers

In a message broker consumers of a queue usually come and go over. There may be multiple consumers. And we need a way to efficiently deliver messages to attached consumers.

In this case a consumer will ultimately be some remote link (that’s AMQP terminology) that wishes to receive messages from a queue. So when a remote client connects, we may create and associate consumer for a queue. The consumer is disposed with the connection/session/link (actually AMQP supports re-attaching broken links by exchanging unsettled state maps, but it’s not a requirement).

If I were building a simple blocking queue for a single consumer, I would utilize System.Threading.AutoResetEvent. The consumer has an event loop which attempts to dequeue the next message, and if unsuccessful calls WaitOne() on the WaitHandle. Each time a message is enqueued, I call Set() on the WaitHandle which signals the blocked thread to continue.

In theory I could create a blocking thread for each consumer. But there are several problems with this approach. 1) Each blocked thread consumes memory, and occupies a ThreadPool thread that could otherwise be doing work. 2) The WaitHandle will only a signal a single thread. So only a single consumer will be signaled, and only that consumer will attempt delivery.

I need to be able to signal all consumers. And do so fairly.

My research lead me to implement a single message delivery event loop thread. When signaled, it will loop over each consumer and attempt delivery of a single message. And continue until all messages are delivered.

While this does in fact work, there are still concerns. a) What if there are messages to deliver but no consumers or b) what if there are consumers but no messages to deliver. One could visualize a situation where the loop is running constantly but not actually doing anything. Thus starving the system of resources.

Instead, we can have the loop end at the point of either a) no subscribers or b) no messages. We need a way to restart the loop. That’s when I learned about ThreadPool.RegisterWaitForSingleObject. This allows us to register a WaitHandle which calls a callback delegate when the WaitHandle is set. Instead of having one of our own threads blocked, the OS efficiently handles it for us.

Each time Set() is called, it will call our callback delegate. I needed to implement a small flag to track whether the loop is already running to prevent it from running concurrently. We return from the callback when we’re out of work to do.

Here’s the implementation:

Flow Control

Another aspect of the message broker is the idea of flow control. Each consumer implements credit-based flow control. Or, in AMQP terms, a link-credit. The “credit” is decremented on each delivery until it reaches 0. At 0, no more messages may be delivered until the credit increases.

This ensures we never deliver more messages than the link can handle.

This is actually quite straightforward to implement in our broker. The consumer object maintains the credit, and is decremented on each delivery. Before each delivery, the credit is inspected to ensure we can deliver. If not, that consumer is skipped. The consumer’s credit can be reset after messages are processed, etc. In the case of AMQP, we’ll receive FLOW frames with updated credit.

However the above message event loop will need to be modified to account for credit. Specifically, when no consumers have available credit we can stop the event loop. Upon setting new available credit, we can again Set() the WaitHandle to restart the event loop.

Next time I’ll talk about making queues durable with append-only-file (AOF) style transaction logs, and idea I borrowed from Redis;.

Introducing LightRail

2016-01-25T00:00:00+00:00

LightRail is actually three different things:

An AMQP-based Message Broker for Windows (based on .NET)
An AMQP Client Library for .NET
A Service Bus Application Framework

The name “LightRail” is based on the transportation metaphor adopted by MassTransit, another open source Service Bus framework. I also wanted the name to imply that it’s a much smaller library and framework.

A Quick History

Many years ago while working at Clearwave we decided to use NServiceBus to build out our asynchronous processing components. NServiceBus is a great framework. At the time it really only targeted MSMQ, however we decided to build our service bus on top of SQL Server Service Broker to take advantage of the scaling and high availability that you get with SQL Server. It was relatively easy to implement ITransport back then, and the code is still kind of around.

Since then several things happened:

NServiceBus went commercial. While still open source, it’s not free to use in commercial applications. The 2.0 version remained open source with Apache V2.
We quickly realized that NServiceBus’ abstractions were too limiting for what we wanted to do with Service Broker.
The 2.0 branch had a few performance issues.

So we actually “rewrote” NServiceBus as our own CWService. It’s kind of the same architecture, but very tightly coupled to Service Broker instead of MSMQ. And we made it open source.

Since I’ve been a consultant with Intellinet have had the opportunity to work on a project that had a need for something extremely similar to what we built with CWServiceBus. So I decided to start a project to rebuild CWServiceBus as a proper open source service bus application framework.

But as I got started, I quickly realized that I was just rewriting NServiceBus… and I didn’t want to do that.

The AMQP Paradigm Shift

A few weeks ago I was listening to this podcast where they interviewed Clemens Vasters. They talked at length on his work with messaging standards and AMQP. Now I’ve known about AMQP from a very high level after having used RabbitMQ. But I didn’t really know AMQP until I decided to dig into the actual 1.0 specification. And I starting learning even more.

A few things struck me:

RabbitMQ doesn’t really support AMQP 1.0. It supports an older and incompatible version of the protocol.
Azure Service Bus (mostly) supports AMQP 1.0. The “cloud” product is awesome, but the “on premise” version is… bad.
There’s not really any AMQP 1.0 based messaging broker for Windows today.

I was also been reading about Disque from Salvatore Sanfilippo (aka Antirez). It’s a super-light-weight message broker based on his work with Redis. And then I got the bug. I decided that I wanted to build a message broker for Windows using .NET and AMQP 1.0. Crazy, I know.

As I mentioned at the top, LightRail is actually three different things:

An AMQP-based Message Broker for Windows (based on .NET)

I’m not doing a technical deep-dive with this article. But I’m planning to build something in .NET that runs on Windows. It’ll be fast. The persistence model is sort of based on what Antirez has done with Disque/Redis. But the networking protocol will be AMQP. There may be a secondary management channel over HTTP for monitoring and managing the service.

I don’t yet know what features it’ll support. As a message broker it’ll be similar to RabbitMQ, Azure Service Bus, and Disque. I.E. the client receives a message and must acknowledge after processing. There will be at-least-once semantics and possibly at-most-once. Topic/subscription queues. Dead-letter queues. TTL, retries, filtering, etc. AMQP supports transactions and so will LightRail.

The first version probably won’t be distributed. But I may experiment with different topologies such as a master/slave replication or even multi-master for some scenarios.

And maybe one day it’ll be cross-platform targeting .NET Core. I could take advantage of libuv, similar to how Microsoft is building Kestrel, their cross-platform HTTP server.

An AMQP Client Library for .NET

The Azure Service Bus library has an AMQP support built in. But it’s built for Azure Service Bus first-and-foremost. It’s also not open source.

Microsoft also maintains a AMQP 1.0 .NET Client Library (aka “amqpnetlite”). Comparing the two, it’s clear that they’re very similar. Interestingly this library supports building both a “client” (which actively connects to a remote host) and a “server” (which listens for connections) or both.

While it is a fairly complete library and seems to be well testing and relatively mature, I’m not a huge fan. The API is rough. The various components (network, protocol, encoding) are all very tightly coupled. And it’s not optimized for either client or server applications. And there may be performance related issues especially for server applications regarding allocations and value type boxing, among other concerns.

So I decided to build my own. I am borrowing quite a bit from amqpnetlite, especially around type encoding. But the “core” library is really just centered around encoding and framing. There will be separate client and server implementations.

A Service Bus Application Framework

An AMQP library is not enough for a Service Bus. For that you need to handle routing, message serialization, dispatch, topics/subscriptions, timeouts, etc. That’s where the application framework comes in.

Now, instead of rewriting NServiceBus, I decided to take a different approach. Rather than being a service bus framework for building applications, I’m building a framework for building a service bus.

But what’s the difference?

I believe that one of the drawbacks of NServiceBus (and MassTransit, et al.) is the idea of completely abstracting away the underlying message queue or broker. In the days of MSMQ, maybe that made more sense. MSMQ didn’t really do anything other than queue and transport messages - the framework had to do a lot.

But the result of all this abstraction is that some platform specific features may get lost, unless they decide to add an abstraction for that feature. I think this is same sort of problem we find today with massive ORM frameworks and libraries. Everything is a leaky abstraction. And this is the problem we ran into with SQL Server Service Broker.

So instead I want the framework to exist as a set of building blocks for building a service bus using some platform. Whether it’s Azure, RabbitMQ, MSMQ, Server Broker, LightRail, whatever. So when you use it in your application, you’re not using some generic service bus. You’re using an ‘X’ service bus including whatever platform specific features it has. Any abstractions will be based conceptually on AMQP. Given that AMQP is a capital “S” Standard, I think it’s a good move to try and view the world through that lens.

Going Forward

Right now the project is very pre-alpha (or RC1 if I were the ASP.NET team ;). It barely does anything I described above.

In fact right now it’s nothing more than a research project. Maybe one day it’ll be mature enough to use in a real project. And then become a real open source project. Until then I’m kind of playing the the role of an architecture astronaut. In the future I’ll likely publish articles centered around specific technical components of the project.

AMQP: A Developer's Digest

2016-01-21T00:00:00+00:00

What is AMQP?

An acronym for the “Advanced Message Queuing Protocol”. It is a Standard (with a capital “S”) application layer protocol defining a messaging protocol for message-oriented software. It’s a binary protocol wire-level protocol. It describes the format and structure of data sent across the network. It’s effectively peer-to-peer protocol, simply defining how two peers may exchange messages, and not does define what roles the peers may or may not have.

You can, for example, build a message broker. Or message queue. Or a gateway. Or an event bus. Or an “exchange”. You get the idea. It supports virtually any type of messaging topology.

This article aims to be update-able reference for those getting started with AMQP.

The Protocol

Here is the actual 1.0 specification. It over 100 pages long, but it’s a relatively quick read. The protocol specification itself is relatively simple and broken into several layers.

Type System

AMQP implements it’s own binary encoding for types. It define primitive types that map to many popular programming languages and platforms. This enables a high-degree of interoperability. It also supports composite types.

Transport Layer / Link Protocol

Data is always transmitted across the wire in a frame. AMQP defines frames for various types of control messages (opening connections, starting sessions, attaching links, transferring messages, etc.).

Connections are usually opened over a reliable protocol such as TCP.
A session is started on a connection. A single connection may multiplex several independent sessions.
On a session, one or more uni-directional links are attached. Either peer may attach a link for sending or receiving. The link is attached, logically, to some terminus such as a queue.
Messages are transferred over links.

Since a single connection can multiplex multiple sessions and/or links, this means that network traffic can be streamlined. You don’t need multiple connections.

Messaging

The messaging layer covers:

Message format. Including structure, properties, and headers.
Delivery state for messages traveling between nodes. Messages need to be settled which ensures the sender and receiver agree on the state of the transfer.
Sources and targets, and the states for messages stored at a distribution node. It also covers resumption of broken sessions and links.

Transactions

The protocol supports transactions across transfers and distinct links in either direction. A transactions is first declared which establishes a transaction-id. Subsequent work is includes that transaction-id. At the end, the transaction is discharged, which will either commit or rollback the transaction.

Security

You can encrypt the connection with TLS. Additionally, the protocol supports SASL to authenticate peers. Both protocols are be negotiated.

Who/what uses AMQP?

Microsoft’s Azure Service Bus supports AMQP. But not all Azure Service Bus features are supported over AMQP. The client library implements AMQP (it’s not currently open source).

Microsoft does have an open source AMQP client library: https://github.com/Azure/amqpnetlite. I believe that it’s very similar to the implementation in WindowsAzure.ServiceBus.

The Apache Foundation has a number of open source projects: Apache Qpid and Apache ActiveMq.

There are a number of other commercial and open source brokers, for example IBM MQ Light. But I don’t know a lot about them.

And then there’s RabbitMQ. RabbitMQ is interesting because it is actually built on AMQP 0.9.1. That version was a pre-release and is a very different and basically incompatible protocol. There is a plugin to support AMQP 1, but it’s marked as “experimental”!

Annotated Further Reading

AMQP Version 1.0. The actual specification.
Clarifying AMQP. Kelly explains the differences between AMQP 1.0 and the prelease 0.9 specification.
AMQP Isn’t So Scary… If You Know How To Start!. Another digest style article with links and references.

Aftermarket ASP.NET MVC - Part 4 - Routing and URL Generation

2016-01-13T00:00:00+00:00

This is part 4 of a multi-part series on “fixing” some of the inherit design problems with ASP.NET MVC.

Part 1 - Introduction
Part 2 - The View Engine
Part 3 - View Model Conventions
Part 4 - Routing and URL Generation
Part 5 - Feature Folders
Part 6 - ActionResults and Content Negotiation

Update 2016-02-18: I’ve also setup a repository on GitHub that includes many of these experiments and implementations.

A common theme of this series is derived from the following quote I once read (but can no longer attribute):

Don’t let your routing solution dictate how to organize your code.

In essence, don’t name your controllers and actions based on what your URLs to look like.

ASP.NET MVC is built upon System.Web.Routing, a powerful HttpModule and abstraction that maps a request (by matching URL, HTTP method, and other headers) to instances of an IRouteHandler. Effectively System.Web.Mvc is just that.

Among the many poor defaults and other conventions established in ASP.NET MVC is the “default route” code that you usually find in the baked-in project template that almost everyone uses:

routes.MapRoute(
    "Default",
    "{controller}/{action}/{id}",
    new { controller = "Home",
          action = "Index",
          id = "" }
);

As we know this establishes the convention of a controller and action name defining the mapped URL. And, like many of these conventions, it is directly “borrowed” from Rails. We even have the silly notion of a “Home” and “Index” which I sort of consider “legacy” HTTP terms.

In the early days of ASP.NET MVC you sort of had to follow this convention because it was a pain to otherwise map URLs. But eventually we saw several open source projects attempt to fix that. My favorite was AttributeRouting: you use simple attributes on your classes and methods to define the URL, or define your own conventions, and the framework wires it all up. You get a really good programming model and good decoupling of code and URL paths.

In fact this library was so good, and popular, that Microsoft effectively re-implemented parts of AttributeRouting; baking it right into the framework!

But there’s a sad post-script: they only implemented a fraction of AttributeRouting’s extensive list of features. And they did so in a way that you can’t really use AttributeRouting as-is in ASP.NET MVC 5. And the project is now pretty much abandoned.

Tip: Use Attribute-Based Routing, Exclusively.

While I definitely recommend using attribute-based routing exclusively, there are some things to watch out for especially as you start adding a lot of mapped routes. I recommend reading about some of the performance issues that can pop up. Here is a good in-depth article that talks about it: https://samsaffron.com/archive/2011/10/13/optimising-asp-net-mvc3-routing.

And don’t name your routes. Yes, it will greatly speed up URL generation. But unless you have a relatively simple application that doesn’t change frequently, you’re going to have to maintain both those names and their usage.

Tip: Avoid String Constants for URL Generation

I really dislike that most of the UrlHelper methods require strings for controller and action names. I encourage the use of Microsoft.Web.Mvc, which includes a special class called ExpressionHelper that can generate a URL string from a lambda expression referencing a controller class and action method. And so that I don’t need to pollute my project with namespace references, I created a small set of extension methods to abstract away that mess:

Code that needs to generate a URL is now quite clean. It’s easier to refactor and to find references via static analysis.

var link
    = this.GetURL<SomeController>(c => c.SomeAction(params...));

Be warned that you do pay a small performance penalty with ExpressionHelper. Your mileage may vary. So do your own testing and performance profiling!

You can also take a look at T4MVC. This takes a different approach that utilizes T4 text templates to generate constants and static helper methods based on your project and source code. I have used this extensively in the past which great success. It’s fast because you’re using constants. But it’s a bit intrusive since it requires changing your controller methods to be virtual and creates messy overloads (not to mention the whole code-gen part that some people dislike).

Experimental Idea Ahead

An approach that I’m beginning to experiment with falls somewhere in between full-on expressions and static helpers. This article on Delegate-based strongly-typed URL generation in ASP.NET MVC shows how we can use an action method delegate to generate route values (action name, parameters, etc.).

But since action methods are instance methods, this approach requires a controller instance. Which, as the article explains, can get tricky if we want an action delegate for a controller other than the one currently executing. It’s not too hard to get a “dummy”, or “uninitialized”, instance like this:

    public static T InstanceOf<T>() where T : ControllerBase
    {
        return System.Runtime.Serialization.FormatterServices.GetUninitializedObject(type);
    }

One could even borrow from the concepts of T4MVC and automatically generate a static helper class with these “uninitialized” controller instances cached for URL generation. But so far any API I’ve created feels too messy compared to a lambda expression or T4MVC’s overrides.

More experimentation is needed.

Tip: Avoid Areas

You can almost think of an area as a little MVC application embedded into another. It adds a new route value for matching named “area”. Often by convention the area name is simply another part of the URL path that precedes everything else. The built-in view engines and controller factory are also aware of areas, so you end up with self-contained “Controllers/” and “Views/” namespaces/folders under the area.

But it’s the completely wrong approach to modularity.

In a future article I plan on discussing some alternatives that actually make sense.

Aftermarket ASP.NET MVC - Part 3 - View Model Conventions

2016-01-07T00:00:00+00:00

This is part 3 of a multi-part series on “fixing” some of the inherit design problems with ASP.NET MVC.

Part 1 - Introduction
Part 2 - The View Engine
Part 3 - View Model Conventions
Part 4 - Routing and URL Generation
Part 5 - Feature Folders
Part 6 - ActionResults and Content Negotiation

Update 2016-02-18: I’ve also setup a repository on GitHub that includes many of these experiments and implementations.

1) Each View Should Have a Strongly Typed View Model

By default, views in ASP.NET MVC have a dynamic view model. The underlying object is simply a dynamic key/value bag. While this aids with “rapid development”, in my opinion it hurts long term maintenance. You lose static code analysis, compilation time error checking, refactoring support, and some debugging support.

Instead, I think it’s far better to always have a strongly typed view model for each and every view.

I also like the convention of putting view models alongside the view itself. By convention the class name is the name of the view followed by “ViewModel”. Thus your view model’s name follows a pattern of [Project].Views.[Feature].[ViewName]ViewModel.

2) A View Should Only Reference Properties and Methods From the View Model

View engines in ASP.NET MVC, such as Razor, typically compile to classes that inherit from System.Web.Mvc.WebViewPage. Unfortunately this includes a lot of cruft that carries over from classic ASP and ASP.NET web forms that inherit from System.Web.UI.Page. Essentially, the view has references to so much more that it probably needs to just simply render a template!

Accessing anything outside of the view model is an anti-pattern. Global or static state is also not a good idea. Remember that the only job of the view engine is to create a string from a template. It can be reduced to just a named function that accepts a model a returns a string.

Don’t access HttpContext/Request/Response.
Don’t access the referenced controller.
Don’t access route data.
Don’t query a database or call external services.

“But I need to know if a user is authenticated to show/hide things.” Wrong. The view is responsible for rendering content and controls. If content or a control should be conditionally rendered, then the view model should include state that explicitly declares that condition.

3) URLs for Links and Buttons Should Be Part of the View Model

ASP.NET MVC has the ability the generate URLs for routes based on a controller/action pair. This is great since since you don’t need to hardcode URLs all over the place (except you still deal with string constants). But you should not use HtmlHelper or UrlHelper to create a link/button or generate an action URL from within the view template. Instead, any generated URLs should be properties in the view model. This helps to further keep the view’s template clean and without references to constants, or worse, controllers and actions.

Additional Tip: Pass a Razor Template as a Parameter to a View Model Helper Function

In your view model you can create methods that accept Razor templates as Func<object, HelperResult>.

public MvcHtmlString RenderIfCanEdit(
    Func<object, HelperResult> template)
{
    if (this.CanEdit) {
        return MvcHtmlString
            .Create(template.Invoke(null)
            .ToHtmlString());
    } else {
        return MvcHtmlString.Empty;
    }
}

Now in the view, you can do awesomeness like this:

<div>
    @Model.RenderIfCanEdit(
    @<p>
        ...
    </p>
    )
</div>

Depending on the scenario this can be a lot cleaner than having nasty looking if-statements. Here’s an article by Phil Haack that explains it more detail.

Aftermarket ASP.NET MVC - Part 2 - The View Engine

2016-01-05T09:00:00+00:00

This is part 2 of a multi-part series on “fixing” some of the inherit design problems with ASP.NET MVC.

Part 1 - Introduction
Part 2 - The View Engine
Part 3 - View Model Conventions
Part 4 - Routing and URL Generation
Part 5 - Feature Folders
Part 6 - ActionResults and Content Negotiation

Update 2016-02-18: I’ve also setup a repository on GitHub that includes many of these experiments and implementations.

The problem with the default view engine in ASP.NET MVC stems not from how it renders a view, but how it selects and finds a view to render. If you dig through the source code you’ll discover that selecting and finding a view to render is tightly coupled with the actual view engine itself. That can be troublesome if we want to implement our own conventions.

Further, the built-in view engine uses a convention of selecting and finding a view based on the controller and action’s name. I consider this an anti-pattern. It couples a view to controllers and action methods by name. It is possible to override this behavior by constructing a ViewResult with the name or path of the specific view you want selected. But again we’re coupled, since the convention is to look in a directory based on the name of controller. We also have to use string constants throughout which means we lose the ability to refactor and perform static code analysis.

I think a better design is to select and find the view based on the type of view model returned. I’ll talk about this more in a future article, but a really good convention to adopt is having a strongly typed view model per view. I also recommend that the view model class’ namespace match the “path” of the view. In fact, just put it in the same directory as the view! A benefit of this convention is that we can select the view simply by inspecting the type of the view model.

Fortunately the code to implement this is just a single class:

Not only does it select the view based on the view model’s class and namespace, but it degrades gracefully in the event that a view model is not supplied, or if the view model does not match any view. It’ll fallback to the default logic.

And then on startup, insert the new view engine:

ViewEngines.Engines.Insert(0,
    new ViewModelSpecifiedViewEngine());

In your controller actions, simply construct the view model and return the view result.

public ActionResult ListSomeResource() {
     var viewModel
         = new SomeResourceListViewModel();
     ...
     return View(viewModel);
}

This works well for partial view results too. However there is a hiccup: the use of the Partial() HTML helper is sort of broken by since it does not have an override that only accepts a view model object. It requires a partial view name… how annoying. But easy enough to work around with this new view engine; the view name can be any string with length > 0. So I simply create an extension method to make it easier:

public static MvcHtmlString PartialView(
    this HtmlHelper htmlHelper,
    object model)
{
    // minor hack, since all the internals of
    // finding a partial view require a partialViewName
    return htmlHelper.Partial("null", model);
}

This is a great technique to adopt with composite or nested views. For example, I have a view model for a page (parent) that contains one or more partial (children) views. There is a list, and for each item I want to render a partial. So in the parent view model I simply have children view model objects. When passed to the PartialView() HTML helper, it automatically selects and renders the correct partial view.

The code also works with sub-classed view models. This enables the use of sub-classes to change the behavior of how a view is rendered without implementing complex switch logic either in the view model or the view itself. It’s probably rare that one would ever do this, but it’s a nice bonus.

Next time I’ll talk more about view model conventions and share some tricks that have made development and maintenance smoother and safer.

Aftermarket ASP.NET MVC - Part 1 - Introduction

2016-01-05T08:00:00+00:00

This is part 1 of a multi-part series on “fixing” some of the inherit design problems with ASP.NET MVC.

Part 1 - Introduction
Part 2 - The View Engine
Part 3 - View Model Conventions
Part 4 - Routing and URL Generation
Part 5 - Feature Folders
Part 6 - ActionResults and Content Negotiation

Update 2016-02-18: I’ve also setup a repository on GitHub that includes many of these experiments and implementations.

ASP.NET MVC has been the contemporary framework of choice for backend web development on the .NET stack almost since its first release around 2008-2009. And now here we are, seven years later, and by-and-large the framework’s programming patterns have remained relatively unchanged.

At the time of it’s initial development the hottest and trendiest web development stack was Ruby on Rails. So it’s no surprise that Microsoft borrowed a great many of Rails’ conventions when designing ASP.NET MVC. Whether good or bad.

Over the years people have discussed and debated the patterns (and anti-patterns?) and other short comings of the framework. In fact, “alternative” frameworks such as FubuMVC sprung up as a result. Unfortunately, virtually all of these “alternative” frameworks see little to no adoption.

Arguably the worst part of being a .NET developer is the dependence on and suffocation by “1st party” frameworks and tools. For many dev shops, “3rd party” and open source is never considered an option or is simply forbidden; it’s either built by Microsoft or it doesn’t exist! I think the .NET development world is, in a bad way, unique in this aspect.

Side note: more recently there’s been a rise in Sinatra style frameworks such as Nancy. I really like NancyFX. And I hope they stay strong.

So while I do have many of my own ideas for how I would refactor ASP.NET MVC into something better (in my opinion), I know it’ll see zero adoption. And it will also be met with heavy resistance for new projects I work on.

Instead, I’m going to show a few tricks I’ve learned over the past seven years that make ASP.NET MVC much smarter. Over the course of this series I’ll discuss an aspect of the framework I dislike, and show how to make it better. I may also add to the series as I learn new tricks, and as the framework itself evolves.

ASP.NET Identity - A Case Study in Monolithic Frameworks

2015-12-11T00:00:00+00:00

Abstract

ASP.NET Identity is a monolithic membership framework, especially when combined with Microsoft.OWIN.Security (which is almost a requirement to successfully using it!). As a monolithic framework, the developer eventually suffers when trying to plugin or extend the framework to do something that is no supported out of the box. Developers may also struggle simply using what’s provided in the box due to severe leaky abstractions.

Separation of Concerns

First I want to briefly cover the difference between authentication and “membership” in web application.

Authentication specifically refers the mechanism by which a user’s identity is determined from an HTTP request. These days it’s quite common to use an HTTP cookie that serializes some sort of identity; be it a session or user ID, and perhaps identity claims. Often the cookie is encrypted so that only the server application is able to read and/or modify.

Other authentication mechanisms include basic access authentication (the username/password are encoded into the HTTP header of every single request), or tokens which are often included in an HTTP header or query string parameter.

In the ASP.NET world the term membership refers to the mechanism by which users are managed. Be it a database or a directory. “Authentication” is sometimes overloaded to refer to the process of validating a token or set of credentials against the user’s membership. This usually occurs during login and frequently also occurs on each HTTP request to validate the authentication token.

To make things a more complicated, modern web applications often do not directly deal with user credentials. Instead relying on external authentication via OAuth or SAML (think Facebook, Google, Microsoft, ADFS, etc.). But for this article, they’re slightly out of scope.

Abridged History of Membership and Authentication in ASP.NET

Most of what’s existed since the first version of ASP.NET exists as part of System.Web.Security. That means it’s part of the BCL and shipped with every installation.

One quick note: some authentication such as Basic and Windows auth are actually implemented natively by IIS and aren’t part of the .NET framework.

Here’s a pretty good article that covers the old membership providers: http://brockallen.com/2012/09/02/think-twice-about-using-membershipprovider-and-simplemembership/. It also offers a critique which aligns with the goal of this article.

Enter ASP.NET Identity

Again, to piggy back on good writings, here is Brock Allen again: http://brockallen.com/2013/10/20/the-good-the-bad-and-the-ugly-of-asp-net-identity/.

Leaky Abstractions

Here’s one egregious example. The UserManager class exposes IQueryable<TUser> Users. In this example the developer is using Microsoft.AspNet.Identity.EntityFramework to provide a concrete implementation for TUser. However the developer must understand how EntityFramework works and how the entities are implemented in order to use IQueryable<TUser> correctly.

I should also point out that the implementation of IQueryable<TUser> Users on UserManager throws a NotSupportedException if the IUserStore<TUser> implementation does not implement the correct interface! This is another perfect example of a leaky abstraction where the developer must understand the implementation in order to use the library.

Missing Features and Abandonware

Brock Allen pretty much nails it again talking about so many security and identity related concerns that the framework simply does not handle.

I find this quote particularly interesting:

Of course, with this redesign, I think Microsoft is in a much better place to add these features in a future release. But it ain’t there now.

Allen’s article was written over two years ago and new features really have not been added!

Using these monolithic frameworks often results in the developer writing much more “glue” code to get things working that it would take to simply write a non-abstracted version of what they’re trying to accomplish. These abstractions allow you to do neat things for demos, such as quickly building a working web application from templates. But for real world production applications? It can get rough.

I should also point that the current version of the library is effectively abandonware. There are no new updates in the current repository. In fact the repository has moved to GitHub to be included in ASP.NET 5. But I would like to point out that this is effectively a “port” of current version (as is evident) since ASP.NET 5 is built on completely new tooling.

Who Uses This Stuff?

I honestly wonder. Outside of small proof-of-concept or very small line-of-business applications, does anyone use these frameworks?

My guess is no. Because at the end of the day it’s almost easier to roll your own. You’ll get better performance and you can make it easier for junior developers to consume your API when building new application features. But this comes with an important caveat: security is hard. And you should not be building your own authentication and security libraries unless you actually know what you’re doing. So it’s a tough choice for a small shop that doesn’t have the expertise.

In a future post I will talk about the OWIN security components. These deal with cookie authentication, and provide implementations for external login providers (i.e. OAuth). But as a preview that library shares most of the same criticisms about monolithic frameworks, leaky abstractions, and more.

Searching for the Ultimate Development Laptop

2015-12-03T00:00:00+00:00

Update #1: I’ve written another short article (dated 2016-04-12) comparing the latest models, focusing on the processor specs. But I’m still waiting for the MacBook Pro refresh.

Some say it’s a futile quest. But I’m going to try anyway. I’m going to find (or at least pick) the perfect laptop for software development.

I’m currently writing this article from a 2013-era Lenovo ThinkPad W530. And it is a beast. It’s actually quite a performant laptop with a decent keyboard and screen. But it’s big and it’s heavy. Even the latest W-series models are just too big. I put this laptop in the class of “desktop-replacement” instead of “portable” - it’s not one I would choose for myself or, frankly, recommend.

I’m looking for a key balance of power and portability.

Dell Latitude E5450

The Dell Latitude 5000 Series has been a laptop I’ve thoroughly enjoyed in the past. Particularly the 14 inch model, which is currently the E5450. The 15 inch model is too large for my preferences (and why does a laptop need a number pad anyway!?). Some people may enjoy the extra portability of a 12 inch display, but for me 13 or 14 inches is the sweet spot.

Of course, Dell’s website is absolutely terrible in terms of actually trying to customize the configuration I want.

But let’s look at the specs. I generally max everything out.

5th Gen Dual Core i7-5600U 2.6ghz
14.0” Display FHD (1920x1080)
DDR3L 1600MHz, 8GB RAM (but upgradeable to 16GB if you can figure out how)
256 GB SSD (should be upgradeable)
Nvidia GeForce 840M
13.2 x 9.1 x .90 inches
4 lbs
Around $2,000 depending on configuration and whatever current deals are available.
Preloaded with Dell management crapware.

Here is the official spec sheet (hopefully the link doesn’t expire too soon).

It’s worth comparing the 14” model to the 15” model. For reference here are the specs to compare. If you notice, it has the exact same internal specs! You just get a bigger screen, but at the same resolution!

Note that the Latitudes now have HDMI output instead of DisplayPort. Personally I find this a drawback as HDMI is really intended for consumer electronics while DisplayPort has many other capabilities (display chaining, etc.).

Dell also makes a 7000 series, considered an “ultrabook”. It appears that you can get similar specs but without a dedicated graphics card. And it’s only marginally thinner and lighter (13.3 x 9.12 x .80 inches and 3.5 lbs). Ultimately it does come with an added cost.

Lenovo ThinkPad T450s

Apples-to-apples comparison suggests that I should look at the ThinkPad T series. Specifically the T450. Note that there is a T450s, but it has only 1 DIMM and thus a smaller max-memory. It’s kind of a weird laptop.

Here is latest Product Specification Reference which lists everything about every ThinkPad. Which is great for comparing.

Specs:

5th Gen Dual Core i7-5600U “up to 3.2gz”
14.0” Display FHD (1920x1080) - ISP display
16GB PC3-12800 1600MHz (2 DIMMs)
256 GB SSD (should be upgradeable)
Integrated Graphics only.
13.35 x 9.15 x .83 inches
“less than 4lbs”
Around $2,000 depending on configuration and whatever current deals are available.
Preloaded with Lenovo management crapware.

So it’s basically the same as the Dell Latitude 5000 series. Except with DisplayPort instead of HDMI. And there is no option for dedicated graphics card.

Lenovo also has a 15” T550, which again like Dell, is the same laptop but with a bigger screen.

Apple MacBook Pro 13” Retina

Apple is interesting when compared to the traditional laptop vendors. They generally don’t allow for customization. Instead, you can “upgrade” the CPU/RAM/disk at the time of purchase.

They make a 13” and 15” display model. But let’s look at the 13” specs:

Dual Core i7 3.1ghz
13” Retina Display, 2560x1600 native
max 16 GB 1866MHz (not upgradeable after purchase!)
512 GB flash (could upgrade to 1GB for an additional $500 at time of purchase)
Intel Iris Graphics 6100
12.35 x 8.62 x .71 inches
3.5 lbs
Pricing is around $2,200.

It’s small, light, sleek. That comes at the cost of not being able to upgrade the memory or hard drive after purchasing. There is a lot less configurability. But the specs are in-line with what you would want.

The 15” model has similar specs, but it’s quad-core and has the addition of a dedicated graphics chipset. It’s priced about $500 more. It is really attractive looking. It’s obviously bigger and heavier. But the dedicated AMD Radeon R9 M370X graphics and quad-core graphics is appealing. Compared to Dell and Lenovo, the 15” model actually offers a compelling reason to get the bigger model.

Unfortunately consulting the MacRumors Buyer’s Guide suggests waiting a few months for the new model.

Windows and .NET Development on a Mac?

I proposed this question on reddit here and here. And I got a lot of good feedback.

This is definitely a gray area and I’m still unsure. Fortunately, new MacBook Pro models aren’t expected for another few months so I have time to do further research before I decide to purchase.

Other Choices?

I know I only covered three vendors, and there are plenty of others. But what other vendor out there makes Laptops suitable for business and software development?

I do plan on updating this article as I receive new information. And I may publish a followup review after I decide and purchase.

If you have opinions or comments, please leave them below!

Cloning Disqus - Introducing CommentR

2015-11-11T00:00:00+00:00

This blog is open source and is built using Jekyll, a static website generator. One of the interesting side effects of hosting a completely static website is that I don’t have a dynamic backend that I can use for comments. Comments, at a minimum, require a) a database b) a form to post comments and c) a dynamically generated view.

In the world of static websites, a new category of SaaS as sprung up around “blog hosting services”. By far the most popular service is Disqus, but there are others such as Discourse.

Rather than simply pull in one of these services, I decided to try building my own! I decided to base my design on Disqus. Except my service won’t serve ads, does not require creating an account, and won’t track you.

The bare minimum requirements:

It must be easy to add to a page: i.e., include a single JavaScript reference and a placeholder element.
It must allow posting of comments without creating an account: just an author and body input.

Eventually I might add these features:

Moderation: I should be able to delete comments, or approve comments before they’re visible.
Moderator comments: My comments should be highlighted in some way.
Self-Service: Someone who has just posted a comment should be able to edit or delete that comment.
SPAM Detection: I don’t want a CAPTCHA, so I might build some trivial SPAM detector.

I decided to name my project CommentR. It’s open source and available on GitHub.

Building CommentR

Architecturally I knew that CommentR would be a simple API service. So I started with two simple API calls:

curl -X GET /Comments?permalink=XXX

curl -X POST --data "permalink=XXX&author=YYY&body=ZZZ" /Comment

The GET call returns the visible comments for a page (keyed using a permalink URL) either as JSON or an HTML fragment depending the HTTP Accept header. The POST call submits a new comment for a page (again keyed using a permalink URL) and returns the same result as the GET call.

And for the simplest level of “security” the API checks the HTTP Refer header from a known whitelist and rejects HTTP requests that are invalid.

To integrate this into a web page I built a really simple JavaScript file that’s hosted alongside the API which simply loads the comments and appends them to the page along with a form. The form, when submitted, submits a new comment and reloads the comments.

This approached worked fine on my computer, but I soon as I deployed it to an Azure site and tried it I ran into the dreaded CORS issue. Rather than open up that can of worms, I decided to change gears and implement CommentR using an IFrame.

The Final Design: IFrames

As it turns out, this is the approach that Disqus uses! It’s actually very simple:

Include a reference to some JavaScript resource. This code is responsible for setting up the IFrame and setting the correct src.
The IFrame src is a page that displays the comments specifically for the parent page (keyed using the same permalink URL as before).
All AJAX calls occur from within the IFrame, so the same-origin policy applies.

One clever trick I implemented was to have the embedded page “publish” a message whenever the content is loaded with the total scrollHeight of the content. The original JavaScript resource in the parent window responds by dynamically re-sizing the IFrame. So for the user, the comments window is completely seamless.

You should be able to see the final working version below. Leave a comment!

Hello World

2015-09-16T00:00:00+00:00

Hello World!

Code Outside of the Box

Hydrogen Extensions for ASP.NET MVC 5 Alpha Release

Getting started

Creating an async filter

This is an alpha release

Improving Async Support in ASP.NET MVC - Part 3 - Async Filter API

Improving Async Support in ASP.NET MVC - Part 2 - Refactoring AsyncControllerActionInvoker

Refactoring APM to TAP

Performance

Using AsyncControllerActionInvokerEx at Runtime

Next Time

Improving Async Support in ASP.NET MVC - Part 1

The Current Landscape

The ASP.NET MVC Async Pipeline

Getting Started

Update: 2017-03-13

Update 2017-05-07

"Code-First" Migrations with Horton

Scaffolding Migrations with Horton

Code Organization, CQRS Style

Command Pattern Therapy

Evolving Command Handler Design

In the beginning

Enter functions

The next generation

Wrapping up

The Modern Joel Test

It's All About the Pentiums

Speaking at SQLSaturday Atlanta in May

Introducing Horton

What is Horton?

What does it do?

Why am I “Introducing” it now at version 4.0?

The Database Versioning Showdown

Model Based Version Control

Migration Based Version Control

Why not both?

Further Reading

Integrating ASP.NET MVC and WebAPI

Aftermarket ASP.NET MVC - Part 5 - Feature Folders

A Quick Word About Areas

Making Feature Folders Work

Things Might Get Better with ASP.NET Core

High Performance Message Broker Design

Concurrent Linked List Queue

Queue Consumers

Flow Control

Introducing LightRail

A Quick History

The AMQP Paradigm Shift

Going Forward

AMQP: A Developer's Digest

What is AMQP?

The Protocol

Type System

Transport Layer / Link Protocol

Messaging

Transactions

Security

Who/what uses AMQP?

Annotated Further Reading

Aftermarket ASP.NET MVC - Part 4 - Routing and URL Generation

Tip: Use Attribute-Based Routing, Exclusively.

Tip: Avoid String Constants for URL Generation

Experimental Idea Ahead

Tip: Avoid Areas

Further Reading and References

Aftermarket ASP.NET MVC - Part 3 - View Model Conventions

1) Each View Should Have a Strongly Typed View Model

2) A View Should Only Reference Properties and Methods From the View Model

3) URLs for Links and Buttons Should Be Part of the View Model

Additional Tip: Pass a Razor Template as a Parameter to a View Model Helper Function

Further Reading

Aftermarket ASP.NET MVC - Part 2 - The View Engine

Aftermarket ASP.NET MVC - Part 1 - Introduction

ASP.NET Identity - A Case Study in Monolithic Frameworks

Abstract

Separation of Concerns

Abridged History of Membership and Authentication in ASP.NET

Enter ASP.NET Identity

Using `AsyncControllerActionInvokerEx` at Runtime