Techie.notepad

Orleans: The new Cloud Development Platform for Windows Azure?

Magnus Mårtensson — Mon, 06 Sep 2010 11:00:00 GMT

There is some rumors that the Orleans software platform will actually be a product coming out from Microsoft that will empower the Windows Azure platform and make it more easy to architecture and manage. This is just a research project but I for one am liking the hopeful outlook on life. ;~)

Somewhere among the few references out there on the web I am grasping for an explanation on what this will be. How about this:

Orleans is to Azure as the CLR is to Windows

It appears Orleans is designed to create a unified programming model for the client and the server. It offers greater abstraction so that applications can be easily ported across different platforms.

In Rob Knies article “Peering into Future of Cloud Computing” from way back when on February 24, 2009 we can read the following:

Orleans software platform: The software that runs in the data center is a complicated, distributed system. It must handle a vast number of requests from across the globe, and the computers on which the software runs fail regularly—but the service itself should not fail, even though the software is continually changing as the service evolves and new features are added. Orleans is a new software platform that runs on Microsoft’s Windows® Azure™ system and provides the abstractions, programming languages, and tools that make it easier to build cloud services.

James Larus has posted some Power Point slides from the Mysore-Park Workshop on Cloud Computing, Mysore India on January 13, 2010. There we can see some nice slides about this concept:

I must say this looks really interesting! “Grains” of apps that can migrate between data centers. This is very nice indeed for software architects. Me likey a LOT!

In an interview with eWEEK at Microsoft’s Worldwide Partner Conference in July, Amitabh Srivastava, senior vice president of Microsoft’s Server and Cloud Division, said Microsoft built its Windows Azure cloud platform with developers in mind from the start.

When we were developing Azure from day one it was done for developers,” Srivastava said. “You have to allow developers to bring their skills, their current set of skills, to the cloud. So we said developers should get to choose the language they want to use. You can use any environment you want. You can use Visual Studio or you can do the entire development in Eclipse. You can’t pigeonhole developers into one or two languages or one or two frameworks. Just because our lineage is Windows Server doesn’t mean we will restrict you to using C# or a Microsoft language.

Unfortunately not all good ideas are more than vaporware but I am very curious to see about this project. Let’s keep our fingers and toes crossed for Orleans and stay tuned for more on this as it comes out because I certainly intend to dip my fingers in that honey jar once it is available.

Cheers,

Technorati Tags: Windows Azure,Orleans

It's time for IoC Container Configuration Detente

Magnus Mårtensson — Sun, 05 Sep 2010 20:30:07 GMT

Want an easy way to configure your Inversion of Control (IoC) container using an API? Don’t care one iota about which specific container you actually are using you just want to get the work done? Want to configure your IoC in a type safe manner? Read on and find out how!

Scroll passed the background and rationale musings for the good stuff. ;~)

Background and rationale musings

Looking at a great initiative; the Common Service Locator and using it because it’s a good design practice to use abstractions one can’t help but to hunger for more! There was one more piece I felt myself wanting to sink my teeth into. After some testing, re-working and re-factoring and re-re-factoring ;~) an embryo for a common simple API begun to emerge. We have found in our team that there is a smallish set of classic configuration tasks involved in setting up your IoC:

Register Instance (with or without a key)
Register Type (with or without a key)
Marking a registration a Singleton
Selecting a constructor for injection
Selecting properties for injection

This is it really!

Really? Yes pretty much. This is likely to fill about 95% of the needs for configuring an IoC.

We’ve tested it in our project and it works really well. We started out using the API of the IoC we chose but felt it imposed a lot of presence in our code making our coders have to know and care a lot about how and what makes the underlying container tick. All they wanted to do was configure how their code would fit together.

That seems quite easy to create and API for. Indeed. ;~)

But wait; one single API can’t possibly fulfill all of the IoC configuration needs out there for ever! True that - it can’t pretend to be a silver bullet. Well let’s make it easy to extend then.

Hmmm... If not all IoC can implement this API; wouldn’t that make the API a little pointless? Well, not really. The API would not be intended to allow you to replace one container with another without changing or even recompiling the code. Though it could be used that way. Just because you can replace one IoC with another doesn’t mean you will! Having a fitting simplification for an API – and adapter if you will – still makes a lot of sense to your code maintainability!

What if all of the existing IoC creators out there couldn’t care less about this little pesky API? Well I sure can’t blame them who has time to read each and every blog post. Fortunately it doesn’t matter. Let’s make it so that anyone who implements the API for their favorite IoC container and contribute.

That sound’s fair! Not all IoC’s might even have a configuration API for that matter and the ones that do certainly do not have to have one that is suitable for such an API we’re talking of. Besides those that like code abstractions over hard references would use the API because it makes good design sense. Those who don’t care about such things would not use the API.

OK – I’m still interested; let’s have a closer look.

Features we like

What kind of features would we like to have in our API?

It is annoying when configuration code breaks when you refactor. Type safety?
It would be nice if it was easy to use and that means easy to read. A fluent interface?
If I need to add my own favorite feature it should be easy to do so. Open source with an easily extensible API?
It should be easy to implement for your own IoC. Good use of a base class implementation?

Sample code

It’s time to look at some sample code for this thing! There is more sample code on the project site and also a listing of the API in full.

This is what the code would look like when you use the API to configure your code; easy to understand, fluent, type safe!

ServiceConfigurator.RegisterType<IFoo, Foo>()
                   .InjectionConstructor(() => new Foo())
                   .InjectionProperties(f => f.Bar, f => f.Baz)
                   .AsSingleton();

That looks just awesome! Yes indeed! Awesome and quite useful. It’s easy to see how this may be extended for specific purposes. (Doh – sorry I’ll try not to sound like a TV chef or a TV advertisement any more – sorry.)

The code above tells your IoC container that you want to register the implementation Foo as the target for requests for instances of the contract IFoo. It will tell the IoC container to use the parameterless constructor (it can use any constructor). It will also tell the IoC container to use the properties Bar and Baz for Dependency Injection. Finally it will tell the IoC container to treat the instance of Foo as a singleton instance once it is created.

The lambdas look messy! Do they? We don’t even notice any more in our project. They are there to ensure type safety and if you implement this API using the base class the expressions that make up this “mess” are parsed once and for all and converted into more easily handled parameters such as types and strings. If you change any of your configured classes and try to compile the configuration code it will not even compile.

The what and the where

If you feel this API would be pretty darn useful and nice to get your hands on you’re in luck! It’s already created and published under open source license on CodePlex.

So without further ado… Introducing

The Common Service Manager

All the information you need about this project is up there on the site so this post does not need to go any further. If you hurry YOU can contribute with the implementation for your favorite IoC and claim the fame!

If you feel this project is missing one key feature please don’t hesitate to provide feedback!

Final words

Thanks to the great guys of coding for the inspirational Common Service Locator project that set our team off in the direction to create a piece of code that extends your work. Together these two pieces of API make our every day much more easy and that is what code by developers and for developers is all about! (The title of this post is a paraphrase from Jeremy Millers classic post.)

Cheers,

Technorati Tags: Arcitecture,API,C#,IoC

Entity Framework and Show & Tell .NET community event in Swenug Skåne in Malmo, Sweden

Magnus Mårtensson — Fri, 20 Aug 2010 05:48:14 GMT

If you are nearby or live here don’t be square! First NUG event in Skåne, Sweden for the fall is one with two agenda entries; Entity Framework and Show & Tell raffle.

Dag König from Microsoft DPE Sweden will join us to give the latest info Entity Framework.

We will have a raffle on who will get to present their latest cool/interesting/noteworthy thing from .NET development with a prize raffled among the participants.

Join Swenug and sigh up for the event here: http://www.swenug.se/events/skaane-entity-framework-show

Sponsor is Jayway.

Cheers,

Technorati Tags: Swenug,NUG,Entity Framework

Lazy<T> with implicit cast to T can give you some pretty code

Magnus Mårtensson — Thu, 05 Aug 2010 19:57:34 GMT

Adding an implicit cast functionality to Lazy<T> can give you an option to writing null checkers in your delayed instantiation properties.

The problem

Today in our project a colleague was cussing over the fact that he kept writing the same block of code over and over again when null checking a property with a backing field.

Given the class Foo:

private Foo oldFooProp;
public Foo OldFooProp
{
    get 
    {
        if (oldFooProp == null)
        {
            oldFooProp = new Foo();
        }
        return oldFooProp;
    }
}

Have any of you written something like this before? It’s a bunch of lines of code that you tend to write (in small variations) over and over again. Sure you can compact it a little but it in the order of 5 – 10 lines of annoying repetitive code.

Some use the null coalescing operator (??) version that looks like this:

public Foo OldAlternativeFooProp
{
    get
    {
        return oldFooProp ?? (oldFooProp = new Foo());
    }
}

This version is a little easier to compact down to even a single line.

But we (my colleague and I) can do you one better:

Here is an optional approach

private Lazy<Foo> foo = new Lazy<Foo>();

public Foo Foo
{
    get
    {
        return foo;
    }
}

Actually I’d prefer to do that like this but then you have to know the solution below and be comfortable with it:

public readonly Lazy<Foo> Foo = new Lazy<Foo>();

But wait a minute: That doesn’t work you say, right? You can’t return a Lazy<Foo> as a Foo you have to write theFoo.Value.

The Trick

The secret trick is to inherit Lazy<T> and implement an implicit operator for T on it!

namespace Jayway.com.Sys
{
    /// <summary>
    /// A version of <see cref="Sys.Lazy{T}"/> with the twist of being implicitly castable to <typeparamref name="T"/>.
    /// </summary>
    /// <typeparam name="T">The type of lazy instance we are creating.</typeparam>
    public class Lazy<T> : System.Lazy<T>
    {
        // Ctors removed for brevity

        /// <summary>
        /// Cast the <see cref="Lazy{T}"/> to it's contained <typeparamref name="T"/>
        /// </summary>
        /// <param name="lazyT">The lazy to cast.</param>
        /// <returns>The instance of <see cref="Lazy{T}.Value"/></returns>
        public static implicit operator T(Lazy<T> lazyT)
        {
            return lazyT.Value;
        }
    }
}

Now you can implicitly cast a Lazy<T> to it’s contained type T.

A few alternative constructions for it might look like this:

public LazyTests()
{
    // With parameter
    theFoo = new Lazy<Foo>(() => new Foo("bar"));

    // With Inversion Of Control
    IServiceLocator serviceLocator = GetServiceLocator();
    theFoo = new Lazy<Foo>(() => serviceLocator.GetInstance());
}

I kind of like this code. It’s compact. It’s easy to read. It’s pretty!

Hope you like it too – HTH!

Cheers,

Technorati Tags: C#,.NET

Handling Lazy + Metadata instances in Managed Extensibility Framework (MEF)

Magnus Mårtensson — Wed, 04 Aug 2010 19:21:32 GMT

Have you ever wanted to dynamically choose from many potential implementations of some code? Would you like to have the power of metadata filtering and lazy instantiation of the selected implementation? Here is how to do just that using Managed Extensibility Framework (MEF) and also a neat trick to handle your implementation/metadata pairs that I recently put to good use. Code samples and optimizations follow so don’t worry; I won’t remain all theoretical in this post. (MEF is part of .NET Framework 4.0 but is available also on CodePlex for other releases – MEF Community Site.)

Up front I want to show what I am aiming for so you can evaluate if this is a worth while read: Using the more advanced features of MEF suddenly I find I use these pretty gruesome variable and parameter declarations: Lazy<IWorkItem, IWorkItemView>. After applying the fix in this post I am left with the exact same behavior and this much improved declaration: IEnumerable<IWork>.

Background on MEF

Using MEF you can easily juggle the concept of lazily (using System.Lazy<T>) importing an open ended set of instances of a given contract. This is built into the library for you. It takes some getting used to the concept but it is quite powerful. As if this wasn’t enough you can also add metadata to the contracts (with the MEF version of Lazy<T> – System.Lazy<T, TMetadata>) to enable you to select which of the - potentially many - implementations of the contract that you want to lazily load into your app domain and instantiate. Does this all sound like geek speech? Well it sure is but as I said this is a very powerful concept. Given that you have a good requirement for this scenario you never have to code extensibility code yourself ever again. I would go so far as to say that MEF makes the extensibility scenario a solved problem!

My Scenario

Working with improving the Worker role concept in Windows AzureI I recently had a use for an open ended set of implementations of what I called “work methods”. I wound up making my work methods implementations of IWorkItem. The metadata I added to my work method classes I decorated with IWorkItemView metadata contract. This is what the two contracts look like:

public interface IWorkItem
{
    bool HasWork { get; }
    void DoWork();
}

public interface IWorkItemView
{
    string Name { get; }
    int Order { get; }
}

When using these contracts lazily in MEF I get the ugly instances of Lazy<IWorkItem, IWorkItemView>. There is some really cool functionality here for sure but I just can’t bare the ugliness of the statement as I begin to pass these instances around in method calls etc. The cool thing with MEF is that I don’t know beforehand how many implementations I have. All I need to know is that I can get them all:

[ImportMany]
IEnumerable<Lazy<IWorkItem, IWorkItemView>> Work { get; set; }

This gives me a whole enumeration of lazy instances that are combined with the metadata of the metadata contract. On each instance of the lazy works you call .Metadata to access the IWorkItemView contract and .Value to instantiate and return the instance of the IWorkItem. Now I don’t know about you but this is beginning to look kind of hairy to me. After all in my code I am not really concerned with any laziness or metadata contracts. Not only that – provided you are a coder that believes in separation of concerns there is one more issue. To explain the last statement let’s go back to my Windows Azure scenario:

I import my many lazy instances with metadata into my code. Then I’d like to begin to use those instances. I wrote some code to filter out which instance I wanted to use. I wrote code to execute the selected instance. I wanted to separate these two code blocks from each other and I also wanted to be able to exchange filtering methods on the fly. I could continue but some of you have certainly begun to see the problem here. I was now passing around parameters of type Lazy<IWorkItem, IWorkItemView>. That’s not exactly pretty! What I found that I was conceptually doing was rather using an instance of something that had the combined contract of my two work-method, and work-metadata contracts:

public interface IWork : IWorkItem, IWorkItemView { }

Passing around parameters of type IWork is a lot better than Lazy<IWorkItem, IWorkItemView>. You have to all agree to that, right?

So now – finally – the issue becomes that of transforming one instance into the other. I also want to keep the Lazy capability. The transformation technique is mostly built into an abstract base class I’ve written which you can download here:

MEFContrib.LazyDomainCompactor.zip

I have also put the code in the MEF Contrib project at Github.

Using the base class comes with a little bit of custom implementation per conversion you want to do from from “lazy with metadata” to “other domain object”. The alternative would have been for me to use a dynamic code generation approach for the implementation but I felt that dynamic code in this instance is a serious overkill. Besides we also need to make sure MEF picks up our implementation which in this case means we have to apply an attribute to our implementation.

The solution

To re-iterate: We had the ImportMany statement above which is a bit cumbersome. Instead we are aiming for this:

[Import]
IEnumerable<IWork> Work { get; set; }

Please notice the two different attributes ImportMany and Import. There is an important difference. The first one will find a lot of implementations of the contract (zero to many) while the latter will find only one exact match to the contract. In the first case we want MEF to find all IWorkItem implementations that are decorated with the IWorkItemView metadata. In the second case we want to find one single collection of IWork.

The trick is to collect all the instances (ImportManyAttribute) and hand them back out (ExportAttribute) as a single collection. What we do in the base class is import the many lazy with metadata instances and then immediately export a whole collection of the new contract to be picked up somewhere in the code as needed.

Details

public abstract class ValueMetadataCompactorBase<TValue, TMetadata, TValueMetadata> :
    IEnumerable<TValueMetadata>
    where TValueMetadata : TValue, TMetadata
{
    protected abstract TValueMetadata Compact(Lazy<TValue, TMetadata> lazyValueMetadata);

    [ImportMany(AllowRecomposition = true)]
    internal IEnumerable<Lazy<TValueMetadata>> ValueMetadatas { get; set; }

    [ImportMany(AllowRecomposition = true)]
    internal IEnumerable<Lazy<TValue, TMetadata>> ValuesWithMetadata { get; set; }

     // Cliped IEnumerable implementation for brevity [...]
}

There are a few things to notice here:

We ImportMany instances of the Lazy<TValue, TMetadata> and also for good measure we ImportMany istances of Lazy<TValueMetadata>. The reason for the latter is that since we offer the contract IWork for any developer to pick up an implement we can’t be 100% certain some one doesn’t use it in a direct implementation. It is no crime to do so even though the intent is to implement IWorkItem and decorate it with the metadata IWorkItemView. The real reason for not implementing the IWork direct is because we can remain lazy with the metadata if it is real metadata. If we want to start looking at the members of an implementation that does not have metadata we need to load that implementations assembly into the AppDomain.
The class implements IEnumerable<TValueMetadata>. This is used in the derived classes to Export the behavior and make the transformation accessible to our needy develoeprs. ;~)

To use this base class you have to do two things:

1) Implement the base class which constitutes implementing one method for compacting together a TValue and TMetadata into a TValueMetadata.
2) Apply the ExportAttribute with the parameter typeof(<your type>).

Let me show you my implementation of the compactor for compacting IWorkItems with IWorkItemView into instances of IWork:

[Export(typeof(IEnumerable<IWork>))]
internal class WorkCompactor : ValueMetadataCompactorBase<IWorkItem, IWorkItemView, IWork>
{
    protected override IWork Compact(Lazy<IWorkItem, IWorkItemView> lazyValueMetadata)
    {
        return new WorkWrapper(lazyValueMetadata);
    }

    private class WorkWrapper : IWork
    {
        private readonly Lazy<IWorkItem, IWorkItemView> lazyWorkWithMetadata;

        public WorkWrapper(Lazy<IWorkItem, IWorkItemView> lazyWorkWithMetadata)
        {
            this.lazyWorkWithMetadata = lazyWorkWithMetadata;
        }

        public bool HasWork
        {
            get { return lazyWorkWithMetadata.Value.HasWork; }
        }

        public void DoWork()
        {
            lazyWorkWithMetadata.Value.DoWork();
        }

        public int Order
        {
            get { return lazyWorkWithMetadata.Metadata.Order; }
        }

        public string Name
        {
            get { return lazyWorkWithMetadata.Metadata.Name; }
        }
    }
}

Notice:

The annoyance here is that I have to custom implement a boring IWork wrapper ourselves since I am not using a dynamic code approach.
The implementation of said IWork retains the lazy behavior in my implementation since I only forward calls to the members to the contained lazy instance.
The [Export(typeof(IEnumerable<IWork>))] attribute on the implementation is that which makes my IEnumerable<IWork> available to the rest of my code through MEF.
The implementation is marked as internal. MEF doesn’t care – it finds internal, protected and private implementations as well as public.. This means that user coders do not even know that this conversion exists. All the handling of lazy etc. is removed from the “subscribing” code.

Summary

Note to self: Write less epic blog posts. ;~)

I hope this post gives you some insight into how MEF works as well as a neat trick to simplify some of your MEF-Import code and lazy-metadata craziness. The functionality is very good to have but I really do not want library specific concerns to affect my code too much. Do you?

Cheers,

M. (the NoOpMan)

Technorati Tags: Microsoft,Arcitecture,Extensibility,MEF

Common Service Locator MEF Adapter now on .NET Framework 4.0

Magnus Mårtensson — Sat, 08 May 2010 20:25:01 GMT

I like the Common Service Locator. It’s nice and clean. Abstractions are good.

This is why I bothered Glenn Block about updating the MEF Adapter all the time as each version of Managed Extensibility Framework MEF came out. Finally he gave in and made me a contributor to the CSL project. – Fine – Do it your self, kind of! ;~) Glenn and I are good mates. I didn’t mind updating it just to have one thing that was in order. Hmmm… sounding bitter and disgruntled on a Saturday night, sorry.

Well now the MEF release is finally here as part of .NET Framework 4.0 and I’ve updated the MEF Adapter for Visual Studio 2010 and .NET 4.0.

Hope you like it and use it!

Cheers,

Technorati Tags: .NET,.NET 4.0,Inversion of Control,Depencency Injection

MEF + Azure + Visual Studio 2010 = Tricky but doable (part 2)

Magnus Mårtensson — Thu, 29 Apr 2010 20:19:28 GMT

In order to make Managed Extensibility Framework (MEF) a part of your Windows Azure project in Visual Studio 2010 while having a Test Project to test your code there is a nasty reference issue you have to resolve. I posted all the steps to resolving this issue in my recent post MEF + Azure + Visual Studio 2010 = Tricky but doable. but I forgot one step! Here is the final piece of the puzzle.

The remaining problem

Following the steps in my previous post (link above) you are good to go with your project in all respects save one. You can compile. You can run your Windows Azure application in the Development Fabric. You can run your tests in your Test Project.

The only thing you cannot do is refer from your tests to the type Tuple<T1, T2> (or Lazy<T>) in your Test Project. This will cause an “Ambiguous reference” error at compile time in your code:

Test Projects in Visual Studio 2010 have to run on .NET Framework 4.0. This new version of the framework includes the new Tuple class. However you cannot use .NET Framework 4.0 components on Windows Azure (until something like next month when Windows Azure upgrades).

Your Windows Azure project has to run on .NET Framework 3.5. Which means you have to download and run Preview 9 of MEF from the CodePlex site (link above) and use that. In my first post on this topic I showed how to get your preview (.NET 3.5) version into your (.NET 4.0) Test Project. If you follow these instructions you can compile and run your application.

However the final problem in this matter is the following: The type "’System.Tuple<T1, T2> exists in both ‘c:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0\mscorlib.dll’ and <your MEF assembly>.

There is a way to fix this issue so that you can refer to System.Tuple from your Test Project.

The Solution

What you need to do is eliminate one of the references. Since you can’t really, in any easy way, eliminate the .NET 4.0 version of mscorlib it’s perhaps best to try with the other one – the MEF (.NET 3.5) assembly.

Here are the steps to follow:

Download the source code to MEF Preview 9 from the CodePlex site.
Open up the ComponentModel Project. (Make sure you don’t get it converted to .NET Framework 4.0!)
Open the three files Tuple, Lazy and LazyOfTTMetadata. The first two are part of mscorlib 4.0 and the last one is part of the 4.0 version of MEF (System.ComponentModel.Composition.dll).
Rename the namespace to something you like; I chose System2 – why not?
Make the ComponentModel Project compile again. This is done by resolving all of the references from the old System.Tuple<T1, T2>, System.Lazy<T> and System.Lazy<T, TMetadata> to the new System2.Tuple<T1, T2>, System2.Lazy<T> and System2.Lazy<T, TMetadata>.
Once the project compiles; take this modified version of MEF (the System.ComponentModel.Composition.dll) and reference it to your solution instead of the real MEF Preview 9 version.
Now of course you have to resolve the usages of the three types that have moved again.

That’s it! Now your Windows Azure Project will be able to run MEF Preview 9 on .NET Framework 3.5 together with your Visual Studio 2010, .NET Framework 4.0, Test Project and both will be able to reference your new version of Tuple, and the two Lazies.

Summary

Sure this is a fix but it enables you to run MEF for open ended extensibility scenarios in Windows Azure right now. When Windows Azure upgrades to .NET Framework 4.0 you can simply drop this fix and refer to the .NET Framework 4.0 version of System.ComponentModel.Composition.dll.

Cheers,

Technorati Tags: .NET Framework,.NET,Extensibility

MEF + Azure + Visual Studio 2010 = Tricky but doable

Magnus Mårtensson — Wed, 28 Apr 2010 18:51:16 GMT

Putting Managed Extensibility Framework (MEF) in your Windows Azure Application while trying to test your code using Visual Studio 2010 does not work OOTB. It is quite easy to modify – in this case MEF – to enable this scenario. The same issue might result from using other libraries as well.

Now that Visual Studio 2010 is out you really do not want to remain on the old version. Sometimes there can be issues working with applications that need to support previous versions of .NET Framework – previous to .NET 4.0 that is. Here is one such issue and how you work around it.

UPDATE: There was one more problem that I did not cover in this post. Once you’ve read this. Continue reading here MEF + Azure + Visual Studio 2010 = Tricky but doable (Part 2) in order to get the final piece of the puzzle.

Note: This issue will go away within a couple of months when .NET Framework 4.0 is deployed to Windows Azure. But right now this may be an issue. Guess the blogging window for this topic is right now. ;~)

Note to self: Stop doing epic refactoring and development tasks on your small(ish) projects. This will halt your blogging.

Prologue

Testing in Visual Studio 2010 requires you to run the new version of the Microsoft.VisualStudio.QualityTools.UnitTestFramework. This artifact is built using .NET Framework 4.0.

If you go File –> New –> Windows Azure Cloud Service project and select an ASP.NET MVC 2.0 Web Role and choose to create also a Test Project for this solution you will get one Web App on .NET Framework 3.5 and one Test Project on .NET Framework 4.0. If you want to develop your Windows Azure Application using Visual Studio 2010 your Azure app requires .NET Framework 3.5 because Windows Azure is on .NET Framework 3.5.

Can you see a versioning conflict a’ brewin’? ;~)

The Problem

Some parts of Windows Azure Applications is a classic extensibility scenario. My colleague Peter and I used a standard .NET Framework library in preview called Managed Extensibility Framework (MEF) to handle extensibility. This worked fine for the old version of Visual Studio but results in a conflict in Visual Studio 2010. MEF includes a couple of small utility classes from .NET Framework 4.0 as part of the preview. These are embedded in the System.ComponentModel.Composition assembly that is MEF. If you try to reference this assembly from your Visual Studio 2010 test project you will first fail. Visual Studio is smart enough to instead not do as you ask but rather go and reference the assembly that is part of your .NET Framework 4.0 installation:

But if you use the preview version of MEF that is compiled against .NET Framework 3.5 in your Windows Azure Application you have to have this particular version referenced in your tests. If you run your tests using the 4.0 version of MEF in your Test Project and the 3.5 version in your Azure Role you will get this error:

It appears you cannot load the 3.5 version of MEF when you have already loaded a newer (4.0) version when you run your test.

The Solution

How do I add the version I want (3.5) then? Let’s resort to good old hand editing of the project file:

I opened up my Azure Role project file in notepad and copied my reference to MEF:

<Reference Include="System.ComponentModel.Composition">
<HintPath>..\lib\System.ComponentModel.Composition.dll</HintPath>
</Reference>

Then I opened up the Visual Studio 2010 Test Project file in notepad and replaced the following with the text above:

Hey presto! It works.

Summary

If there is a will there is a way. Due to the collision between the MEF Preview that works in .NET Framework 3.5 and the Visual Studio 2010 Test Project type which defaults to the .NET Framework 4.0 version it was non-trivial to use the MEF library in the Cloud.

Using MEF in Windows Azure is something I automatically turned to when I saw that my scenario had an open ended extensibility point. Since you know what it is you deploy to Windows Azure and if you are not using a bunch of third party libraries that might utilize extensibility through MEF it is unlikely that this problem will apply to you.

That said if you are stringent about using the right tool for the right job and strive to make your code designs and software architecture adhere to a good yet for your domain appropriate number of design patterns and techniques then MEF will very likely find it’s way into Your Application in the Cloud.

Cheers,

Technorati Tags: .NET Framework,.NET,Extensibility

Neo4j .NET Client over HTTP using REST and json

Magnus Mårtensson — Fri, 16 Apr 2010 09:51:09 GMT

Here it is; a Proof of Concept of the world’s first Neo4j .NET Client. In other words: Here follows a discussion on how to create a client library for communicating with a graph database over REST.

UPDATE: There is now a live CodePlex project for the realization of this concept; A .NET Client Library for Neo4j over HTTP using REST and JSON; http://neo4jrestsharp.codeplex.com/

Intro

Neo Technologies have come out with a Neo4j REST API for their popular Neo4j graph database. Since I am employed at Jayway and a couple of colleagues working with Neo are too a little bird by the name of Peter Neubauer tweeted in my ear that there was a free T-Shirt involved in making a working sample for .NET that communicated with the Neo4j server. Not only was that a handsome offer – I wanted to check out what the Architecture of such a client library for a graph database would look like.

I’ve recently been involved in client library abstractions and good architecture with the RESTful interface to Windows Azure Storage. (More on Azure and storage on my blog here, here, here, here and here.) Enough with the side tracking (sorry)!

Said and done; We set out to code up a working sample for .NET. I hacked the client code. The Neo-guys cheered me on and offered very helpful assistance when I ran into trouble or was just plane stumped on how to proceed with the API. A very nice team effort, including a remote team debugging session, later and we have a small and running POC

Tweeting on this effort it is notable how a bunch of people answer back with an interest in this idea for potential applications in the .NET world. I’ve been approached both on twitter (@noopman) and on LinkedIn with questions on this.

This blog post is to submit our findings to the world and hope that someone will be inspired to get going with a real .NET client library for Neo4j.

Architecture

First let’s look at the architecture for creating a client library for a RESTful service. This is a more general discussion that leads to our client library POC below. If your main reason for reading is the actual sample; scroll down.

When doing a client library abstractions and domain thinking comes into play. What we want is for the forgotten users (developers) of your application to work with their normal domain objects in the business code and for them to use an API that makes sense to their domain while talking to a highly specific graph database backend store. This means you have to keep all the RESTfulness, HTTP and json away from the domain code. Let me show you a sketch and see if I can make this explanation a bit less wordy:

At the other side of the Cloud from our .NET code is a Java based graph database. It is friendly enough to expose a RESTful API so that we may speak to it.

Doing so we want to create a client library that is capable of a couple of things:1) Taking care of serializing objects to json (and deserializing them back again). 2) Knowing about HTTP requests and how to handle them. In our sample we employed the power of the HttpWebRequest class to handle things like HTTP Headers, Content Types, URIs etc. 3) Finally this library must understand to expose an interface to your code that you can use from a specialized Data Access Component.

To protect the business layer from knowing anything about the specifics of the Neo4j API you really should use a specialized Data Access Component. There is now law being broken if you don’t do that. However a good rule of thumb is to add abstraction layers in your system to shield your domain code in your business components from any specifics of an underlying storage structure. The details of serialization of objects to json and making requests over HTTP are not a natural fit to the business problem you are solving. The code that does this separation is code you write yourself in your application. Only you know how to adapt a foreign library and technology to your domain. The sole purpose of this code, in other words, is to abstract away from your business logic the underlying technologies used to persist and query data used in the application. This component translates between the two worlds of your business logic and the techniques used to store data in a graph database. Most of the specific technical details of storage are wrapped up in the Client Lib mind you but the Data Access Layer DAL knows about this problem domain. It understands that the persistence indeed is a REST based service over HTTP with all that this entails. This component in your code can handle synchonisity and take care of specific errors or faults that may occur when using the storage. All of this is done in a language that is fairly close to the API that Neo4j exposes. In the end this component translates into terms useful to the domain in which your business operates. Instead of saying .PostNode() in your code perhaps you instead expose a .StorePerson() action.

What we achieve is a good and sound (and classic) layering model: Business Logic – DAL – Client Lib.

There is one more observation to make in the picture and that is the matter of IoC - Inversion of Control. In order to keep your code testable, maintainable and untangled from the implementations at lower layers you should always employ the method of Inversion of Control – making your business code for instance not reference the DAL direct but instead use a common contract to do so. The Depencency Inversion Principle states briefly that a higher layer should never depend direct on a lower layer but instead depend on an abstraction. If this is true you are able to test your business logic without using the actual underlying storage. This is for instance key in a TDD – Test Driven Development development style.

Code that depends on abstractions can be tested very quickly and does not depend on specific data in a data store to be correct in order to test the intended behavior. The Business Logic uses a contract for the DAL and the DAL implements it. Then you use something called an IoC container (read on at Wikipedia in the link above) to create the DAL instances and hand them to the Business Logic employing the technique of DI -Dependency Injection. This separation enables you to modify, re-deploy or even exchange DAL components without having to touch your Business Layer at all. If separating the Business Logic from the DAL using contracts is important it is almost event more important to do so in the lower layer that in run time speaks direct to the data store! You never want to end up in the situation where in order to test your DAL code you have to have access to a specific data store with specific data in it. Not only is that an inconvenience; it also makes executing your tests run as slow as transferring data over the Internet compared to as fast as your CPU can execute code. From experience it is an obvious conclusion that tests that are slow to run are not in fact run. The conclusion from all of these IoC discussions can only be that it is imperative that your Client Library comes with an abstraction layer.

What did we do with the Architecture in our POC?

All of these architectural considerations are of course premature in a POC. This is why in our sample we did not really separate the DAL code from the Client Lib. However to prove the point of IoC and DI I did however separate out the Business Logic from the Data Access logic using a simple contract and an equally simple Factory Method Pattern. Instead of new:ing up a DAL component in my code I called a factory method to access an abstraction of the DAL.

Our POC sample

Our code sample creates a small node graph with a few relationships. Then it makes a traversal of the graph to find all who “knows” someone according to the relationships. Finally it deletes the graph again.

Here is the graph we create in our sample: http://wiki.neo4j.org/content/The_Matrix We only create names on nodes and types on relationships in our simple sample.

And here is also the code so that you may download and follow along with our explanation your self:

Neo4j.POC.zip

First, in accordance with architectural considerations (above) we get hold of our DAL using a factory method. We also create a new Person object and store him (in this case Thomas Anderson – Neo):

INeo4jStorageClient neo4JStorageClient = 
Neo4jStorageFactory.GetNodeStorageClient(neo4jServerUri);

neo = new Person("Thomas Anderson");
neo4JStorageClient.StoreNode(neo);


As you can see in our business code this is just standard .NET classes being instantiated and a “Neo” Person is stored in a very normal way. In the business logic code there is no real details on how this storage happens. There is of course a “node” concept and you can consider for your self if that should be hidden away from business code too.
Since this is a POC the implementation of .StoreNode(neo) is very very trivial. If this were a real client library we would have included things like a library to transform (serialize) the neo-Person node into a json string. Now we just do it statically:
public void StoreNode(Person person)
{
    var name = person.Name;

    string jsonString = @"{""Name"":""" + name + @"""}";

    var nodeProtocol = new NodeProtocol(uri);

    var nodeId = nodeProtocol.PostNode(jsonString);
    person.Id = nodeId;
}

The interesting part here is the existence of a NodeProtocol implementation. I’ve found it useful working with a remote API like Windows Azure Storage and also with this Neo4j REST API to implement methods in C# code that correlate very closely to the API methods available. In our case we have a PostNode operation on the REST API and consequently we have a .PostNode() method in the C# implementation of that HTTP method call.
public int PostNode(string nodeData)
{
    var uri = HttpHelper.ConcatUri(baseUri, "node");

    HttpWebRequest request =
        Request.CreateWebRequest(HttpWebRequestMethod.POST, uri);

    HttpHelper.AddBody(request, nodeData);

    var response = requestMaker.MakeRequest(request);

    return HttpHelper.FindNodeId(response.Item3);
}

I won’t bore you with the exact details of all of the helper classes. To dig in to those it is better if you just download the code and look at the details your self.
As you can see we create a HttpWebRequest, add some headers to it (in the CreateWebRequest method) specifying things like request.Accept = "application/json"; meaning we accept a json formatted response. Also we specify that the body we send out is formatted as json with request.ContentType = "application/json";. Finally we make the call and return the id of the node we just created.
Again this is a solution bordering on triviality but still it is to the point of making a REST request and receiving a response. Also there is no code to handle any errors present here and the request is done synchronously.
What does the actual HTTP REST and json request and response look like for this simple sample?
Request:
POST http://try.neo4j.org:9999/node HTTP/1.1
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 26
Expect: 100-continue
Connection: Keep-Alive

{"Name":"Thomas Anderson"}

Response (note that the added line breaks in the response body were added by me):
HTTP/1.1 201 Created
server: grizzly/1.8.1
Content-Encoding: UTF-8
Location: http://try.neo4j.org:9999/node/1
Content-Type: application/json
Content-Length: 860
Date: Fri, 16 Apr 2010 08:36:45 GMT

{"incoming typed relationships":
http://try.neo4j.org:9999/node/1/relationships/in/{-list|&|types},
"incoming relationships":http://try.neo4j.org:9999/node/1/relationships/in,
"all relationships":http://try.neo4j.org:9999/node/1/relationships/all,
"create relationship":http://try.neo4j.org:9999/node/1/relationships,
"data":{"Name":"Thomas Anderson"},
"traverse":http://try.neo4j.org:9999/node/1/traverse/{returnType},
"property":http://try.neo4j.org:9999/node/1/properties/{key},
"self":http://try.neo4j.org:9999/node/1,
"properties":http://try.neo4j.org:9999/node/1/properties,
"all typed relationships":
http://try.neo4j.org:9999/node/1/relationships/all/{-list|&|types},
"outgoing typed relationships":
http://try.neo4j.org:9999/node/1/relationships/out/{-list|&|types},
"outgoing relationships":http://try.neo4j.org:9999/node/1/relationships/out}

As you can see the Neo4j REST API is self explanatory. Creating a node gives you back a response with all the things you can now do with this new node over the API. Since this quite a lot of data when the only thing that really matters is in the header (Location: http://try.neo4j.org:9999/node/1) I’ve already given feedback to the Neo team that I’d like a simple Uri switch (like APIDocumentation=0) to shut off the detailed response. But in fact I do like that you get so many Uris back that you may use direct. Depending on your needs this is either good or to verbose a payload to get back in your response. As you can see the node we created got the Id set to 1.
Now we proceed to create all the other nodes and in a very similar fashion the relationships between nodes. Here is the relationship that says that “Neo knows Trinity” (who has a node id of 2). This is a POST operation to /node/1/relationships:
POST http://try.neo4j.org:9999/node/1/relationships HTTP/1.1
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 56
Expect: 100-continue

{"type":"Knows","to":http://try.neo4j.org:9999/node/2}
Once the node graph for the Matrix is created we proceed to make a traversal of the graph. We want to find anyone who knows someone throughout the graph.
The request looks like this in REST (after a similar path through the code as above with a call from business logic to DAL and through to the client lib that makes the actual call using REST):
POST http://try.neo4j.org:9999/node/1/traverse/node HTTP/1.1
Accept: application/json
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 57
Expect: 100-continue
Connection: Keep-Alive

{ "relationships": { "type": "Knows" }, "max depth": 10 }

Again we simplified to a “max depth” of 10 relationships deep into the graph which works fine for our sample.
The result is a list of all the nodes that match. Because of the self explanatory API mentioned above the result also gives you actions to perform on each node that comes back in the response thus making the response body quite long. Rather than pasting the whole thing here into the blog post you can try it out your self (by downloading the application) and you will get a result exactly like this from the app:
 
Neo knows Trinity and Morpheus, who knows Cypher, who in turn knows Agent Smith.
Difficulties
We had only two real challenges in coding this POC.
The first one was an issue with streams and the optional so called Byte Order Mark (BOM) that you can put at the head of a stream. I’ve posted on this here: Would you like a Byte Order Mark to go with that?.
The other challenge was that I wrote a small app for Windows to show our results. But the guys on the Neo team are certainly not Windows users. The wanted to show the code on Mono. Just to be sure of compatibility I scaled back to a good old Windows Forms demo app in .NET Framework 3.0. This worked great on Mono which was our intent. But the drawback was that the library I’d found to handle object serialization didn’t respond well to that old version of the framework. That’s why I replaced my lib with static string creations (as you can see in the implementation of the .StoreNode() method above).
What have we learned from this?
First of all creating a client library for .NET to speak REST and json over HTTP to Neo4j is very doable; quite easy and straight forward.
Second (and that’s just a word of warning); keep your layers separated. This goes for all applications I guess but if you start to put code to create instances of web requests with web methods and content types and such in your business code you will end up in a very tangled and messy situation very fast which will be a pain and a high cost to maintain.
Finally we have considered looking into using WCF data services (and OData – the Open Data Protocol) and also possibly Linq – Language Integrated Query on the client side to make implementations in the library even more powerful and useful. And we do need a object to json serializer library.
Oh and here is another post on using the Neo4j REST API by the team itself: http://blog.neo4j.org/2010/04/neo4j-rest-server-part1-get-it-going.html
That’s it!
We hope this is inspirational enough for you to get going with your own samples or even client libraries in .NET to speak to Neo4j. If you do; please get back in touch with us!
Cheers,
M.
Technorati Tags: .NET,Graph Databases,C#,REST,json

Would you like a Byte Order Mark to go with that?

Magnus Mårtensson — Wed, 14 Apr 2010 06:46:20 GMT

It is possible to encode a little bit of metadata at the beginning of your byte streams to let the stream itself carry information on how it has been encoded. This is known as a Byte Order Mark (BOM) and it is as far as we know completely optional. Some .NET Framework implementations add this BOM to the start of streams given that they are to have a specific encoding. Here is how to find the BOM, and if necessary, remove it from your stream.

Context

Recently Neo4j graph database announced a REST interface. A couple of colleagues at Jayway are working on this. This was (very much) a too intriguing temptation to resist. I naturally had to write a small POC .NET based REST client to be able to talk to that server. (Blog post on this is pending within a few days.) While trying to send REST data in HTTP requests to the server we came across something interesting that caused compatibility issues between the Java and .NET worlds. The culprit was that optional byte order mark that I unintentionally embedded in my streams of bytes. The server did not recognize them. In a joint effort from both sides of the world (.NET client and Java server) we were able to find and identify the problem. Not that this is a new problem - It is very well documented in Wikipedia (link above). However I feel that one more post on this topic with a good test to show what’s going on is warranted.

A very good tool was also used to nail down the problem; Fiddler2. This little baby of a program is an http proxy that sits on your machine and listens to your http traffic. Using the tool you can inspect exactly what you are sending over the wire in your http requests and also view the responses coming back. Sweet stuff indeed. This is a MUST tool for you if you develop anything that sends data over HTTP.

Using Fiddler2 we were able to observe that when sending a stream of bytes as a body for an http POST the byte length, when the request came from my code, was 108 bytes. However when I used the built in request builder in Fiddler2 to reproduce and resend the exact same request it was reported that the length was 105. Why was there a difference in three (3) bytes?

Problem was that these tree bytes, as I stated above, caused the server to throw a fit. It could not recognize the body of the message to be the expected json string in UTF8 encoding. Json, I’ve learned, is always in UTF8 encoding.

The java guys googled and I binged ;~) and we found that these three bytes is an optional “byte order mark” or BOM. It is not wrong to send these three bytes and it is not wrong not to send them. Guess what? .The .NET world tends to send these extra bytes and everyone else tends not to. Why not have a bit of more incompatibility, right? It’s not like we want to talk to each other anyway!

What’s up with the BOM? What is it’s purpose?

The BOM is a way to add a kind of metadata inside of a stream of bytes instead of sending some actual metadata along next to your stream. Option one is saying “Here is my byte stream and btw it is in UTF8 encoding”. The other way to do it is saying “Here is my byte stream and if you look at the first three bytes you can read the encoding of it”. Which is better? I can’t say I care all that much other than the fact that it caused us a problem when I tried to send requests from .NET code to the Neo4j server.

How can you then – finally – handle these three bytes?

Well that depends… on what you intend to do. What I can show you is a piece of code that tells you exactly what this is and then you can copy that behavior into your code and modify it to serve your purpose. The code below should be pretty self explanatory but just to be sure. I take the string "foo" and encode it into a MemoryStream using a StreamWriter. The issue is that I tell the stream writer to use Encoding.UTF8 and this is where .NET Framework adds the BOM. The resulting stream is not 3 bytes as perhaps expected. Instead it is 6 bytes long. What you have to do if you read this stream ‘raw’ – by hand – in some library, is skip over the first three bytes. A better way to do it is to use a reader that handles the BOM. Finally if you don’t want to add the BOM in the first place you can write your bytes yourself in a more ‘raw’ fashion byte by byte to the stream. As you can see .NET Framework is good enough to have a way to find an actual BOM for different encodings. The UTF8 encoding does it this way: Encoding.UTF8.GetPreamble() (msdn library link to .GetPremable()).

[TestMethod]
public void StreamWriter_with_encoding_writes_ByteOrderMark()
{
    string str = "foo";

    var memStream = new MemoryStream();

    // Test
    var writer = new StreamWriter(memStream, Encoding.UTF8);
    writer.Write(str);
    writer.Flush();

    Assert.AreEqual(6, memStream.Length);

    var buffer = new byte[3];

    memStream.Position = 0;
    memStream.Read(buffer, 0, 3);
    string result = Encoding.UTF8.GetString(buffer);

    var preamble = Encoding.UTF8.GetPreamble();
    var byteOrderMarkUtf8 = Encoding.UTF8.GetString(preamble);

    Assert.AreEqual(byteOrderMarkUtf8, result);

    memStream.Position = 3;
    memStream.Read(buffer, 0, 3);
    result = Encoding.UTF8.GetString(buffer);

    Assert.AreEqual(str, result);

    // Alternative: Write the bytes yourself to skip the BOM

    memStream = new MemoryStream();

    foreach (var @byte in Encoding.UTF8.GetBytes(str))
    {
        memStream.WriteByte(@byte);
    }

    Assert.AreEqual(3, memStream.Length);
}

You can also if you like compare bytes for the premable one by one rather than converting to a string comparison.

As you can see this BOM can be handled easily if you like. The thing is I did not have a clue that it was there.

Oh – and btw now the Neo4j server accepts REST json bodys that both can have and skip the BOM! Good ‘bug’ to solve or ‘feature’ to have.

Cheers,

Technorati Tags: .NET,Arcitecture,json,Java,REST