Techie.notepad

Entity Framework and Show & Tell .NET community event in Swenug Skåne in Malmo, Sweden

Magnus Mårtensson — Fri, 20 Aug 2010 05:48:14 GMT

If you are nearby or live here don’t be square! First NUG event in Skåne, Sweden for the fall is one with two agenda entries; Entity Framework and Show & Tell raffle.

Dag König from Microsoft DPE Sweden will join us to give the latest info Entity Framework.

We will have a raffle on who will get to present their latest cool/interesting/noteworthy thing from .NET development with a prize raffled among the participants.

Join Swenug and sigh up for the event here: http://www.swenug.se/events/skaane-entity-framework-show

Sponsor is Jayway.

Cheers,

Technorati Tags: Swenug,NUG,Entity Framework

Lazy<T> with implicit cast to T can give you some pretty code

Magnus Mårtensson — Thu, 05 Aug 2010 19:57:34 GMT

Adding an implicit cast functionality to Lazy<T> can give you an option to writing null checkers in your delayed instantiation properties.

The problem

Today in our project a colleague was cussing over the fact that he kept writing the same block of code over and over again when null checking a property with a backing field.

Given the class Foo:

private Foo oldFooProp;
public Foo OldFooProp
{
    get 
    {
        if (oldFooProp == null)
        {
            oldFooProp = new Foo();
        }
        return oldFooProp;
    }
}

Have any of you written something like this before? It’s a bunch of lines of code that you tend to write (in small variations) over and over again. Sure you can compact it a little but it in the order of 5 – 10 lines of annoying repetitive code.

Some use the null coalescing operator (??) version that looks like this:

public Foo OldAlternativeFooProp
{
    get
    {
        return oldFooProp ?? (oldFooProp = new Foo());
    }
}

This version is a little easier to compact down to even a single line.

But we (my colleague and I) can do you one better:

Here is an optional approach

private Lazy<Foo> foo = new Lazy<Foo>();

public Foo Foo
{
    get
    {
        return foo;
    }
}

Actually I’d prefer to do that like this but then you have to know the solution below and be comfortable with it:

public readonly Lazy<Foo> Foo = new Lazy<Foo>();

But wait a minute: That doesn’t work you say, right? You can’t return a Lazy<Foo> as a Foo you have to write theFoo.Value.

The Trick

The secret trick is to inherit Lazy<T> and implement an implicit operator for T on it!

namespace Jayway.com.Sys
{
    /// <summary>
    /// A version of <see cref="Sys.Lazy{T}"/> with the twist of being implicitly castable to <typeparamref name="T"/>.
    /// </summary>
    /// <typeparam name="T">The type of lazy instance we are creating.</typeparam>
    public class Lazy<T> : System.Lazy<T>
    {
        // Ctors removed for brevity

        /// <summary>
        /// Cast the <see cref="Lazy{T}"/> to it's contained <typeparamref name="T"/>
        /// </summary>
        /// <param name="lazyT">The lazy to cast.</param>
        /// <returns>The instance of <see cref="Lazy{T}.Value"/></returns>
        public static implicit operator T(Lazy<T> lazyT)
        {
            return lazyT.Value;
        }
    }
}

Now you can implicitly cast a Lazy<T> to it’s contained type T.

A few alternative constructions for it might look like this:

public LazyTests()
{
    // With parameter
    theFoo = new Lazy<Foo>(() => new Foo("bar"));

    // With Inversion Of Control
    IServiceLocator serviceLocator = GetServiceLocator();
    theFoo = new Lazy<Foo>(() => serviceLocator.GetInstance());
}

I kind of like this code. It’s compact. It’s easy to read. It’s pretty!

Hope you like it too – HTH!

Cheers,

Technorati Tags: C#,.NET

Handling Lazy + Metadata instances in Managed Extensibility Framework (MEF)

Magnus Mårtensson — Wed, 04 Aug 2010 19:21:32 GMT

Have you ever wanted to dynamically choose from many potential implementations of some code? Would you like to have the power of metadata filtering and lazy instantiation of the selected implementation? Here is how to do just that using Managed Extensibility Framework (MEF) and also a neat trick to handle your implementation/metadata pairs that I recently put to good use. Code samples and optimizations follow so don’t worry; I won’t remain all theoretical in this post. (MEF is part of .NET Framework 4.0 but is available also on CodePlex for other releases – MEF Community Site.)

Up front I want to show what I am aiming for so you can evaluate if this is a worth while read: Using the more advanced features of MEF suddenly I find I use these pretty gruesome variable and parameter declarations: Lazy<IWorkItem, IWorkItemView>. After applying the fix in this post I am left with the exact same behavior and this much improved declaration: IEnumerable<IWork>.

Background on MEF

Using MEF you can easily juggle the concept of lazily (using System.Lazy<T>) importing an open ended set of instances of a given contract. This is built into the library for you. It takes some getting used to the concept but it is quite powerful. As if this wasn’t enough you can also add metadata to the contracts (with the MEF version of Lazy<T> – System.Lazy<T, TMetadata>) to enable you to select which of the - potentially many - implementations of the contract that you want to lazily load into your app domain and instantiate. Does this all sound like geek speech? Well it sure is but as I said this is a very powerful concept. Given that you have a good requirement for this scenario you never have to code extensibility code yourself ever again. I would go so far as to say that MEF makes the extensibility scenario a solved problem!

My Scenario

Working with improving the Worker role concept in Windows AzureI I recently had a use for an open ended set of implementations of what I called “work methods”. I wound up making my work methods implementations of IWorkItem. The metadata I added to my work method classes I decorated with IWorkItemView metadata contract. This is what the two contracts look like:

public interface IWorkItem
{
    bool HasWork { get; }
    void DoWork();
}

public interface IWorkItemView
{
    string Name { get; }
    int Order { get; }
}

When using these contracts lazily in MEF I get the ugly instances of Lazy<IWorkItem, IWorkItemView>. There is some really cool functionality here for sure but I just can’t bare the ugliness of the statement as I begin to pass these instances around in method calls etc. The cool thing with MEF is that I don’t know beforehand how many implementations I have. All I need to know is that I can get them all:

[ImportMany]
IEnumerable<Lazy<IWorkItem, IWorkItemView>> Work { get; set; }

This gives me a whole enumeration of lazy instances that are combined with the metadata of the metadata contract. On each instance of the lazy works you call .Metadata to access the IWorkItemView contract and .Value to instantiate and return the instance of the IWorkItem. Now I don’t know about you but this is beginning to look kind of hairy to me. After all in my code I am not really concerned with any laziness or metadata contracts. Not only that – provided you are a coder that believes in separation of concerns there is one more issue. To explain the last statement let’s go back to my Windows Azure scenario:

I import my many lazy instances with metadata into my code. Then I’d like to begin to use those instances. I wrote some code to filter out which instance I wanted to use. I wrote code to execute the selected instance. I wanted to separate these two code blocks from each other and I also wanted to be able to exchange filtering methods on the fly. I could continue but some of you have certainly begun to see the problem here. I was now passing around parameters of type Lazy<IWorkItem, IWorkItemView>. That’s not exactly pretty! What I found that I was conceptually doing was rather using an instance of something that had the combined contract of my two work-method, and work-metadata contracts:

public interface IWork : IWorkItem, IWorkItemView { }

Passing around parameters of type IWork is a lot better than Lazy<IWorkItem, IWorkItemView>. You have to all agree to that, right?

So now – finally – the issue becomes that of transforming one instance into the other. I also want to keep the Lazy capability. The transformation technique is mostly built into an abstract base class I’ve written which you can download here:

MEFContrib.LazyDomainCompactor.zip

I have also put the code in the MEF Contrib project at Github.

Using the base class comes with a little bit of custom implementation per conversion you want to do from from “lazy with metadata” to “other domain object”. The alternative would have been for me to use a dynamic code generation approach for the implementation but I felt that dynamic code in this instance is a serious overkill. Besides we also need to make sure MEF picks up our implementation which in this case means we have to apply an attribute to our implementation.

The solution

To re-iterate: We had the ImportMany statement above which is a bit cumbersome. Instead we are aiming for this:

[Import]
IEnumerable<IWork> Work { get; set; }

Please notice the two different attributes ImportMany and Import. There is an important difference. The first one will find a lot of implementations of the contract (zero to many) while the latter will find only one exact match to the contract. In the first case we want MEF to find all IWorkItem implementations that are decorated with the IWorkItemView metadata. In the second case we want to find one single collection of IWork.

The trick is to collect all the instances (ImportManyAttribute) and hand them back out (ExportAttribute) as a single collection. What we do in the base class is import the many lazy with metadata instances and then immediately export a whole collection of the new contract to be picked up somewhere in the code as needed.

Details

public abstract class ValueMetadataCompactorBase<TValue, TMetadata, TValueMetadata> :
    IEnumerable<TValueMetadata>
    where TValueMetadata : TValue, TMetadata
{
    protected abstract TValueMetadata Compact(Lazy<TValue, TMetadata> lazyValueMetadata);

    [ImportMany(AllowRecomposition = true)]
    internal IEnumerable<Lazy<TValueMetadata>> ValueMetadatas { get; set; }

    [ImportMany(AllowRecomposition = true)]
    internal IEnumerable<Lazy<TValue, TMetadata>> ValuesWithMetadata { get; set; }

     // Cliped IEnumerable implementation for brevity [...]
}

There are a few things to notice here:

We ImportMany instances of the Lazy<TValue, TMetadata> and also for good measure we ImportMany istances of Lazy<TValueMetadata>. The reason for the latter is that since we offer the contract IWork for any developer to pick up an implement we can’t be 100% certain some one doesn’t use it in a direct implementation. It is no crime to do so even though the intent is to implement IWorkItem and decorate it with the metadata IWorkItemView. The real reason for not implementing the IWork direct is because we can remain lazy with the metadata if it is real metadata. If we want to start looking at the members of an implementation that does not have metadata we need to load that implementations assembly into the AppDomain.
The class implements IEnumerable<TValueMetadata>. This is used in the derived classes to Export the behavior and make the transformation accessible to our needy develoeprs. ;~)

To use this base class you have to do two things:

1) Implement the base class which constitutes implementing one method for compacting together a TValue and TMetadata into a TValueMetadata.
2) Apply the ExportAttribute with the parameter typeof(<your type>).

Let me show you my implementation of the compactor for compacting IWorkItems with IWorkItemView into instances of IWork:

[Export(typeof(IEnumerable<IWork>))]
internal class WorkCompactor : ValueMetadataCompactorBase<IWorkItem, IWorkItemView, IWork>
{
    protected override IWork Compact(Lazy<IWorkItem, IWorkItemView> lazyValueMetadata)
    {
        return new WorkWrapper(lazyValueMetadata);
    }

    private class WorkWrapper : IWork
    {
        private readonly Lazy<IWorkItem, IWorkItemView> lazyWorkWithMetadata;

        public WorkWrapper(Lazy<IWorkItem, IWorkItemView> lazyWorkWithMetadata)
        {
            this.lazyWorkWithMetadata = lazyWorkWithMetadata;
        }

        public bool HasWork
        {
            get { return lazyWorkWithMetadata.Value.HasWork; }
        }

        public void DoWork()
        {
            lazyWorkWithMetadata.Value.DoWork();
        }

        public int Order
        {
            get { return lazyWorkWithMetadata.Metadata.Order; }
        }

        public string Name
        {
            get { return lazyWorkWithMetadata.Metadata.Name; }
        }
    }
}

Notice:

The annoyance here is that I have to custom implement a boring IWork wrapper ourselves since I am not using a dynamic code approach.
The implementation of said IWork retains the lazy behavior in my implementation since I only forward calls to the members to the contained lazy instance.
The [Export(typeof(IEnumerable<IWork>))] attribute on the implementation is that which makes my IEnumerable<IWork> available to the rest of my code through MEF.
The implementation is marked as internal. MEF doesn’t care – it finds internal, protected and private implementations as well as public.. This means that user coders do not even know that this conversion exists. All the handling of lazy etc. is removed from the “subscribing” code.

Summary

Note to self: Write less epic blog posts. ;~)

I hope this post gives you some insight into how MEF works as well as a neat trick to simplify some of your MEF-Import code and lazy-metadata craziness. The functionality is very good to have but I really do not want library specific concerns to affect my code too much. Do you?

Cheers,

M. (the NoOpMan)

Technorati Tags: Microsoft,Arcitecture,Extensibility,MEF

Common Service Locator MEF Adapter now on .NET Framework 4.0

Magnus Mårtensson — Sat, 08 May 2010 20:25:01 GMT

I like the Common Service Locator. It’s nice and clean. Abstractions are good.

This is why I bothered Glenn Block about updating the MEF Adapter all the time as each version of Managed Extensibility Framework MEF came out. Finally he gave in and made me a contributor to the CSL project. – Fine – Do it your self, kind of! ;~) Glenn and I are good mates. I didn’t mind updating it just to have one thing that was in order. Hmmm… sounding bitter and disgruntled on a Saturday night, sorry.

Well now the MEF release is finally here as part of .NET Framework 4.0 and I’ve updated the MEF Adapter for Visual Studio 2010 and .NET 4.0.

Hope you like it and use it!

Cheers,

Technorati Tags: .NET,.NET 4.0,Inversion of Control,Depencency Injection

MEF + Azure + Visual Studio 2010 = Tricky but doable (part 2)

Magnus Mårtensson — Thu, 29 Apr 2010 20:19:28 GMT

In order to make Managed Extensibility Framework (MEF) a part of your Windows Azure project in Visual Studio 2010 while having a Test Project to test your code there is a nasty reference issue you have to resolve. I posted all the steps to resolving this issue in my recent post MEF + Azure + Visual Studio 2010 = Tricky but doable. but I forgot one step! Here is the final piece of the puzzle.

The remaining problem

Following the steps in my previous post (link above) you are good to go with your project in all respects save one. You can compile. You can run your Windows Azure application in the Development Fabric. You can run your tests in your Test Project.

The only thing you cannot do is refer from your tests to the type Tuple<T1, T2> (or Lazy<T>) in your Test Project. This will cause an “Ambiguous reference” error at compile time in your code:

Test Projects in Visual Studio 2010 have to run on .NET Framework 4.0. This new version of the framework includes the new Tuple class. However you cannot use .NET Framework 4.0 components on Windows Azure (until something like next month when Windows Azure upgrades).

Your Windows Azure project has to run on .NET Framework 3.5. Which means you have to download and run Preview 9 of MEF from the CodePlex site (link above) and use that. In my first post on this topic I showed how to get your preview (.NET 3.5) version into your (.NET 4.0) Test Project. If you follow these instructions you can compile and run your application.

However the final problem in this matter is the following: The type "’System.Tuple<T1, T2> exists in both ‘c:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0\mscorlib.dll’ and <your MEF assembly>.

There is a way to fix this issue so that you can refer to System.Tuple from your Test Project.

The Solution

What you need to do is eliminate one of the references. Since you can’t really, in any easy way, eliminate the .NET 4.0 version of mscorlib it’s perhaps best to try with the other one – the MEF (.NET 3.5) assembly.

Here are the steps to follow:

Download the source code to MEF Preview 9 from the CodePlex site.
Open up the ComponentModel Project. (Make sure you don’t get it converted to .NET Framework 4.0!)
Open the three files Tuple, Lazy and LazyOfTTMetadata. The first two are part of mscorlib 4.0 and the last one is part of the 4.0 version of MEF (System.ComponentModel.Composition.dll).
Rename the namespace to something you like; I chose System2 – why not?
Make the ComponentModel Project compile again. This is done by resolving all of the references from the old System.Tuple<T1, T2>, System.Lazy<T> and System.Lazy<T, TMetadata> to the new System2.Tuple<T1, T2>, System2.Lazy<T> and System2.Lazy<T, TMetadata>.
Once the project compiles; take this modified version of MEF (the System.ComponentModel.Composition.dll) and reference it to your solution instead of the real MEF Preview 9 version.
Now of course you have to resolve the usages of the three types that have moved again.

That’s it! Now your Windows Azure Project will be able to run MEF Preview 9 on .NET Framework 3.5 together with your Visual Studio 2010, .NET Framework 4.0, Test Project and both will be able to reference your new version of Tuple, and the two Lazies.

Summary

Sure this is a fix but it enables you to run MEF for open ended extensibility scenarios in Windows Azure right now. When Windows Azure upgrades to .NET Framework 4.0 you can simply drop this fix and refer to the .NET Framework 4.0 version of System.ComponentModel.Composition.dll.

Cheers,

Technorati Tags: .NET Framework,.NET,Extensibility

MEF + Azure + Visual Studio 2010 = Tricky but doable

Magnus Mårtensson — Wed, 28 Apr 2010 18:51:16 GMT

Putting Managed Extensibility Framework (MEF) in your Windows Azure Application while trying to test your code using Visual Studio 2010 does not work OOTB. It is quite easy to modify – in this case MEF – to enable this scenario. The same issue might result from using other libraries as well.

Now that Visual Studio 2010 is out you really do not want to remain on the old version. Sometimes there can be issues working with applications that need to support previous versions of .NET Framework – previous to .NET 4.0 that is. Here is one such issue and how you work around it.

UPDATE: There was one more problem that I did not cover in this post. Once you’ve read this. Continue reading here MEF + Azure + Visual Studio 2010 = Tricky but doable (Part 2) in order to get the final piece of the puzzle.

Note: This issue will go away within a couple of months when .NET Framework 4.0 is deployed to Windows Azure. But right now this may be an issue. Guess the blogging window for this topic is right now. ;~)

Note to self: Stop doing epic refactoring and development tasks on your small(ish) projects. This will halt your blogging.

Prologue

Testing in Visual Studio 2010 requires you to run the new version of the Microsoft.VisualStudio.QualityTools.UnitTestFramework. This artifact is built using .NET Framework 4.0.

If you go File –> New –> Windows Azure Cloud Service project and select an ASP.NET MVC 2.0 Web Role and choose to create also a Test Project for this solution you will get one Web App on .NET Framework 3.5 and one Test Project on .NET Framework 4.0. If you want to develop your Windows Azure Application using Visual Studio 2010 your Azure app requires .NET Framework 3.5 because Windows Azure is on .NET Framework 3.5.

Can you see a versioning conflict a’ brewin’? ;~)

The Problem

Some parts of Windows Azure Applications is a classic extensibility scenario. My colleague Peter and I used a standard .NET Framework library in preview called Managed Extensibility Framework (MEF) to handle extensibility. This worked fine for the old version of Visual Studio but results in a conflict in Visual Studio 2010. MEF includes a couple of small utility classes from .NET Framework 4.0 as part of the preview. These are embedded in the System.ComponentModel.Composition assembly that is MEF. If you try to reference this assembly from your Visual Studio 2010 test project you will first fail. Visual Studio is smart enough to instead not do as you ask but rather go and reference the assembly that is part of your .NET Framework 4.0 installation:

But if you use the preview version of MEF that is compiled against .NET Framework 3.5 in your Windows Azure Application you have to have this particular version referenced in your tests. If you run your tests using the 4.0 version of MEF in your Test Project and the 3.5 version in your Azure Role you will get this error:

It appears you cannot load the 3.5 version of MEF when you have already loaded a newer (4.0) version when you run your test.

The Solution

How do I add the version I want (3.5) then? Let’s resort to good old hand editing of the project file:

I opened up my Azure Role project file in notepad and copied my reference to MEF:

<Reference Include="System.ComponentModel.Composition">
<HintPath>..\lib\System.ComponentModel.Composition.dll</HintPath>
</Reference>

Then I opened up the Visual Studio 2010 Test Project file in notepad and replaced the following with the text above:

Hey presto! It works.

Summary

If there is a will there is a way. Due to the collision between the MEF Preview that works in .NET Framework 3.5 and the Visual Studio 2010 Test Project type which defaults to the .NET Framework 4.0 version it was non-trivial to use the MEF library in the Cloud.

Using MEF in Windows Azure is something I automatically turned to when I saw that my scenario had an open ended extensibility point. Since you know what it is you deploy to Windows Azure and if you are not using a bunch of third party libraries that might utilize extensibility through MEF it is unlikely that this problem will apply to you.

That said if you are stringent about using the right tool for the right job and strive to make your code designs and software architecture adhere to a good yet for your domain appropriate number of design patterns and techniques then MEF will very likely find it’s way into Your Application in the Cloud.

Cheers,

Technorati Tags: .NET Framework,.NET,Extensibility

Neo4j .NET Client over HTTP using REST and json

Magnus Mårtensson — Fri, 16 Apr 2010 09:51:09 GMT

Here it is; a Proof of Concept of the world’s first Neo4j .NET Client. In other words: Here follows a discussion on how to create a client library for communicating with a graph database over REST.

UPDATE: There is now a live CodePlex project for the realization of this concept; A .NET Client Library for Neo4j over HTTP using REST and JSON; http://neo4jrestsharp.codeplex.com/

Intro

Neo Technologies have come out with a Neo4j REST API for their popular Neo4j graph database. Since I am employed at Jayway and a couple of colleagues working with Neo are too a little bird by the name of Peter Neubauer tweeted in my ear that there was a free T-Shirt involved in making a working sample for .NET that communicated with the Neo4j server. Not only was that a handsome offer – I wanted to check out what the Architecture of such a client library for a graph database would look like.

I’ve recently been involved in client library abstractions and good architecture with the RESTful interface to Windows Azure Storage. (More on Azure and storage on my blog here, here, here, here and here.) Enough with the side tracking (sorry)!

Said and done; We set out to code up a working sample for .NET. I hacked the client code. The Neo-guys cheered me on and offered very helpful assistance when I ran into trouble or was just plane stumped on how to proceed with the API. A very nice team effort, including a remote team debugging session, later and we have a small and running POC

Tweeting on this effort it is notable how a bunch of people answer back with an interest in this idea for potential applications in the .NET world. I’ve been approached both on twitter (@noopman) and on LinkedIn with questions on this.

This blog post is to submit our findings to the world and hope that someone will be inspired to get going with a real .NET client library for Neo4j.

Architecture

First let’s look at the architecture for creating a client library for a RESTful service. This is a more general discussion that leads to our client library POC below. If your main reason for reading is the actual sample; scroll down.

When doing a client library abstractions and domain thinking comes into play. What we want is for the forgotten users (developers) of your application to work with their normal domain objects in the business code and for them to use an API that makes sense to their domain while talking to a highly specific graph database backend store. This means you have to keep all the RESTfulness, HTTP and json away from the domain code. Let me show you a sketch and see if I can make this explanation a bit less wordy:

At the other side of the Cloud from our .NET code is a Java based graph database. It is friendly enough to expose a RESTful API so that we may speak to it.

Doing so we want to create a client library that is capable of a couple of things:1) Taking care of serializing objects to json (and deserializing them back again). 2) Knowing about HTTP requests and how to handle them. In our sample we employed the power of the HttpWebRequest class to handle things like HTTP Headers, Content Types, URIs etc. 3) Finally this library must understand to expose an interface to your code that you can use from a specialized Data Access Component.

To protect the business layer from knowing anything about the specifics of the Neo4j API you really should use a specialized Data Access Component. There is now law being broken if you don’t do that. However a good rule of thumb is to add abstraction layers in your system to shield your domain code in your business components from any specifics of an underlying storage structure. The details of serialization of objects to json and making requests over HTTP are not a natural fit to the business problem you are solving. The code that does this separation is code you write yourself in your application. Only you know how to adapt a foreign library and technology to your domain. The sole purpose of this code, in other words, is to abstract away from your business logic the underlying technologies used to persist and query data used in the application. This component translates between the two worlds of your business logic and the techniques used to store data in a graph database. Most of the specific technical details of storage are wrapped up in the Client Lib mind you but the Data Access Layer DAL knows about this problem domain. It understands that the persistence indeed is a REST based service over HTTP with all that this entails. This component in your code can handle synchonisity and take care of specific errors or faults that may occur when using the storage. All of this is done in a language that is fairly close to the API that Neo4j exposes. In the end this component translates into terms useful to the domain in which your business operates. Instead of saying .PostNode() in your code perhaps you instead expose a .StorePerson() action.

What we achieve is a good and sound (and classic) layering model: Business Logic – DAL – Client Lib.

There is one more observation to make in the picture and that is the matter of IoC - Inversion of Control. In order to keep your code testable, maintainable and untangled from the implementations at lower layers you should always employ the method of Inversion of Control – making your business code for instance not reference the DAL direct but instead use a common contract to do so. The Depencency Inversion Principle states briefly that a higher layer should never depend direct on a lower layer but instead depend on an abstraction. If this is true you are able to test your business logic without using the actual underlying storage. This is for instance key in a TDD – Test Driven Development development style.

Code that depends on abstractions can be tested very quickly and does not depend on specific data in a data store to be correct in order to test the intended behavior. The Business Logic uses a contract for the DAL and the DAL implements it. Then you use something called an IoC container (read on at Wikipedia in the link above) to create the DAL instances and hand them to the Business Logic employing the technique of DI -Dependency Injection. This separation enables you to modify, re-deploy or even exchange DAL components without having to touch your Business Layer at all. If separating the Business Logic from the DAL using contracts is important it is almost event more important to do so in the lower layer that in run time speaks direct to the data store! You never want to end up in the situation where in order to test your DAL code you have to have access to a specific data store with specific data in it. Not only is that an inconvenience; it also makes executing your tests run as slow as transferring data over the Internet compared to as fast as your CPU can execute code. From experience it is an obvious conclusion that tests that are slow to run are not in fact run. The conclusion from all of these IoC discussions can only be that it is imperative that your Client Library comes with an abstraction layer.

What did we do with the Architecture in our POC?

All of these architectural considerations are of course premature in a POC. This is why in our sample we did not really separate the DAL code from the Client Lib. However to prove the point of IoC and DI I did however separate out the Business Logic from the Data Access logic using a simple contract and an equally simple Factory Method Pattern. Instead of new:ing up a DAL component in my code I called a factory method to access an abstraction of the DAL.

Our POC sample

Our code sample creates a small node graph with a few relationships. Then it makes a traversal of the graph to find all who “knows” someone according to the relationships. Finally it deletes the graph again.

Here is the graph we create in our sample: http://wiki.neo4j.org/content/The_Matrix We only create names on nodes and types on relationships in our simple sample.

And here is also the code so that you may download and follow along with our explanation your self:

Neo4j.POC.zip

First, in accordance with architectural considerations (above) we get hold of our DAL using a factory method. We also create a new Person object and store him (in this case Thomas Anderson – Neo):

INeo4jStorageClient neo4JStorageClient = 
Neo4jStorageFactory.GetNodeStorageClient(neo4jServerUri);

neo = new Person("Thomas Anderson");
neo4JStorageClient.StoreNode(neo);


As you can see in our business code this is just standard .NET classes being instantiated and a “Neo” Person is stored in a very normal way. In the business logic code there is no real details on how this storage happens. There is of course a “node” concept and you can consider for your self if that should be hidden away from business code too.
Since this is a POC the implementation of .StoreNode(neo) is very very trivial. If this were a real client library we would have included things like a library to transform (serialize) the neo-Person node into a json string. Now we just do it statically:
public void StoreNode(Person person)
{
    var name = person.Name;

    string jsonString = @"{""Name"":""" + name + @"""}";

    var nodeProtocol = new NodeProtocol(uri);

    var nodeId = nodeProtocol.PostNode(jsonString);
    person.Id = nodeId;
}

The interesting part here is the existence of a NodeProtocol implementation. I’ve found it useful working with a remote API like Windows Azure Storage and also with this Neo4j REST API to implement methods in C# code that correlate very closely to the API methods available. In our case we have a PostNode operation on the REST API and consequently we have a .PostNode() method in the C# implementation of that HTTP method call.
public int PostNode(string nodeData)
{
    var uri = HttpHelper.ConcatUri(baseUri, "node");

    HttpWebRequest request =
        Request.CreateWebRequest(HttpWebRequestMethod.POST, uri);

    HttpHelper.AddBody(request, nodeData);

    var response = requestMaker.MakeRequest(request);

    return HttpHelper.FindNodeId(response.Item3);
}

I won’t bore you with the exact details of all of the helper classes. To dig in to those it is better if you just download the code and look at the details your self.
As you can see we create a HttpWebRequest, add some headers to it (in the CreateWebRequest method) specifying things like request.Accept = "application/json"; meaning we accept a json formatted response. Also we specify that the body we send out is formatted as json with request.ContentType = "application/json";. Finally we make the call and return the id of the node we just created.
Again this is a solution bordering on triviality but still it is to the point of making a REST request and receiving a response. Also there is no code to handle any errors present here and the request is done synchronously.
What does the actual HTTP REST and json request and response look like for this simple sample?
Request:
POST http://try.neo4j.org:9999/node HTTP/1.1
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 26
Expect: 100-continue
Connection: Keep-Alive

{"Name":"Thomas Anderson"}

Response (note that the added line breaks in the response body were added by me):
HTTP/1.1 201 Created
server: grizzly/1.8.1
Content-Encoding: UTF-8
Location: http://try.neo4j.org:9999/node/1
Content-Type: application/json
Content-Length: 860
Date: Fri, 16 Apr 2010 08:36:45 GMT

{"incoming typed relationships":
http://try.neo4j.org:9999/node/1/relationships/in/{-list|&|types},
"incoming relationships":http://try.neo4j.org:9999/node/1/relationships/in,
"all relationships":http://try.neo4j.org:9999/node/1/relationships/all,
"create relationship":http://try.neo4j.org:9999/node/1/relationships,
"data":{"Name":"Thomas Anderson"},
"traverse":http://try.neo4j.org:9999/node/1/traverse/{returnType},
"property":http://try.neo4j.org:9999/node/1/properties/{key},
"self":http://try.neo4j.org:9999/node/1,
"properties":http://try.neo4j.org:9999/node/1/properties,
"all typed relationships":
http://try.neo4j.org:9999/node/1/relationships/all/{-list|&|types},
"outgoing typed relationships":
http://try.neo4j.org:9999/node/1/relationships/out/{-list|&|types},
"outgoing relationships":http://try.neo4j.org:9999/node/1/relationships/out}

As you can see the Neo4j REST API is self explanatory. Creating a node gives you back a response with all the things you can now do with this new node over the API. Since this quite a lot of data when the only thing that really matters is in the header (Location: http://try.neo4j.org:9999/node/1) I’ve already given feedback to the Neo team that I’d like a simple Uri switch (like APIDocumentation=0) to shut off the detailed response. But in fact I do like that you get so many Uris back that you may use direct. Depending on your needs this is either good or to verbose a payload to get back in your response. As you can see the node we created got the Id set to 1.
Now we proceed to create all the other nodes and in a very similar fashion the relationships between nodes. Here is the relationship that says that “Neo knows Trinity” (who has a node id of 2). This is a POST operation to /node/1/relationships:
POST http://try.neo4j.org:9999/node/1/relationships HTTP/1.1
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 56
Expect: 100-continue

{"type":"Knows","to":http://try.neo4j.org:9999/node/2}
Once the node graph for the Matrix is created we proceed to make a traversal of the graph. We want to find anyone who knows someone throughout the graph.
The request looks like this in REST (after a similar path through the code as above with a call from business logic to DAL and through to the client lib that makes the actual call using REST):
POST http://try.neo4j.org:9999/node/1/traverse/node HTTP/1.1
Accept: application/json
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 57
Expect: 100-continue
Connection: Keep-Alive

{ "relationships": { "type": "Knows" }, "max depth": 10 }

Again we simplified to a “max depth” of 10 relationships deep into the graph which works fine for our sample.
The result is a list of all the nodes that match. Because of the self explanatory API mentioned above the result also gives you actions to perform on each node that comes back in the response thus making the response body quite long. Rather than pasting the whole thing here into the blog post you can try it out your self (by downloading the application) and you will get a result exactly like this from the app:
 
Neo knows Trinity and Morpheus, who knows Cypher, who in turn knows Agent Smith.
Difficulties
We had only two real challenges in coding this POC.
The first one was an issue with streams and the optional so called Byte Order Mark (BOM) that you can put at the head of a stream. I’ve posted on this here: Would you like a Byte Order Mark to go with that?.
The other challenge was that I wrote a small app for Windows to show our results. But the guys on the Neo team are certainly not Windows users. The wanted to show the code on Mono. Just to be sure of compatibility I scaled back to a good old Windows Forms demo app in .NET Framework 3.0. This worked great on Mono which was our intent. But the drawback was that the library I’d found to handle object serialization didn’t respond well to that old version of the framework. That’s why I replaced my lib with static string creations (as you can see in the implementation of the .StoreNode() method above).
What have we learned from this?
First of all creating a client library for .NET to speak REST and json over HTTP to Neo4j is very doable; quite easy and straight forward.
Second (and that’s just a word of warning); keep your layers separated. This goes for all applications I guess but if you start to put code to create instances of web requests with web methods and content types and such in your business code you will end up in a very tangled and messy situation very fast which will be a pain and a high cost to maintain.
Finally we have considered looking into using WCF data services (and OData – the Open Data Protocol) and also possibly Linq – Language Integrated Query on the client side to make implementations in the library even more powerful and useful. And we do need a object to json serializer library.
Oh and here is another post on using the Neo4j REST API by the team itself: http://blog.neo4j.org/2010/04/neo4j-rest-server-part1-get-it-going.html
That’s it!
We hope this is inspirational enough for you to get going with your own samples or even client libraries in .NET to speak to Neo4j. If you do; please get back in touch with us!
Cheers,
M.
Technorati Tags: .NET,Graph Databases,C#,REST,json

Would you like a Byte Order Mark to go with that?

Magnus Mårtensson — Wed, 14 Apr 2010 06:46:20 GMT

It is possible to encode a little bit of metadata at the beginning of your byte streams to let the stream itself carry information on how it has been encoded. This is known as a Byte Order Mark (BOM) and it is as far as we know completely optional. Some .NET Framework implementations add this BOM to the start of streams given that they are to have a specific encoding. Here is how to find the BOM, and if necessary, remove it from your stream.

Context

Recently Neo4j graph database announced a REST interface. A couple of colleagues at Jayway are working on this. This was (very much) a too intriguing temptation to resist. I naturally had to write a small POC .NET based REST client to be able to talk to that server. (Blog post on this is pending within a few days.) While trying to send REST data in HTTP requests to the server we came across something interesting that caused compatibility issues between the Java and .NET worlds. The culprit was that optional byte order mark that I unintentionally embedded in my streams of bytes. The server did not recognize them. In a joint effort from both sides of the world (.NET client and Java server) we were able to find and identify the problem. Not that this is a new problem - It is very well documented in Wikipedia (link above). However I feel that one more post on this topic with a good test to show what’s going on is warranted.

A very good tool was also used to nail down the problem; Fiddler2. This little baby of a program is an http proxy that sits on your machine and listens to your http traffic. Using the tool you can inspect exactly what you are sending over the wire in your http requests and also view the responses coming back. Sweet stuff indeed. This is a MUST tool for you if you develop anything that sends data over HTTP.

Using Fiddler2 we were able to observe that when sending a stream of bytes as a body for an http POST the byte length, when the request came from my code, was 108 bytes. However when I used the built in request builder in Fiddler2 to reproduce and resend the exact same request it was reported that the length was 105. Why was there a difference in three (3) bytes?

Problem was that these tree bytes, as I stated above, caused the server to throw a fit. It could not recognize the body of the message to be the expected json string in UTF8 encoding. Json, I’ve learned, is always in UTF8 encoding.

The java guys googled and I binged ;~) and we found that these three bytes is an optional “byte order mark” or BOM. It is not wrong to send these three bytes and it is not wrong not to send them. Guess what? .The .NET world tends to send these extra bytes and everyone else tends not to. Why not have a bit of more incompatibility, right? It’s not like we want to talk to each other anyway!

What’s up with the BOM? What is it’s purpose?

The BOM is a way to add a kind of metadata inside of a stream of bytes instead of sending some actual metadata along next to your stream. Option one is saying “Here is my byte stream and btw it is in UTF8 encoding”. The other way to do it is saying “Here is my byte stream and if you look at the first three bytes you can read the encoding of it”. Which is better? I can’t say I care all that much other than the fact that it caused us a problem when I tried to send requests from .NET code to the Neo4j server.

How can you then – finally – handle these three bytes?

Well that depends… on what you intend to do. What I can show you is a piece of code that tells you exactly what this is and then you can copy that behavior into your code and modify it to serve your purpose. The code below should be pretty self explanatory but just to be sure. I take the string "foo" and encode it into a MemoryStream using a StreamWriter. The issue is that I tell the stream writer to use Encoding.UTF8 and this is where .NET Framework adds the BOM. The resulting stream is not 3 bytes as perhaps expected. Instead it is 6 bytes long. What you have to do if you read this stream ‘raw’ – by hand – in some library, is skip over the first three bytes. A better way to do it is to use a reader that handles the BOM. Finally if you don’t want to add the BOM in the first place you can write your bytes yourself in a more ‘raw’ fashion byte by byte to the stream. As you can see .NET Framework is good enough to have a way to find an actual BOM for different encodings. The UTF8 encoding does it this way: Encoding.UTF8.GetPreamble() (msdn library link to .GetPremable()).

[TestMethod]
public void StreamWriter_with_encoding_writes_ByteOrderMark()
{
    string str = "foo";

    var memStream = new MemoryStream();

    // Test
    var writer = new StreamWriter(memStream, Encoding.UTF8);
    writer.Write(str);
    writer.Flush();

    Assert.AreEqual(6, memStream.Length);

    var buffer = new byte[3];

    memStream.Position = 0;
    memStream.Read(buffer, 0, 3);
    string result = Encoding.UTF8.GetString(buffer);

    var preamble = Encoding.UTF8.GetPreamble();
    var byteOrderMarkUtf8 = Encoding.UTF8.GetString(preamble);

    Assert.AreEqual(byteOrderMarkUtf8, result);

    memStream.Position = 3;
    memStream.Read(buffer, 0, 3);
    result = Encoding.UTF8.GetString(buffer);

    Assert.AreEqual(str, result);

    // Alternative: Write the bytes yourself to skip the BOM

    memStream = new MemoryStream();

    foreach (var @byte in Encoding.UTF8.GetBytes(str))
    {
        memStream.WriteByte(@byte);
    }

    Assert.AreEqual(3, memStream.Length);
}

You can also if you like compare bytes for the premable one by one rather than converting to a string comparison.

As you can see this BOM can be handled easily if you like. The thing is I did not have a clue that it was there.

Oh – and btw now the Neo4j server accepts REST json bodys that both can have and skip the BOM! Good ‘bug’ to solve or ‘feature’ to have.

Cheers,

Technorati Tags: .NET,Arcitecture,json,Java,REST

Effective and Infinite storage in the cloud session at TechDays 2010 Sweden

Magnus Mårtensson — Tue, 06 Apr 2010 20:31:07 GMT

The main points of our Effective and Infinite Storage in the Cloud session at TechDays 2010 in Sweden are listed below as well as links to all of our related material and the zip with our demos. The slides are here too but they are in Swedish only.

Our session at TechDays 2010 gave an overview of what storage in the Cloud is, according to Microsoft at present date, and some of the goals for the future in this area. We intended our session to provide the basis for companies to be able to evaluate if and how moving storage to the Cloud can enhance their business for their own specific scenarios.

Speaking about Cloud storage and Microsoft today means you need to give a session divided into two parts; SQL Azure and Windows Azure storage. And so our presentation had two distinct parts.

As consultants working with different clients we see that companies in general are not yet ready to adopt Cloud storage on a broad scale. However we also see a continuous increase in interest in this area and we believe that this is directly related to the richness of the offerings and services now becoming available. The story matures in the Microsoft camp and the pace is rapid. Cloud storage from Microsoft is admittedly still very young. (Youngest in the business?) But now Microsoft has thrown the full weight of the company behind making the cloud story stick. Expect to see a lot of new offerings, services, products, frameworks etc. in the Cloud coming out from Microsoft in the near future.

There are a few common concerns when it comes to storing data in the cloud. A quick (and perhaps not complete) listing follows:

Storage cost in the Cloud

One big concern we hear from clients and in the buzz at conferences that take place around now is a concern related to the cost of storage and data transfer in the Cloud. This is naturally an equally important issue regardless in what Cloud you choose to store your data.

The thing about storing your data in the Cloud is that the cost for this storage is a lot more self evident compared to storing your data in your own servers on premise. This is because you receive a bill each month stating exactly what you have spent for storage in the passed period. What we tend to mitigate is that storage, backups, crash plans and managing all of this on premise, or even hosted, does not exactly come for free either. The big question then becomes what difference in cost there will be for your company for storing and transferring data to a Cloud compared to local hardware operated by your local IT department.

Another aspect of the cost for storage in general is the question of how you build your apps. We believe that many have been a bit oblivious to this issue when they have run their apps and stored our data only on premise or with traditional hosting. Perhaps we have been cost inefficient in this area until now? In the age of Soft Deletes - deleting data does not remove it but rather marks it as deleted – no data is ever actually deleted - managing costs for storing data will become a more and more important question for everyone. Building Cloud applications and storing data in the Cloud has to be done with data storing and management costs in mind. As we hinted to above; the same should have been true for all of your applications up until now. Storing data in the Cloud just makes this point more obvious.

Storage Security when your data in the Cloud

Perhaps the fist fear when it comes to storing data in the Cloud is the fear of security breeches and the perceived loss of control of this security. Your data is no longer at home.

What you need to ask yourself, and here we choose to quote a good colleague Sergio Molero @sergio_molero, is; “What sense of security do you have today? What security is it that you think you will loose when storing your data in the Cloud?”

The same exact thing can be said for hosted data as compared to data stored in the Cloud because the two concepts are very much the same thing in this regard. Data in the Cloud is also hosted data!

One important thing to also keep in mind is that moving to a data storage solution in the Cloud does not mean all of your data must move “out there” somewhere. It is still quite possible to store the most sensitive data on premise and use Cloud storage for more public data only. We believe that the higher focus on security for data in the Cloud will put higher focus on your security efforts compared to your present security on premise. The move to the Cloud could in conclusion prove to heighten the overall security level for your data.

Also look into Windows Identity Foundation (WIF) for insights on how to secure your data in the Cloud.

Trusting someone else with your data in the Cloud

If security from external attacks to your data in the Cloud is a concern you feel somewhat confident you will be able to handle then perhaps trusting someone else with storing your company sensitive data is a concern that causes you to think twice? Trust is a complex issue not to be taken lightly. We will only raise the concern here and not delve too deep into it.

Can you trust a hosted data storage provider today? If you can, then this is no difference compared to trusting data stored in the Cloud.

If you can’t trust anyone else with your data, then the Cloud is not an option for you. Clean and simple.

Legal ramifications for storing data in the Cloud

It feels important to also comment briefly on the potential legal ramifications for storing your data in a Cloud in another country somewhere. That being said we’d just like to point out that we are no lawyers. Let’s be clear on that.

Some data cannot legally be moved outside the borders of your own country. Hospital patient information is a good example of this. Other data must for legal reasons stay within your geo-region, say for instance the EU.

As we hinted to above; not all of your data has this limiting concern. You do have more public data as well as more private data in your business. Only the latter is a potential legal problem. There will be more hybrid solutions for data Storage in the future. Some data for a company will be stored in the Clouds and other data will be stored on premise.

But how can we store data both on premise and in the Cloud without having two separate data storage solutions? One thing that will assist in this endeavor is the rumored upcoming availability of local on-premise, or hosted, Clouds that will be possible to install locally and behave in the same way the big Clouds out there on the Internet do. Right now there does not exist many offerings of this nature. There will in time when companies start to demand them. In fact, given that the storage Clouds today has a REST interface, it is not very difficult to implement that same interface on your own on premise machines! Doing this however will require a good effort on your part and you will end up with a proprietary solution that you will have to evolve over time as the real Clouds shift. Another option is to wait a bit and let the Cloud vendors come up with packaged solutions that you may install simply install in your own data centers. The hybrid solutions that will result from this will play an important role in shaping the solutions for data storage for the coming decade.

SQL Azure

In our SQL Azure part of our talk we pointed out three pros and three cons for attendees to take away from our session.

Because Windows Azure takes care of all of the maintenance normally associated with a SQL Server server developers are now allowed to focus on the database rather than performing administrative tasks.
The database is always available in the sense that upgrades are seamless as well as failover, backup and crash plan which means that the SQL Azure database is always available.
The scaling story is very strong in SQL Azure. If you can shard your database into many pieces you can scale out from one database to many (say 100) databases to meet a peek load and then back down again to just one at will. For now there is no built in support for this in SQL Azure but this will likely be a story further down the road and you can build it your self today.

There are some things to consider when looking into deploying to SQL Azure.

Security is locked down compared to SQL Server. The sys-tables are not yet available for instance. This is because your databases are co-located on the same servers as data belonging to some one else.
Your data is no longer at home in the sense that it is moved outside your company and even outside your country.
The size right now for one database in SQL Azure is 10 GB but this will increase to 50 GB in June.

Windows Azure Storage

With Windows Azure Storage Microsoft addresses a set of data storage stories that do not really fit well into a standard relational database structure.

There is a powerful and “infinite” message queuing system that enables messaging between the different machines in your applications. This queuing system does not guarantee 100% ordering of the messages in the queue. This is in case of more than one machine working on the same queue. However the queues require dequeue (GET) and delete (DELETE) in order to complete an actual removal of a message. This method will ensure each message will actually be handled in full in case of some error when decueing and handling the message the first time.
There is also a concept in Windows Azure of that which is known as a “Big Table”. Say you want to store 200 million user data sets for Facebook or why not all the messages ever sent to Twitter. For this you need an “infinite” tabular structure that has no end that you will ever hit.
The third thing you can store in Windwos Azure are two kinds of blobs. 1) A so called Block Blob. A blob that is handled in blocks when uploaded. A block Blob is not really intended to be changed once uploaded. You can exchange blocks in the blob but that is not the intent. This is typically a video file or a picture or something that is rarely if ever changed. You would use block blobs to build the next youtube where you can stream files down into a player. 2) A Page Blob. This kind of blob is divided into pages rather than blocks. You can real or write pages at will into this kind of blob. This means that you are able to create a Virtual HardDrive (VHD) and upload it as a page blob to Windows Azure. Then you can actually mount this blob to one or more virtual machines in Windows Azure as an extra hard drive. This comes in handy when moving an application to Windows Azure that uses a lot of disk. All of your old File.Open will continue to work.

When considering Windows Azure for your storage think about:

You should use only the parts of Windows Azure that you really need. If you need to store backup blob type of data – use only that. There is a good advantage to consider when you have applications to run that will use Windows Azure Storage. If you deploy your application to Windows Azure as well as the data to the same datacenter then the internal data transfer costs between this application and storage is without cost. If you instead run your apps on premise and store data in the cloud the transfer of data to and from the cloud carries a cost.
Regarding the cost you should really make a data storage estimate and analysis before you start up deploying. Your application should have a plan for the amount of data stored, transferred and the duration of storage. This is true on premise too, as discussed above, and data storage plans and budgets will become a more and more important part of any application for the future.
Windows Azure Storage means learning to use a new API. Perhaps not a dramatic thing but the three storage options also affects the way you build the rest of your application. If you disagree chances are you need to learn more about storing data in the cloud.

Links

Windows Azure SDK

Windows Azure Tools for Visual Studio

Cloud Storage Studio – a SSMS look alike for Windows Azure Storage

Fiddler2 – Web debugging proxy

CloudStorage.API @ CodePlex

Azure Contrib @ CodePlex

SQL Azure Migration Wizard

Downloads

Ziped version of the demo code in the Windows Azure Queue Storage sample:

AzureStorage.zip

Slides from the presentation

EffektivOchOandligLagringIMolnet.ppsx

EffektivOchOandligLagringIMolnet.pdf

Slides for the demo attached as a zip above

DemoWindowsAzureCloudQueue.ppsx

DemoWindowsAzureCloudQueue.pdf

Final words

We would like to thank Microsoft for arranging TechDays, we had a blast! Thanks to the audience for listening.

Cheers,

Peter von Lochow @vonlochow and Magnus Mårtensson @noopman

Technorati Tags: Windows Azure,SQL Azure,Cloud,Microsoft

Microsoft Enterprise Library 5.0 and Microsoft Unity 2.0 Beta 1 released

Magnus Mårtensson — Wed, 10 Feb 2010 18:14:28 GMT

There are a whole bunch of changes in the just released Microsoft Enterprise Library 5.0 Beta 1 and in Microsoft Unity 2.0 Beta 1. I have been using these libraries for years and have greatly been looking forward to this new release.

The list of changes for Enterprise Library is very long indeed and going into specific changes here on this page seems redundant when you can go check it out for your self.

Regarding Unity it seems it is not required any more to reference the ObjectBuilder assembly when using Unity; any functionality previously required from this assembly are now included in the Unity assembly. Nice.

Cheers,

Magnus

Technorati Tags: Microsoft,Patterns & Practices,Arcitecture