Bing! - My Claim to Fame

Judging by the date of my last post, it turns out that the “it’s the thought that counts” - is all lies!   Despite my best intentions, an entire year has disappeared into the ether.

While away from my blogosphere I have continued my programming and my math/statistics degree, but I have changed jobs (10 months ago) – which brings me to this post.

My ‘new’ employer is “Bing Technologies” and a fun trivia fact is that Microsoft bought the www.bing.com domain from us.

Cool huh!

Structure Inlining with .Net 3.5 SP1

I just stumbled on a fascinating performance improvement coming up in .Net 3.5 SP1.

When working with value types (structs) the JIT currently hits the stack each time you access a field - which is quite inefficient.

In SP1 they've added an algorithm called "value type scalar replacement"; which intuitively does exactly what it sounds like!

Take the following code:

static int MyMethod1(int v)
{
    Point point;  // gets a stack location.
 
    // All 3 field accesses involve stack operations.
    point.x = v; 
    point.y = v * 2;
 
    return point.x;
}

The JIT will replace the value type (Point is a struct) with scalars:

static int MyMethod2(int v)
{
    int x; int y;  // Was “struct point;”. Now replaced by x and y.
    x = v; 
    y = v * 2;
    
    return x;
}

After the JIT inlines the structure, it can optimise the routine by placing the integers on the register rather than the stack.

What's the difference?

We are roughly looking at a 10x improvement! (Using a quick and dodgy test app)

This is fantastic, especially when dealing with lots of structs like points and rectangles etc.

What's the catch?

This isn't magically going to work on all your value types; there are strict guidelines when this replacement can occur:

- The value type must have no more than 4 fields

- These fields must be value types, or object references

- You must use [StructLayout(LayoutKind.Sequential)], which is the default anyhow.


There some other catches, but if you are creating complicated structures, you really should think about making them a class.

More Improvements

This is just a snippet of the algorithm, whereas the original article explains it in much more detail.

BTW: This is in the CLR, so all .Net languages benefit! And of course this includes F# and FParsec.

Cheers to SP1!

Advanced Domain Model queries using Linq - Part IV

This should be the last part in my series of posts.  In this part I'll cover support for parameters via method calls.

Parameter Support (Contrived Example)

Continuing the Library example, how would you query the books that will be overdue next week?

Well you could write the following:

var nextWeek = from a in repository.AsQueryable()
               where a.CheckOverdue(DateTime.Today.AddDays(7))
               select a;

How it works

The CheckOverdue Property simply returns a Lambda Expression as it's result.

class CheckOverdueProperty : QueryProperty<Loan, Func<DateTime, bool>>
{
    public CheckOverdueProperty()
        : base(x => y => (x.DateReturned == null && (y - x.DateBorrowed).TotalDays > x.LoanPeriod))
    { }
}

I've modified the expression parsing routines to support parameters (including multiple) and even nested lambdas!

Nested Lambdas?

I can't really see the need for it, but hey... why not! Curry anyone? :)

class MyProperty : QueryProperty<Loan, Func<int, Func<int, Func<int, double>>>>
{
    public MyProperty()
        : base(a => x => y => z => x + y + z + a.LoanPeriod)
    { }
}

So how would you call such a monstrosity?

var test = from a in repository.AsQueryable()
           select a.MyProperty(5)(4)(3);

But check out the SQL! Great work Linq to Sql guys!

SELECT (CONVERT(Float,5 + 4 + 3)) + [t0].[LoanPeriod] AS [value]
FROM [dbo].[Loan] AS [t0]

Declaration Syntax

If you're particularly astute, you'll have noticed that my syntax for declaring a QueryProperty has changed slightly since the last post.

Here's the class declaration of the CheckOverdue Property:

[QueryProperty(typeof(CheckOverdueProperty))]
public bool CheckOverdue(DateTime date)
{
    return CheckOverdueProperty.GetValue<CheckOverdueProperty>(this)(date);
}

Although this is a method, you can still define it as a property (notice that I return a lambda expression).

[QueryProperty(typeof(CheckOverdueProperty))]
public Func<DateTime, bool> CheckOverdue
{
    get { return CheckOverdueProperty.GetValue<CheckOverdueProperty>(this); }
}

So What's Changed?

Well, QueryProperties no longer derive from an Attribute class.  Although it worked, I felt dirty about it.

I borrowed from Fredrik's syntax and pass the property type to the Attribute - much cleaner.

Since I'm no longer defining the QueryProperty as static, I had to change the way it is referenced by the class property.

The code above shows my attempt at making this easy, yet efficient - but it's up to you how to implement it.  E.g. this would work well too:

[QueryProperty(typeof(CheckOverdueProperty))]
public Func<DateTime, bool> CheckOverdue
{
    get { return CheckOverdueProperty.Value(this); }
}
 
private static CheckOverdueProperty CheckOverdueProperty = new CheckOverdueProperty();

If you can think of better ways I'd love to hear it! 

Expression Parser

Since I posted the parser code in my last post, here it is again - and my has it grown!

class QueryPropertyEvaluator : ExpressionVisitor
{
    // Basic member access support for QueryProperties
    protected override Expression VisitMemberAccess(MemberExpression m)
    {
        if (m.Expression != null)
        {
            var property = GetProperty(m.Member);
 
            if (property != null)
                return Expression.Invoke(property.GetExpression(), m.Expression);
        }
 
        return base.VisitMemberAccess(m);
    }
 
    // Support method call abstractions over QueryProperties
    // Assumes method call has same parameters (in order!) as QueryProperty
    // Also supports static methods, as long as first parameter is the class object
    protected override Expression VisitMethodCall(MethodCallExpression m)
    {
        Expression obj = this.Visit(m.Object);
        IEnumerable<Expression> args = this.VisitExpressionList(m.Arguments);
 
        var property = GetProperty(m.Method);
 
        if (property != null)
        {
            if (obj != null)
            {
                // Method on QueryProperty class
                return VisitLambdaInvocation(Expression.Invoke(property.GetExpression(), obj), args);
            }
            else
            {
                // Assumes static method where first parameter is object instance
                return VisitLambdaInvocation(Expression.Invoke(property.GetExpression(), args.First()), args.Skip(1));
            }
        } 
 
        if (obj != m.Object || args != m.Arguments)
            return Expression.Call(obj, m.Method, args);
 
        return m;
    }
 
    // Support lambda expressions within QueryProperties
    protected override Expression VisitInvocation(InvocationExpression iv)
    {
        IEnumerable<Expression> args = this.VisitExpressionList(iv.Arguments);
        Expression expr = this.Visit(iv.Expression);
 
        // Rewrite InvokeExpression - Folds out inner lambda
        if (expr.NodeType == ExpressionType.Invoke)
            return VisitLambdaInvocation((InvocationExpression)expr, args);
 
        if (args != iv.Arguments || expr != iv.Expression)
            return Expression.Invoke(expr, args);
 
        return iv;
    }
 
 
    // Recursively 'flattens' the lambda invocation expressions
    // Rewrites the nested lambda expression tree to work with Linq to SQL
    protected Expression VisitLambdaInvocation(Expression expression, IEnumerable<Expression> args)
    {
        if (expression.NodeType != ExpressionType.Invoke)
            return Expression.Invoke(expression, args);
 
        var invoke = (InvocationExpression)expression;
        var lambda = (LambdaExpression)invoke.Expression;
 
        // Func<TClass, Func<Arg0, Arg1..., Result>> -> Func<TClass, Result>
        var arguments = lambda.Type.GetGenericArguments().ToArray();
        arguments[arguments.Length - 1] = arguments.Last().GetGenericArguments().Last();
 
        return Expression.Invoke(
                    Expression.Lambda(
                        lambda.Type.GetGenericTypeDefinition().MakeGenericType(arguments),
                        VisitLambdaInvocation(lambda.Body, args),
                        lambda.Parameters),
                    invoke.Arguments);
    }
 
    #region MemberInfo caching
 
    private Dictionary<MemberInfo, QueryProperty> lookup = new Dictionary<MemberInfo, QueryProperty>();
 
    // This is not strictly needed, but adds a performance improvement.
    // Feel free to remove this code if you think memory is more important
    protected QueryProperty GetProperty(MemberInfo info)
    {
        QueryProperty result = null;
 
        if (lookup.TryGetValue(info, out result) == false)
        {
            var attribute = Attribute.GetCustomAttribute(info, typeof(QueryPropertyAttribute)) as QueryPropertyAttribute;
 
            if (attribute != null)
                lookup.Add(info, result = attribute.GetProperty());
        }
 
        return result;
    }
 
    #endregion
}

Conclusion

As usual, here is the sample code for you to experiment with yourself.

I hope you've enjoyed this series, I certainly have. 


If I get the chance I might write a post explaining the expression parsing in detail, or the compiled queries - but till then...

Cheers!

Advanced Domain Model queries using Linq - Part III

I didn't intend to post so quickly after my last post, but some comments from Nick motivated me to try something different.

So what's changed?

Not much.  Defining your query properties has changed (and obviously some implementation details).  All the linq syntax stuff stays the same.

Here's the same Loan class, rewritten:

class Loan
{
    [OverdueFee]
    public decimal OverdueFee
    {
        get { return OverdueFeeAttribute.Property.GetValue(this); }
    }
 
    [IsOverdue]
    public bool IsOverdue
    {
        get { return IsOverdueAttribute.Property.GetValue(this); }
    }
 
    [DaysOverdue]
    public double DaysOverdue
    {
        get { return DaysOverdueAttribute.Property.GetValue(this); }
    }
}

And here's our query property definition:

class IsOverdueAttribute : QueryPropertyAttribute
{
    public IsOverdueAttribute() : base(Property) { }
 
    public static readonly QueryProperty<Loan, bool> Property = new QueryProperty<Loan,bool>(
        x => (x.DateReturned == null && (DateTime.Now - x.DateBorrowed).TotalDays > x.LoanPeriod));
}
 
class OverdueFeeAttribute : QueryPropertyAttribute
{
    public OverdueFeeAttribute() : base(Property) { }
 
    public static readonly QueryProperty<Loan, decimal> Property = new QueryProperty<Loan, decimal>(
        x => (decimal)(x.DaysOverdue * 0.45));
}
 
class DaysOverdueAttribute : QueryPropertyAttribute
{
    public DaysOverdueAttribute() : base(Property) { }
 
    public static readonly QueryProperty<Loan, double> Property = new QueryProperty<Loan, double>(
        x => x.IsOverdue ? (DateTime.Now - x.DateBorrowed).TotalDays - x.LoanPeriod : 0);
}

What's the difference?

Well, depending on your programming tastes, it's cleaner.  We've reduced some of the tight coupling, and we now have the ability to spread the logic (which is more suitable in certain situations).

And if you're wondering why I seem to have an obsession with static members, it's mainly due to performance - and truth be told, I'm pre-optimizing here.  Generally I'm not such a fan... honest!

Performance anxiety

Speaking of performance, using Attributes has a penalty.  According to my tests, it's roughly 30% slower to parse the expression.  

This might sound like a lot, but we're talking about 0.37 vs 0.28 milliseconds per query here - and once you consider the database retrieval time... it's really nothing.

I don't even parse the expression tree with linq to objects - so no impact at all!

Bonus code (Expression Parser)

Here's my expression parsing code, I know you want to see it:

class QueryPropertyEvaluator : ExpressionVisitor
{
    protected override Expression VisitMemberAccess(MemberExpression m)
    {
        if (m.Expression != null)
        {
            var attribute = Attribute.GetCustomAttribute(m.Member, typeof(QueryPropertyAttribute)) as QueryPropertyAttribute;
 
            if (attribute != null)
                return Expression.Invoke(attribute.Property.GetExpression(), m.Expression);
        }
 
        return base.VisitMemberAccess(m);
    }
}

Beautiful isn't it!

Feedback please

If you've been following this little series, I'd love to hear what you think of the new syntax.

I'm still hesitant, since this new syntax results in more code - but I think it's a cleaner approach.


As usual, you can find the sample app here - enjoy!

Advanced Domain Model queries using Linq - Part II

Since my previous post I've rewritten the internals to make use of LinqKit - a fantastic lightweight project.

See the changes

I took my rewrite as an opportunity to support the full query syntax:

var overdue = from a in repository.AsQueryable()
              where a.IsOverdue == true
              select a.OverdueFee;
 
var days = repository.AsQueryable()
                     .Where(x => x.IsOverdue)
                     .Select(x => x.DaysOverdue);

and added easy support for compiled queries (currently only supporting linq to objects and linq to sql):

var query = repository.Compile(
    source => from x in source
              where x.IsOverdue == true
              select x.OverdueFee);
 
var results = repository.Execute(query);

I've also optimized the expression parsing (speed difference is negligible to normal queries), and I tweaked the QueryProperty construction:

#region IsOverdue Property
 
public bool IsOverdue
{
    get { return IsOverdueProperty.GetValue(this); }
}
 
static public readonly QueryProperty<Loan, bool> IsOverdueProperty = new QueryProperty<Loan, bool>(
    x => x.IsOverdue, 
    x => x.DateReturned == null && (DateTime.Now - x.DateBorrowed).TotalDays > x.LoanPeriod);
 
#endregion

The big change here is that I've added an extra parameter; a reference back to the mapping property.

I'd really like a memberof(Loan.IsOverdue) keyword style syntax in C#, but since there's not, I use this expression to grab the MemberInfo - which is used in the expression parser.

It's a slight DRY drawback, but it's a faster way than using reflection (and avoids an ugly string lookup) - and it's much faster than using attributes.

Get the code

Most of the big changes are all implementation specific, so be sure to check out the sample app.

Please let me know what you think!

BTW: Joseph, I made some architectural changes to the LinqKit.ExpandableQuery stuff.  Externally it behaves exactly the same, but it now lets me reuse a lot more code.

It's light on the comments, but I'm hoping my next post will explain myself.


ed: Continue to Part III

Ctrl . - My favourite shortcut

I'm a huge keyboard shortcut fan (for a windows developer)... for common tasks it's just annoying to have to grab the mouse.

For me, one such task was the rename / resolve support in Visual Studio: 
Using the mouse to bring up this dialog has got to be the most frustrating task in Visual Studio.

So much so, I refused to use it; instead opting for the right-click 'resolve' approach, or even manually typing in namespaces.

A few months ago, I stumbled on "Ctrl ." and to my delight ended my coding frustration.

I'm now more productive, less frustrated... do yourself a favour and give "Ctrl ." a go!

Advanced Domain Model queries using Linq

I read a fantastic blog article from Nick Blumhardt yesterday which kicked off some crazy ideas in my head.

The Problem

With the following class, I can query on IsOverdue using linq... but not using linq to sql (or linq to nhibernate).  How frustrating!

class Loan
{
  // Member details omitted
  public bool IsOverdue
  {
    get 
    {
       return DateReturned == null && 
          (DateTime.Now - DateBorrowed).TotalDays > LoanPeriod;
    }
  }
}

This is a fundamental issue in relational to object mapping systems, but we can fix it using some linq trickery.

Disclaimer

I think my solution is very cool, but I'm not sure how acceptable it is for a production environment. 

Please leave feedback if you like it / makes you want to vomit.

The change

Let's modify the Loan class implementation of IsOverdue:

#region IsOverdue Property
 
public bool IsOverdue
{
  get { return IsOverdueProperty.GetValue(this); }
}
 
static public readonly QueryProperty<Loan, bool> IsOverdueProperty = QueryProperty.Register<Loan, bool>(
    x => x.DateReturned == null && (DateTime.Now - x.DateBorrowed).TotalDays > x.LoanPeriod);
 
#endregion

Here we move the logic for IsOverdue into a separate QueryProperty member that takes care of the nasty details.

Hopefully you'll agree that the code is still fairly readable, and encapsulated with all the logic still bundled inside the Loan class.

By moving this logic into a lambda expression, we can now write queries for both database and object collections! e.g

// Database Access
using (var db = new RepositoryDatabase<Loan>())
  var overdue = db.Where(y => y.IsOverdue && y.LoanPeriod == 5);
 
// Collection Access
using (var list = new RepositoryCollection<Loan>())
  var overdue = list.Where(y => y.IsOverdue && y.LoanPeriod == 5);

More Goodness

When coding these query properties it can be very handy to reference other query properties. e.g.

#region DaysOverdue Property
 
public double DaysOverdue
{
  get { return DaysOverdueProperty.GetValue(this); }
}
 
static public readonly QueryProperty<Loan, double> DaysOverdueProperty = QueryProperty.Register<Loan, double>(
  x => x.IsOverdue ? (DateTime.Now - x.DateBorrowed).TotalDays - x.LoanPeriod : 0);
 
#endregion
 
#region OverdueFee Property
 
public decimal OverdueFee
{
    get { return OverdueFeeProperty.GetValue(this); }
}
 
static public readonly QueryProperty<Loan, decimal> OverdueFeeProperty = QueryProperty.Register<Loan, decimal>(
  x => (decimal)(x.DaysOverdue * 0.45));
 
#endregion

This allows us to create very complicated queries in a very simple manner.

The Catch

Too good to be true?  You're right; there's a devil in the details. 

You might have picked it up already - how does linq interpret my query property correctly?

By modifying the expression tree!

My Repository class implements the Where() and translates the expression; replacing any QueryProperty references with a call to the actual lambda expression.

The really ugly part is that it's a string based lookup ie. "<name>Property".  Though I also check the type to be extra safe.

It's really not a bad trade off, considering it's very similar to how WPF DependencyProperty's work.

The Conclusion

This is a really easy and seamless way to implement custom queryable properties in your domain objects.

I've only touched on a fraction of the possibilities with this technique, but it shows the power and flexibility of linq.

Make sure you check out the sample application for the full code.


ed: Continue to Part II