Monday, July 7, 2008

Advanced Domain Model queries using Linq

I read a fantastic blog article from Nick Blumhardt yesterday which kicked off some crazy ideas in my head.

The Problem

With the following class, I can query on IsOverdue using linq... but not using linq to sql (or linq to nhibernate).  How frustrating!
class Loan
// Member details omitted
public bool IsOverdue
return DateReturned == null &&
(DateTime.Now - DateBorrowed).TotalDays > LoanPeriod;

This is a fundamental issue in relational to object mapping systems, but we can fix it using some linq trickery.


I think my solution is very cool, but I'm not sure how acceptable it is for a production environment. 

Please leave feedback if you like it / makes you want to vomit.

The change

Let's modify the Loan class implementation of IsOverdue:
#region IsOverdue Property

public bool IsOverdue
get { return IsOverdueProperty.GetValue(this); }

static public readonly QueryProperty<Loan, bool> IsOverdueProperty = QueryProperty.Register<Loan, bool>(
x => x.DateReturned == null && (DateTime.Now - x.DateBorrowed).TotalDays > x.LoanPeriod);


Here we move the logic for IsOverdue into a separate QueryProperty member that takes care of the nasty details.

Hopefully you'll agree that the code is still fairly readable, and encapsulated with all the logic still bundled inside the Loan class.

By moving this logic into a lambda expression, we can now write queries for both database and object collections! e.g
// Database Access
using (var db = new RepositoryDatabase<Loan>())
var overdue = db.Where(y => y.IsOverdue && y.LoanPeriod == 5);

// Collection Access
using (var list = new RepositoryCollection<Loan>())
var overdue = list.Where(y => y.IsOverdue && y.LoanPeriod == 5);

More Goodness

When coding these query properties it can be very handy to reference other query properties. e.g.
#region DaysOverdue Property

public double DaysOverdue
get { return DaysOverdueProperty.GetValue(this); }

static public readonly QueryProperty<Loan, double> DaysOverdueProperty = QueryProperty.Register<Loan, double>(
x => x.IsOverdue ? (DateTime.Now - x.DateBorrowed).TotalDays - x.LoanPeriod : 0);


#region OverdueFee Property

public decimal OverdueFee
get { return OverdueFeeProperty.GetValue(this); }

static public readonly QueryProperty<Loan, decimal> OverdueFeeProperty = QueryProperty.Register<Loan, decimal>(
x => (decimal)(x.DaysOverdue * 0.45));


This allows us to create very complicated queries in a very simple manner.

The Catch

Too good to be true?  You're right; there's a devil in the details. 

You might have picked it up already - how does linq interpret my query property correctly?

By modifying the expression tree!

My Repository class implements the Where() and translates the expression; replacing any QueryProperty references with a call to the actual lambda expression.

The really ugly part is that it's a string based lookup ie. "<name>Property".  Though I also check the type to be extra safe.

It's really not a bad trade off, considering it's very similar to how WPF DependencyProperty's work.

The Conclusion

This is a really easy and seamless way to implement custom queryable properties in your domain objects.

I've only touched on a fraction of the possibilities with this technique, but it shows the power and flexibility of linq.

Make sure you check out the sample application for the full code.

ed: Continue to Part II


Nick said...

Wow - another one bites the dust :)

Awesome idea.

Perhaps it is time to create a 'Linq data access extensions'-type project somewhere?

Luke Marshall said...

Cheers Nick!

It's quite fun to play around with those queries - I'm still amazed how well it works :)

Good idea with the project, or maybe it would be a good addition to an existing one?

Nick said...

I think putting it in something like Umbrella will discourage people because of the unneeded baggage they'll have to take on as well.

Perhaps there might be a place for these things in a C# functional programming library somewhere? The goals still diverge a bit (enterprise data access vs. extra language features.)

Luke Marshall said...

Fair call on that one.

If it generates any interest, it'd be pretty easy to bundle it into a project.

Side note: I'd like to make the QueryProperty declarations less verbose... but for now it's a good starting point :)

Mark said...

This looks really good. Soon there will be no excuse to use SQL to query data once we can write complex object queries that translate to the same SQL query.

Nick said...

Might be the project we're looking for? Maybe the author will be interested in expanding the project?

Luke Marshall said...

That LinqKit is an awesome find!

Not sure if it's an 'active' project though.

It's definitely in line with with what we're trying to achieve.

Nick said...

I'll make some contact :)

Joe said...

Hi there

This is absolutely brilliant! It solves a problem elegantly and is actually quite simple. I've done a quick test and it integrates perfectly with LINQKit, via the AsExpandable infrastructure.

In terms of real-world readiness, one thing that I think we need to check is the performance overhead of using reflection to test for the "XXXProperty" field on every VisitMemberAccess call. (I've racked my brain on alternatives to this, but can't come up with anything that doesn't violate the DRY principle).

If there are any other improvements you can think of, let me know. I'm very keen on integrating this into LINQKit.

Joseph Albahari

Luke Marshall said...

Thanks Joe! I'm honored!

Totally agree...I was actually looking into the performance concern the other day, but from the compiled query point of view.

It works great*, but I'm very keen to redo the implementation to use AsExpandable to combine my expression trees (much more elegant).

*It would be great if compiled queries could work with other providers too! That's next on my research list :)

I'll definitely have a think regarding improvements / reflective performance etc. I'm hoping there's a neat solution for that.


Thomas Eyde said...

I have a bad feeling when it comes to modify the expression tree. My fear is that other developers which haven't touched this particular code will have a hard time finding out what's wrong when disaster happens.

Also, doesn't this introduce code which you just have to know about? Is it possible to design this particular api to always suggest the right thing to do?

The thing that bugs me, is the underlying assumption that the specification pattern breaks encapsulation. When we need to break the encapsulation so we can query the database, doesn't that tell us the encapsulation is wrong?

Luke Marshall said...

Thanks for the feedback Thomas!

I tend to agree with your feelings on modifying the expression tree, but it's power and flexibility can't be ignored.

And once you've played with them, they're not so hard to understand.

Hopefully you've had a look at my later posts, I improved the support for linq syntax - so it should be harder for developers to hit problems.

Although it's not always practical, I think developers need to understand code they copy from the net.

I agree with your view on the specification pattern - though what I've written here is quite far from that pattern... calculated fields is a more apt description.

Joe said...

I've thought on this some more, and am now vascillating on whether this solution adds enough value.

The problem is that the alternative is not actually all that awkward, if you use PredicateBuilder (part of LINQKit). To demonstrate, consider the IsOverdue property in your demo. You could instead define it simply as a static property as follows (your blog won't allow angle-brackets):

static public readonly Expression[Func[Loan, bool]] IsOverdue =
x => x.DateReturned == null && (DateTime.Now - x.DateBorrowed).TotalDays > x.LoanPeriod);

and then use it like this:

var query = Loans.Where (Loan.IsOverdue);

or, if you wanted to query books that were overdue, and lent to Luke:

var query = Loans.Where (Loan.IsOverdue.And (l => l.Name == "Luke"));

(The .And operator is defined by PredicateBuilder).

All in all, not too bad. In contrast, a boolean IsOverdue instance property is tidier to consume, but messier to define. More signficantly, I think QueryProperty has a higher WTF-factor than PredicateBuilder, and requires a specially suffixed field - which grates for reasons I can't quite define :)


Luke Marshall said...

I totally get where you're coming from - it definitely has the WTF factor.

I've struggled to find a declaration syntax that I like, but I haven't quite managed to hit the mark.

However I'm moderately happy with the attribute based syntax from my last post.

That PredicateBuilder is a cool alternative, however it does push some of the WTF to the client developer.

That's arguably the best part of my solution, since the 'implementation' is abstracted away.

Check out my latest posts, I added support for method calls etc... I don't think PredicateBuilder can support this functionality - but please prove me wrong!

Either way, coding is a personal preference, so I'm not offended if you don't include this with LinqKit.

However you might want to include some of the changes I made to ExpandableQuery.

A minor change that allows other users to easily write their own expression tree parsing routines.

As always, I appreciate the feedback!

Joe said...

Yes - you're right. With PredicateBuilder you can't reproduce the functionality, say, of a calculated string property for use in projections. So I guess it's still worth considering.

I'll ponder on it more...