Math Geek Coder

Bing! - My Claim to Fame

2009-07-22T19:50:00.001+10:00

Judging by the date of my last post, it turns out that the “it’s the thought that counts” - is all lies! Despite my best intentions, an entire year has disappeared into the ether.

While away from my blogosphere I have continued my programming and my math/statistics degree, but I have changed jobs (10 months ago) – which brings me to this post.

My ‘new’ employer is “Bing Technologies” and a fun trivia fact is that Microsoft bought the www.bing.com domain from us.

Cool huh!

Structure Inlining with .Net 3.5 SP1

2008-08-01T17:45:00.000+10:00

I just stumbled on a fascinating performance improvement coming up in .Net 3.5 SP1.

When working with value types (structs) the JIT currently hits the stack each time you access a field - which is quite inefficient.

In SP1 they've added an algorithm called "value type scalar replacement"; which intuitively does exactly what it sounds like!

Take the following code:

static int MyMethod1(int v)
{
    Point point;  // gets a stack location.
 
    // All 3 field accesses involve stack operations.
    point.x = v; 
    point.y = v * 2;
 
    return point.x;
}

The JIT will replace the value type (Point is a struct) with scalars:

static int MyMethod2(int v)
{
    int x; int y;  // Was “struct point;”. Now replaced by x and y.
    x = v; 
    y = v * 2;
    
    return x;
}

After the JIT inlines the structure, it can optimise the routine by placing the integers on the register rather than the stack.

What's the difference?

We are roughly looking at a 10x improvement! (Using a quick and dodgy test app)

This is fantastic, especially when dealing with lots of structs like points and rectangles etc.

What's the catch?

This isn't magically going to work on all your value types; there are strict guidelines when this replacement can occur:

- The value type must have no more than 4 fields

- These fields must be value types, or object references

- You must use [StructLayout(LayoutKind.Sequential)], which is the default anyhow.

There some other catches, but if you are creating complicated structures, you really should think about making them a class.

More Improvements

This is just a snippet of the algorithm, whereas the original article explains it in much more detail.

BTW: This is in the CLR, so all .Net languages benefit! And of course this includes F# and FParsec.

Cheers to SP1!

Advanced Domain Model queries using Linq - Part IV

2008-07-23T17:29:00.006+10:00

This should be the last part in my series of posts. In this part I'll cover support for parameters via method calls.

Parameter Support (Contrived Example)

Continuing the Library example, how would you query the books that will be overdue next week?

Well you could write the following:

var nextWeek = from a in repository.AsQueryable()
               where a.CheckOverdue(DateTime.Today.AddDays(7))
               select a;

How it works

The CheckOverdue Property simply returns a Lambda Expression as it's result.

class CheckOverdueProperty : QueryProperty<Loan, Func<DateTime, bool>>
{
    public CheckOverdueProperty()
        : base(x => y => (x.DateReturned == null && (y - x.DateBorrowed).TotalDays > x.LoanPeriod))
    { }
}

I've modified the expression parsing routines to support parameters (including multiple) and even nested lambdas!

Nested Lambdas?

I can't really see the need for it, but hey... why not! Curry anyone? :)

class MyProperty : QueryProperty<Loan, Func<int, Func<int, Func<int, double>>>>
{
    public MyProperty()
        : base(a => x => y => z => x + y + z + a.LoanPeriod)
    { }
}

So how would you call such a monstrosity?

var test = from a in repository.AsQueryable()
           select a.MyProperty(5)(4)(3);

But check out the SQL! Great work Linq to Sql guys!

SELECT (CONVERT(Float,5 + 4 + 3)) + [t0].[LoanPeriod] AS [value]
FROM [dbo].[Loan] AS [t0]

Declaration Syntax

If you're particularly astute, you'll have noticed that my syntax for declaring a QueryProperty has changed slightly since the last post.

Here's the class declaration of the CheckOverdue Property:

[QueryProperty(typeof(CheckOverdueProperty))]
public bool CheckOverdue(DateTime date)
{
    return CheckOverdueProperty.GetValue<CheckOverdueProperty>(this)(date);
}

Although this is a method, you can still define it as a property (notice that I return a lambda expression).

[QueryProperty(typeof(CheckOverdueProperty))]
public Func<DateTime, bool> CheckOverdue
{
    get { return CheckOverdueProperty.GetValue<CheckOverdueProperty>(this); }
}

So What's Changed?

Well, QueryProperties no longer derive from an Attribute class. Although it worked, I felt dirty about it.

I borrowed from Fredrik's syntax and pass the property type to the Attribute - much cleaner.

Since I'm no longer defining the QueryProperty as static, I had to change the way it is referenced by the class property.

The code above shows my attempt at making this easy, yet efficient - but it's up to you how to implement it. E.g. this would work well too:

[QueryProperty(typeof(CheckOverdueProperty))]
public Func<DateTime, bool> CheckOverdue
{
    get { return CheckOverdueProperty.Value(this); }
}
 
private static CheckOverdueProperty CheckOverdueProperty = new CheckOverdueProperty();

If you can think of better ways I'd love to hear it!

Expression Parser

Since I posted the parser code in my last post, here it is again - and my has it grown!

class QueryPropertyEvaluator : ExpressionVisitor
{
    // Basic member access support for QueryProperties
    protected override Expression VisitMemberAccess(MemberExpression m)
    {
        if (m.Expression != null)
        {
            var property = GetProperty(m.Member);
 
            if (property != null)
                return Expression.Invoke(property.GetExpression(), m.Expression);
        }
 
        return base.VisitMemberAccess(m);
    }
 
    // Support method call abstractions over QueryProperties
    // Assumes method call has same parameters (in order!) as QueryProperty
    // Also supports static methods, as long as first parameter is the class object
    protected override Expression VisitMethodCall(MethodCallExpression m)
    {
        Expression obj = this.Visit(m.Object);
        IEnumerable<Expression> args = this.VisitExpressionList(m.Arguments);
 
        var property = GetProperty(m.Method);
 
        if (property != null)
        {
            if (obj != null)
            {
                // Method on QueryProperty class
                return VisitLambdaInvocation(Expression.Invoke(property.GetExpression(), obj), args);
            }
            else
            {
                // Assumes static method where first parameter is object instance
                return VisitLambdaInvocation(Expression.Invoke(property.GetExpression(), args.First()), args.Skip(1));
            }
        } 
 
        if (obj != m.Object || args != m.Arguments)
            return Expression.Call(obj, m.Method, args);
 
        return m;
    }
 
    // Support lambda expressions within QueryProperties
    protected override Expression VisitInvocation(InvocationExpression iv)
    {
        IEnumerable<Expression> args = this.VisitExpressionList(iv.Arguments);
        Expression expr = this.Visit(iv.Expression);
 
        // Rewrite InvokeExpression - Folds out inner lambda
        if (expr.NodeType == ExpressionType.Invoke)
            return VisitLambdaInvocation((InvocationExpression)expr, args);
 
        if (args != iv.Arguments || expr != iv.Expression)
            return Expression.Invoke(expr, args);
 
        return iv;
    }
 
 
    // Recursively 'flattens' the lambda invocation expressions
    // Rewrites the nested lambda expression tree to work with Linq to SQL
    protected Expression VisitLambdaInvocation(Expression expression, IEnumerable<Expression> args)
    {
        if (expression.NodeType != ExpressionType.Invoke)
            return Expression.Invoke(expression, args);
 
        var invoke = (InvocationExpression)expression;
        var lambda = (LambdaExpression)invoke.Expression;
 
        // Func<TClass, Func<Arg0, Arg1..., Result>> -> Func<TClass, Result>
        var arguments = lambda.Type.GetGenericArguments().ToArray();
        arguments[arguments.Length - 1] = arguments.Last().GetGenericArguments().Last();
 
        return Expression.Invoke(
                    Expression.Lambda(
                        lambda.Type.GetGenericTypeDefinition().MakeGenericType(arguments),
                        VisitLambdaInvocation(lambda.Body, args),
                        lambda.Parameters),
                    invoke.Arguments);
    }
 
    #region MemberInfo caching
 
    private Dictionary<MemberInfo, QueryProperty> lookup = new Dictionary<MemberInfo, QueryProperty>();
 
    // This is not strictly needed, but adds a performance improvement.
    // Feel free to remove this code if you think memory is more important
    protected QueryProperty GetProperty(MemberInfo info)
    {
        QueryProperty result = null;
 
        if (lookup.TryGetValue(info, out result) == false)
        {
            var attribute = Attribute.GetCustomAttribute(info, typeof(QueryPropertyAttribute)) as QueryPropertyAttribute;
 
            if (attribute != null)
                lookup.Add(info, result = attribute.GetProperty());
        }
 
        return result;
    }
 
    #endregion
}

Conclusion

As usual, here is the sample code for you to experiment with yourself.

I hope you've enjoyed this series, I certainly have.

If I get the chance I might write a post explaining the expression parsing in detail, or the compiled queries - but till then...

Cheers!

Advanced Domain Model queries using Linq - Part III

2008-07-17T14:22:00.004+10:00

I didn't intend to post so quickly after my last post, but some comments from Nick motivated me to try something different.

So what's changed?

Not much. Defining your query properties has changed (and obviously some implementation details). All the linq syntax stuff stays the same.

Here's the same Loan class, rewritten:

class Loan
{
    [OverdueFee]
    public decimal OverdueFee
    {
        get { return OverdueFeeAttribute.Property.GetValue(this); }
    }
 
    [IsOverdue]
    public bool IsOverdue
    {
        get { return IsOverdueAttribute.Property.GetValue(this); }
    }
 
    [DaysOverdue]
    public double DaysOverdue
    {
        get { return DaysOverdueAttribute.Property.GetValue(this); }
    }
}

And here's our query property definition:

class IsOverdueAttribute : QueryPropertyAttribute
{
    public IsOverdueAttribute() : base(Property) { }
 
    public static readonly QueryProperty<Loan, bool> Property = new QueryProperty<Loan,bool>(
        x => (x.DateReturned == null && (DateTime.Now - x.DateBorrowed).TotalDays > x.LoanPeriod));
}
 
class OverdueFeeAttribute : QueryPropertyAttribute
{
    public OverdueFeeAttribute() : base(Property) { }
 
    public static readonly QueryProperty<Loan, decimal> Property = new QueryProperty<Loan, decimal>(
        x => (decimal)(x.DaysOverdue * 0.45));
}
 
class DaysOverdueAttribute : QueryPropertyAttribute
{
    public DaysOverdueAttribute() : base(Property) { }
 
    public static readonly QueryProperty<Loan, double> Property = new QueryProperty<Loan, double>(
        x => x.IsOverdue ? (DateTime.Now - x.DateBorrowed).TotalDays - x.LoanPeriod : 0);
}

What's the difference?

Well, depending on your programming tastes, it's cleaner. We've reduced some of the tight coupling, and we now have the ability to spread the logic (which is more suitable in certain situations).

And if you're wondering why I seem to have an obsession with static members, it's mainly due to performance - and truth be told, I'm pre-optimizing here. Generally I'm not such a fan... honest!

Performance anxiety

Speaking of performance, using Attributes has a penalty. According to my tests, it's roughly 30% slower to parse the expression.

This might sound like a lot, but we're talking about 0.37 vs 0.28 milliseconds per query here - and once you consider the database retrieval time... it's really nothing.

I don't even parse the expression tree with linq to objects - so no impact at all!

Bonus code (Expression Parser)

Here's my expression parsing code, I know you want to see it:

class QueryPropertyEvaluator : ExpressionVisitor
{
    protected override Expression VisitMemberAccess(MemberExpression m)
    {
        if (m.Expression != null)
        {
            var attribute = Attribute.GetCustomAttribute(m.Member, typeof(QueryPropertyAttribute)) as QueryPropertyAttribute;
 
            if (attribute != null)
                return Expression.Invoke(attribute.Property.GetExpression(), m.Expression);
        }
 
        return base.VisitMemberAccess(m);
    }
}

Beautiful isn't it!

Feedback please

If you've been following this little series, I'd love to hear what you think of the new syntax.

I'm still hesitant, since this new syntax results in more code - but I think it's a cleaner approach.

As usual, you can find the sample app here - enjoy!

Advanced Domain Model queries using Linq - Part II

2008-07-16T17:53:00.005+10:00

Since my previous post I've rewritten the internals to make use of LinqKit - a fantastic lightweight project.

See the changes

I took my rewrite as an opportunity to support the full query syntax:

var overdue = from a in repository.AsQueryable()
              where a.IsOverdue == true
              select a.OverdueFee;
 
var days = repository.AsQueryable()
                     .Where(x => x.IsOverdue)
                     .Select(x => x.DaysOverdue);

and added easy support for compiled queries (currently only supporting linq to objects and linq to sql):

var query = repository.Compile(
    source => from x in source
              where x.IsOverdue == true
              select x.OverdueFee);
 
var results = repository.Execute(query);

I've also optimized the expression parsing (speed difference is negligible to normal queries), and I tweaked the QueryProperty construction:

#region IsOverdue Property
 
public bool IsOverdue
{
    get { return IsOverdueProperty.GetValue(this); }
}
 
static public readonly QueryProperty<Loan, bool> IsOverdueProperty = new QueryProperty<Loan, bool>(
    x => x.IsOverdue, 
    x => x.DateReturned == null && (DateTime.Now - x.DateBorrowed).TotalDays > x.LoanPeriod);
 
#endregion

The big change here is that I've added an extra parameter; a reference back to the mapping property.

I'd really like a memberof(Loan.IsOverdue) keyword style syntax in C#, but since there's not, I use this expression to grab the MemberInfo - which is used in the expression parser.

It's a slight DRY drawback, but it's a faster way than using reflection (and avoids an ugly string lookup) - and it's much faster than using attributes.

Get the code

Most of the big changes are all implementation specific, so be sure to check out the sample app.

Please let me know what you think!

BTW: Joseph, I made some architectural changes to the LinqKit.ExpandableQuery stuff. Externally it behaves exactly the same, but it now lets me reuse a lot more code.

It's light on the comments, but I'm hoping my next post will explain myself.

ed: Continue to Part III

Ctrl . - My favourite shortcut

2008-07-15T11:55:00.002+10:00

I'm a huge keyboard shortcut fan (for a windows developer)... for common tasks it's just annoying to have to grab the mouse.

For me, one such task was the rename / resolve support in Visual Studio:
Using the mouse to bring up this dialog has got to be the most frustrating task in Visual Studio.

So much so, I refused to use it; instead opting for the right-click 'resolve' approach, or even manually typing in namespaces.

A few months ago, I stumbled on "Ctrl ." and to my delight ended my coding frustration.

I'm now more productive, less frustrated... do yourself a favour and give "Ctrl ." a go!

Advanced Domain Model queries using Linq

2008-07-07T18:21:00.005+10:00

I read a fantastic blog article from Nick Blumhardt yesterday which kicked off some crazy ideas in my head.

The Problem

With the following class, I can query on IsOverdue using linq... but not using linq to sql (or linq to nhibernate). How frustrating!

class Loan
{
  // Member details omitted
  public bool IsOverdue
  {
    get 
    {
       return DateReturned == null && 
          (DateTime.Now - DateBorrowed).TotalDays > LoanPeriod;
    }
  }
}

This is a fundamental issue in relational to object mapping systems, but we can fix it using some linq trickery.

Disclaimer

I think my solution is very cool, but I'm not sure how acceptable it is for a production environment.

Please leave feedback if you like it / makes you want to vomit.

The change

Let's modify the Loan class implementation of IsOverdue:

#region IsOverdue Property
 
public bool IsOverdue
{
  get { return IsOverdueProperty.GetValue(this); }
}
 
static public readonly QueryProperty<Loan, bool> IsOverdueProperty = QueryProperty.Register<Loan, bool>(
    x => x.DateReturned == null && (DateTime.Now - x.DateBorrowed).TotalDays > x.LoanPeriod);
 
#endregion

Here we move the logic for IsOverdue into a separate QueryProperty member that takes care of the nasty details.

Hopefully you'll agree that the code is still fairly readable, and encapsulated with all the logic still bundled inside the Loan class.

By moving this logic into a lambda expression, we can now write queries for both database and object collections! e.g

// Database Access
using (var db = new RepositoryDatabase<Loan>())
  var overdue = db.Where(y => y.IsOverdue && y.LoanPeriod == 5);
 
// Collection Access
using (var list = new RepositoryCollection<Loan>())
  var overdue = list.Where(y => y.IsOverdue && y.LoanPeriod == 5);

More Goodness

When coding these query properties it can be very handy to reference other query properties. e.g.

#region DaysOverdue Property
 
public double DaysOverdue
{
  get { return DaysOverdueProperty.GetValue(this); }
}
 
static public readonly QueryProperty<Loan, double> DaysOverdueProperty = QueryProperty.Register<Loan, double>(
  x => x.IsOverdue ? (DateTime.Now - x.DateBorrowed).TotalDays - x.LoanPeriod : 0);
 
#endregion
 
#region OverdueFee Property
 
public decimal OverdueFee
{
    get { return OverdueFeeProperty.GetValue(this); }
}
 
static public readonly QueryProperty<Loan, decimal> OverdueFeeProperty = QueryProperty.Register<Loan, decimal>(
  x => (decimal)(x.DaysOverdue * 0.45));
 
#endregion

This allows us to create very complicated queries in a very simple manner.

The Catch

Too good to be true? You're right; there's a devil in the details.

You might have picked it up already - how does linq interpret my query property correctly?

By modifying the expression tree!

My Repository class implements the Where() and translates the expression; replacing any QueryProperty references with a call to the actual lambda expression.

The really ugly part is that it's a string based lookup ie. "<name>Property". Though I also check the type to be extra safe.

It's really not a bad trade off, considering it's very similar to how WPF DependencyProperty's work.

The Conclusion

This is a really easy and seamless way to implement custom queryable properties in your domain objects.

I've only touched on a fraction of the possibilities with this technique, but it shows the power and flexibility of linq.

Make sure you check out the sample application for the full code.

ed: Continue to Part II

Loop optimization trickery

2008-06-27T16:37:00.005+10:00

A while ago I was playing around with C# 3.5 lambda expressions trying to work out how it behaves with variable scope.

What do you think is written to the console?

class Program
{
    static void Main(string[] args)
    {
        // Write number to console in worker thread
        for (int i = 1; i < 9; i++)
            DoLater(() => Console.Write(i));
    }
     
    // Call action delegate on separate thread
    static void DoLater(Action action)
    {
        // I'm avoiding using another lambda here to simplify test
        new Thread(new ParameterizedThreadStart(DoLater)).Start(action);
    }
    
    // Calls the action after waiting for a second
    static void DoLater(object action)
    {
        Thread.Sleep(1000);
        ((Action)action)();
    }
}

If you initially guessed (like me) that it displays the numbers 1-9 in some random order (eg. '14385726') - you'd be wrong... it actually produces '999999999' every time.

So I cranked up .net reflector (I love it) and had a look at my code (optimization set to .NET 1.0 - then translated by me):

private static void Main(string[] args)
{
    AutoDelegateClass tmpVar = new AutoDelegateClass();
    tmpVar.i = 1;
    while (tmpVar.i < 9)
    {
        DoLater(new Action(tmpVar.Method));
        tmpVar.i++;
    }
}

This is much more obvious... .net optimizes the lambda expression out of the loop!

Well that was unexpected, but guess what happens when I change main to this:

static void Main(string[] args)
{
    for (int i = 1; i < 9; i++)
    {
        int j = i;
        DoLater(() => Console.Write(j));
    }
}

This works! ie. it does not optimize the lambda expression out of the loop.

Although confusing at first, it actually makes a lot of sense from the compiler point of view - the initial expression doesn't actually change, since we are referencing a mutable variable... so why not optimize it?

It might be convenient to turn off this optimization programmatically - kind of like a volatile keyword or something, but it's no big deal.

I guess it goes to show that functional code works best with immutable data... and that reflector is way cool.

BTW: The exact same thing occurs in foreach over a collection; and also using classes rather than structs.

UI - You don't know how much you suck

2008-06-23T13:35:00.002+10:00

I made a small mistake in my last post - instead of 'do what you want' what I meant to say was:

Hire a damn graphic designer and buy some icons!

This is a subtle difference, but the delusion of competency is unfortunately common.

Like your home handyman disasters, the DYI attitude for user interfaces can cost you big time.

However IT departments are usually too cheap, and some developer ego's are too big to realize that they can't do everything... thus UI abominations are created everyday.

UI development is hard, harder than most people think.

The UI layer is where everything comes together, and as a result it's here you find all the issues with the underlying architecture.

If you're a UI developer - you suck, so make sure you check out the UI guidelines for your dev system (Microsoft, Apple, Gnome or KDE).

If you're not a UI developer; shut up, fix your code and give the UI guys a break.

Oh, and good luck!

Disclaimer: I've done both jobs, so I suck twice as much.

UI - You can't please everyone

2008-04-22T10:38:00.002+10:00

Although it's difficult to create good UI, it doesn't mean that bad UI is easy - in fact some people go to great lengths to produce bad UI.

That said, good UI simply boils down to:

Intuitive, Pretty and Performant - in a combination that targets the intended user.

eg. A programmer is going to prefer performance over pretty, and a hairdresser might prefer pretty over intuitive... but you still need all three for a good user experience.

If you don't have access to your user (and that's generally the case), just pretend that you are the user.

If you wouldn't use your own app; that's a good sign of bad UI.

In the end, you can't please everyone - so please yourself, then at least one person is happy.

Six Percent Genius

2008-02-20T12:35:00.003+10:00

Project Euler (pronounced "Oiler") provides a number (~182) of challenging math/programming problems... which is a fantastic environment to play with F#!

I've only completed 6% of the questions but I've already learnt a lot. (F# is huge)

If you have some free time (or get bored during your compiles), I'd definitely recommend having a go - in your own language of choice of course!

Deriving Miss Daisy

2008-01-15T12:20:00.000+10:00

As I've mentioned previously, I'm slowly reading through SICP - it's an amazing book and reading it has been a humbling experience.

'Geeking' Out

I completely geeked out reading one example on functional derivatives. (Take a math expression as a parameter and return the derivative as an expression)

It's in the first chapter, and they use this to implement Newton's Method -> to then implement a Sqrt function!

Approximate Derivative

After I picked my brain off the floor, I decided to have a crack at implementing derivatives in C# using lambda expressions:

Using the formula of a derivative (above), we can simply code an approximate derivative (below):

Func<double, double> Approximate(Func<double, double> f)
{
    var h = 0.0001;
    return a => ((f(a + h) - f(a)) / h);
}

Symbolic Derivation

Easy huh! So for fun I decided to use linq expression trees to derive algebraically.

It's easy to get an expression tree in C# - simply use the Expression<> type.

Func<double, double> Derive(Expression<Func<double, double>> e)
{
    var lambda = new Derivative().Eval(e) as LambdaExpression;
    return (Func<double, double>) lambda.Compile();
}

I parse the expression tree using Matt Warren's ExpressionVisitor class - and using the basic rules of Derivation (Constant, Product and Quotient rules) we get:

class Derivative : ExpressionVisitor
{
    public Expression Eval(Expression exp)
    {
        return this.Visit(Evaluator.PartialEval(exp));
    }
    
    protected override Expression VisitBinary(BinaryExpression b)
    {
        // Product Rule: (fg)' = f'g + fg' 
        if (b.NodeType == ExpressionType.Multiply)
        {
            return Expression.Add(
                Expression.Multiply(Eval(b.Left), b.Right),
                Expression.Multiply(b.Left, Eval(b.Right)));
        }
    
        // Quotient Rule: (f/g)' = (f'g - fg') / (g*g)
        if (b.NodeType == ExpressionType.Divide)
        {
            return Expression.Divide(
                Expression.Subtract(
                    Expression.Multiply(Eval(b.Left), b.Right),
                    Expression.Multiply(b.Left, Eval(b.Right))),
                Expression.Multiply(b.Right, b.Right));
        }
    
        return base.VisitBinary(b);
    }
    
    // Parameter Derivation: f(x)' = 1
    protected override Expression VisitParameter(ParameterExpression p)
    {
        return Expression.Constant(1.0);
    }
    
    // Constant Rule f(a)' = 0
    protected override Expression VisitConstant(ConstantExpression c)
    {
        return Expression.Constant(0.0);
    }
}

Wrapping it up

To quickly finish this long blog post, here's the usage code:

// f(x * x * x)' = 3 * x * x
var approx = Approximate(x => x * x * x);
var derive = Derive(x => x * x * x);
 
var y1 = approx(1); // 3.0000300001109532
var y2 = derive(1); // 3.0

And there it is! Useful? maybe not. Fun? Definitely!

The 'Y' in Blog

2008-01-07T10:21:00.000+10:00

It's quite demoralizing when you consider how much you don't know. I am continually reminded that I underestimate it's sheer quantity... it's a sad realization.

Since there is so much already written by people more knowledgeable and intelligent than myself, why do I write?

Well, I started this blog to show my appreciation, and hopefully to contribute something back to the development community.

So kudos to all of you... keep sharing the love (of maths & programming)!

Associate with people who are searching for answers, avoid those who have found them.

Happy New Year!

MSDTC + Linq to Sql Update

2007-12-07T15:57:00.000+10:00

When I was writing my previous article, I was keenly aware that it had it's limitations. One in particular was using an already open DataContext.

Take the following code:

// Create outside TransactionScope
var c1 = new UserDataContext();

using (var scope = new TransactionScope())
{
    c1.Users.ToList();

    // This will promote to MSDTC
    using (var c2 = new UserDataContext())
        c2.Users.ToList();

    scope.Complete();
}

Since it was created outside the TransactionScope I have no way of automatically sharing these connections.

To help solve this problem (cleanly) I added a SharedConnectionScope class and a UseExistingConnection call (shown below) - both take an existing DataContext.

// Create outside TransactionScope
var c1 = new UserDataContext();

using (var scope = new TransactionScope())
using (SharedDataContext.UseExistingConnection(c1))
{
    c1.Users.ToList();

    // This will share existing connection
    using (var c2 = new UserDataContext())
        c2.Users.ToList();

    scope.Complete();
}

Caveat: It only works with one 'external' connection, but that's still better than none.

You can grab the updated code here.

Quick Application - Tickle Me Pink

2007-12-06T11:44:00.001+10:00

I do all my WPF coding in straight XAML, and for my prototypes I like to use the standard .Net colors.

So instead of playing a fun colorname -> color guessing game, I thought I'd have some fun and whip up this tiny color app (it's ridiculously easy in WPF).

The following code is the entire application, it combines WPF, reflection, Linq and Anonymous Types.

UPDATE: Adjusted layout to look better in google reader.

<Window x:Class="Colors.ColorWindow" Title="Colors" 
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:my="clr-namespace:Colors" WindowStyle="ToolWindow"
    Height="500" Width="200">
  <Window.Resources>
    <DataTemplate x:Key="ColorTemplate">
      <StackPanel Orientation="Horizontal">
        <Border BorderBrush="Black" BorderThickness="1" Width="30" Height="30" Margin="10"
                Background="{Binding Brush}" ToolTip="{Binding Brush.Color}" />
        
        <TextBlock Text="{Binding Name}" VerticalAlignment="Center" />
        </StackPanel>
    </DataTemplate>
  </Window.Resources>

  <ListBox x:Name="colors" ItemTemplate="{StaticResource ColorTemplate}" />
</Window>

public partial class ColorWindow : Window
{
    public ColorWindow()
    {
        InitializeComponent();

        colors.ItemsSource = typeof(Brushes)
            .GetProperties(BindingFlags.Static | BindingFlags.Public)
            .Select(x => new { 
                                Name = x.Name, 
                                Brush = x.GetValue(null, null)
                             });
    }
}

DataModel-View-ViewModel

2007-12-04T18:59:00.000+10:00

WPF changed the rules for UI best practice, but it didn't update the book.

With all the new and improved DataBinding and Command features - a new set of patterns have emerged, including a recommended pattern called DataModel-View-ViewModel (DM-V-VM).

At a conceptual level this pattern looks fantastic – but there are issues... namely it doesn’t work for non-trivial scenarios.

Rather than a boring high level rant, I'll use boring examples to explain why:

Cannot support events

The recommended approach is to use DataTemplates and Commands - but DataTempates cannot handle events, and the command system is not extensive enough (eg. doesn't support mouse clicks).

Supporting events requires using code behind; which means using at least a UserControl base class.

No Keyboard support for Standard Commands

Dan Crevier and John Gossman don't mention the standard Commands, instead opting to create their own – not because it's better practice, but it's easier to demonstrate.

Their suggested approach uses CommandParameter bindings on the buttons, which only works with OnClick (i.e when the shortcut Key is pressed "null" is passed through).

Since the InputBindings.KeyBinding.CommandParameter does not support DataBinding - supporting the keyboard (and still using commands) means we can't use CommandParameters.

UI support breaks abstraction

Say that we want to delete all the selected items from a list; this means storing a list of items in the ViewModel (since we not using CommandParameters), which we then bind from our View.

But this is purely for UI logic rather than business logic!

Complicated scenarios might need multiple properties - and since the binding is not clearly defined, it will introduce bugs.

The abstraction is now broken, so let's just move our ViewModel into the code behind.

Unit testing the UI logic now not easy

No kidding, that's why we initially moved the code into the ViewModel!

All of the real logic should be implemented in the data bound objects or by a class separate to the UI. These classes are unit testable, but the UI integration is not.

Aren't we back to where we started?

Not quite, WPF's new style allows us to develop a cleaner system than the days of WinForms and MFC - and while there's not a published best practice for it, it doesn't mean that it doesn't exist.

UI programming is hard, especially since it looks so easy.

Avoiding MSDTC with Linq to Sql

2007-12-03T07:34:00.000+10:00

Now MSDTC is not evil; it's just overkill when it isn't needed. It slows performance, increases complexity and requires client side configuration... and it can activate when you least expect it.

Take the following code:

void CreateUsers()
{
    using (var scope = new TransactionScope(TransactionScopeOption.Required))
    {
        CreateUser("Fred");
        CreateUser("Joe");

        scope.Complete();
    }
}

void CreateUser(string name)
{
    using (var context = new UserDataContext())
    {
        context.Users.InsertOnSubmit(new User { name = name });
        context.SubmitChanges();
    }
}

This code uses the recommended (and very helpful) TransactionScope class - and unfortunately invokes MSDTC.

Since we're using separate instances of UserDataContext this creates separate database connections - and causes TransactionScope to "promote" the transaction to require MSDTC.

Avoiding this is as simple as sharing the database connection - after all, it is the same database - but passing database connections to sub methods gets ugly really quick.

My Solution

To solve this problem in a cleaner manner, I wrote a Transaction Resource Manager - which works behind the scenes and makes the above code work without change.

It simply shares database connections across the instances (only within a transaction). It's thread safe, and easy to integrate - simply change the DataContext base class in the dbml designer.

Grab the code and check it out. There's an example program and a whole slew of unit tests (around 275 - testing the various permutations of nested TransactionScopes).

Kudos to Nick Guerrea's Weak Dictionary - which I use to weakly track the transactions and share the connections.

Same Solution for other Scenarios

There are other scenarios where this problem arises:

Say you've got a database with many tables (or a plug-in system that shares a database) - a modular approach uses multiple DataContext classes.

Our problem occurs when you need to modify tables from different DataContexts within a single transaction.

Thankfully, the root cause is still the same and our solution solves these cases as well.

This has been a great learning experience for me, and I must say that the Transaction Resource Managers are very interesting (might even make a good alternate ScopeGuard implementation?)

EDIT: See updated article for extra goodness.

Quick Tip - Read Only Automatic Properties

2007-11-28T14:08:00.000+10:00

This is a simple little trick, which I only discovered a few weeks ago.

In the past providing public "read only" access to a member would look something like:

private string connection;

public string Connection
{ 
    get { return connection; }
}

With Automatic Properties in C# 3.5 the code becomes:

public string Connection { get; private set; }

The neat trick is the access modifier (private) next to the set. This allow you to internally modify the member, while publicly providing "read only" access.

VS2008 RTM Update

2007-11-26T14:52:00.001+10:00

I have updated all the downloadable code to compile with the RTM release of VS2008.

It's not a big change, so I'll use this blog entry to update my Xceed <-> Linq code. I've fixed some minor bugs, improved performance a lot and added better linq syntax support.

So if you're using the code, it'd be a good idea to update.

IEnumerable Joy - Batch / Split

2007-11-19T15:35:00.001+10:00

I was inspired by my good friend Nick's latest blog entry Return of the Batch Enumerable.

Here he demonstrates a neat idea of using extension methods and yield to split an IEnumerable into arbitrary batches.

My version is slightly less efficient, but treats the batches as separate lazy collections - so you can use it on multiple threads etc.

I've also added a Split extension, which I'll describe - but first, the code!

static class InBatchesExtension
{
    public static IEnumerable<IEnumerable<T>> InBatches<T>(this IEnumerable<T> source, int batchSize)
    {
        for (IEnumerable<T> s = source; s.Any(); s = s.Skip(batchSize))
            yield return s.Take(batchSize);
    }

    public static IEnumerable<IEnumerable<T>> Split<T>(this IEnumerable<T> source, int items)
    {
        for (int i = 0; i < items; ++i)
            yield return source.Where((x, index) => (index % items) == i);
    }
}

The InBatches returns the same output as Nicks:

Batch: 1 2 3
Batch: 4 5 6
Batch: 7

And the Split batches the output across the collections:

Batch: 1 4 7
Batch: 2 5
Batch: 3 6

So this gives you 2 different batch splitting methods for a generic enumerable.

I love this stuff.

Wield the Yield

2007-11-13T10:29:00.000+10:00

Recently I've noticed an IO pattern emerging combining yield and linq; a powerful and efficient mechanism to read data.

Check the following code: We safely open a file (with the using statement), and yield the results with a loop.

IEnumerable<string> ReadFile(string filename)
{
    using (StreamReader reader = new StreamReader(File.OpenRead(filename)))
        while (reader.EndOfStream == false)
            yield return reader.ReadLine();
}

This simple (and beautiful) code allows us to treat a file like an array of strings - which becomes very powerful when coupled with linq.

// Contrived example - How many file lines refer to "fred"?
var result = ReadFile("file.txt").Where(x => x.Contains("fred")).Count();

I'm sure this same pattern could also be applied to other data retrieval methods, eg algorithm calculation (prime numbers, Fibonacci, etc), network communication (message queue) and even as an abstraction over asynchronous data methods.

Yield provides a lazy enumeration (doesn't execute unless needed) so the solution is generic and efficient - I like it.

n! - let me count the ways

2007-11-05T08:10:00.000+10:00

The other day I had an interesting discrete maths assignment question involving permutations - so I decided to codify it.

"How many ways can six people (Josh, Toby, CJ, Sam, Leo & Donna) sit, if Josh must sit to the left of Leo and to the right of Sam."

Now the astute reader will immediately recognize the answer to be 2 * (4! + 3!3!) = 120; but to be sure I wanted to double check this via C#. (I'm keen to redo this in F# too)

My solution was a quick hack, and is by no means "the best way" - but using linq it sure looks pretty.

class Program
{
    static IEnumerable<List<string>> Permutate(IEnumerable<string> people, 
                                               IEnumerable<string> current)
    {
        // Get remaining people and recurse
        foreach (var person in people.Where(x => current.Contains(x) == false))
            foreach (var result in Permutate(people, current.Concat(new string[] { person })))
                yield return result;
 
        if (current.Count() == people.Count())
            yield return current.ToList();
    }
   
    static void Main(string[] args)
    {
        string[] people = new string[] { "Josh", "Toby", "CJ", "Sam", "Leo", "Donna" };
  
        var result = Permutate(people, new string[] { }).ToList();
   
        // Josh sits to the left of Leo
        var Josh_Leo = result.Where(r =>
            r.FindIndex(x => x == "Josh") < r.FindIndex(x => x == "Leo"));
 
        // Josh sits to the right of Sam
        var Sam_Josh = Josh_Leo.Where(r =>
            r.FindIndex(x => x == "Sam") < r.FindIndex(x => x == "Josh"));
 
        var count = Sam_Josh.Count();
    }
}

This simple example shows why I'm a such fan of linq in C# - the pre-linq code for this certainly wouldn't be as eloquent.

Oh, and it also shows why I'm a fan of maths... it's much more efficient to just to calculate the factorials.

Update: Formatting code in a blog sure is a pain!

F# is so Passé

2007-10-30T08:51:00.000+10:00

Lately I've been trying to juggle learning a mix of technology. WPF, .net 3.5 (linq), maths, F#... Sometimes I feel like I have Internet Multitasking Syndrome.

Although Internet Multitasking Syndrome is not a known medical disorder (I just made it up five minutes ago), it is not uncommon for people to become so immersed in their online activities that their cognitive abilities wane.

After hours starring at a screen, flipping between web pages and information outlets, people can develop a feeling of anxiety, stress and a decrease in mental performance, said John Suler, author of The Psychology of Cyberspace.

"There are limits to how much information one person can process," he continued.

I started learning about a little known language called F# a couple of months ago, where I read most of the Foundations of F# book over a rather intense week.

Since then I've dropped the ball... there's just too much to learn within our software field.

I'm keen to get back into it, as F#'s functional principles are very appealing to me. There's a sense of power and elitism playing with such a language. It's close to it's mathematical roots, and has the potential to create highly concurrent systems.

These things are addictive for software engineers. Power and elitism - it's why C/C++ are so popular and why VB programmers get no respect.

Now that F# is an official MS language, it's becoming quite popular. While this is fantastic for the language - a small selfish part of me wants to keep it for myself. I don't want it to become 'common' and spoilt by countless bad programmers writing spaghetti code.

Unfortunately this also includes me - so to avoid becoming an Italian Chef, I'm planning to read Structure and Interpretation of Computer Programs during my holidays. I've heard this is a classic book covering functional programming (in scheme).

Wish me luck!

Death Trap

2007-10-10T14:28:00.000+10:00

When I first started riding my motorbike I became much more aware of my environment... and not in a pleasant way. After coming out of my metal cocoon (car), everything became a threat and I felt fully exposed to the world.

It's with good reason too; I live in beautiful Queensland where every week 1 rider dies and 15 are injured. And I don't need a maths degree to know that the statistics are against me.

Even still I love riding my bike, it got me out of my comfort zone and I have a new perspective, and a new sense of freedom and opportunity.

So here comes the clichéd analogy to programming.... Our comfort zone is our greatest death trap.

Our industry moves far too quickly learn everything, so it's an easy trap to become stagnant with familiar technology.

We've all been victim's of stagnancy at some point, but tragically some developers spiral down until they feel so far behind there's no point playing catch up.

Then they're stuck working with an obscure, unsupported technology - or they get promoted into management.

Others are trapped by their language of choice. I met a guy just recently who, with great passion, thought 'C' was the language for everything. Web apps, database apps... everything.

I felt for this guy, because I've been there before... and it's utter madness.

It takes effort and it's painful to learn new things. There's also a high risk that what you learn is completely redundant... but while it's important to pick your technology carefully - it's better to learn something than nothing at all.

My challenge is to never stop learning; it's not without risk or effort - but the rewards of freedom and perspective are invaluable.

I don't want to be a Maths Teacher

2007-10-02T08:02:00.001+10:00

Before I started my maths degree I braced myself for the inevitable question "why maths?"

That's okay, maths isn't everyones favorite subject - but as it turns out the more popular question is "do you want to be a Maths teacher?"...

No!

Don't get me wrong, I have no problems with teaching, in fact it might be cool to be a lecturer one day... but it makes me sad that people think learning maths is only good for teaching maths.

It's like the definition of Recursion: see Recursion; a tragic cycle of futility; a perpetual motion machine that affects nothing and goes nowhere.

No!

I want to shout out from the roof tops - don't people realize that maths is the basis of all technology and human advancement?

Economically, math brings order to chaos. The steady job, the home (or mortgage), even food on the table and clothes on our backs - thank maths. Too melodramatic? Google the Great Depression sometime.

Math heals us when we are sick. Statistical drug trials determine what medicine works, mathematical models of the human body are used by doctors and physiotherapists to determine better techniques for recovery.

I won't even get started on computers or civil construction.

Above all, maths is eloquent, philosophical and beautiful. It changes your perceptions and makes you appreciate everyday life even more.

</rant>

Updatable, inheritable, WPF bounded Linq queries

2007-09-27T13:26:00.000+10:00

This is part 4 of my "binding Linq to Xceed's data grid" blog series; however I've changed direction dramatically.

Skipping the details, heres the code binding to a linq query:


  users.LinqContext = new UserDataContext();
  
  users.ItemsSource = from i
                      in users.DataSource<User>() 
                      select i;

And this code fully supports inserts/updates/deletes.

The implementation lives inside the DataSource<User>() call. This devious character returns a reference to a IBindingList (which supports the table modifications) and implements IQueryable (to support linq queries).

What about inheritance?

Imagine you had a sub class called SuperUser; if you've played with linq inheritance you'll know that you have to use the base type User to actually do anything. It's kind of a pain, and you get real friendly with the .OfType<>() call.

So here's the code again, this time selecting all SuperUsers.


  users.ItemsSource = from i 
                      in users.DataSource<User, SuperUser>() 
                      select i;

Can I use .Take(x)?

As far as I can tell, all the standard linq statements are supported (including aggregate operators). I haven't looked into partial classes and anonymous types, though I doubt they'd work well.

I won't go into the laborious details of how it's all implemented, just know that it's been fun playing with linq expression trees and writing my own IQueryProvider.

If you are interested, the code can be found here.

I haven't seen anything quite like this out there, so if you have some opinions - I'd love to hear some feedback.

Linq Table and GetNewBindingList()

2007-09-24T10:25:00.000+10:00

While I'm talking about binding linq data tables to WPF, I should mention the simplest method, namely Table.GetNewBindingList();

This method allows you to add/insert/update (even with Xceed's grid) quite easily, but does not save implicitly, ie. you have to manage it yourself.

If you prefer to batch up all your changes and save en mass at a special point (this is a common UI pattern) - then this might be a great solution for you.

One issue with with approach however, is that it only supports the whole table, ie. not based on queries. This might be fine if you are using the WPF's data filters, but it's not an optimal result with large amounts of data.

I'm planning to investigate these two aspects in the near future, and hopefully I'll find an elegant solution.

Linq with Xceed's WPF Grid - Part III - Hooking up the Delete

2007-09-21T12:15:00.000+10:00

One of the under appreciated features of WPF is its Command support. This feature allows you to hook up events in a more abstract and loosely coupled way; with a great collection of common actions provided by default.

To hook up delete functionality from Xceed's data grid to a Linq data table, I took advantage of these Command gems and as you will see, they make things very easy to use.

The provided ApplicationCommands.Delete command, comes with a localized string ("Delete") and associated key bindings ("Del" Key).

Handling this command in the LinqGrid is as simple as calling the following in the initialization routines:


CommandBindings.Add(new CommandBinding(ApplicationCommands.Delete, 
  OnDeleteCommand, OnCanDeleteCommand));

Implementing the OnCanDeleteCommand to return true, if any items are selected and the OnDeleteCommand to delete the selected items, and it's done! Well almost.

Automatically deleting the selected items (with the "Del" key) without any warning is usually not the best behaviour, so I implemented a RoutedEvent called PreviewDelete, which allows you to show a message box and/or cancel the delete.

Show me the XAML

Another great benefit of using Commands is the ability to bind other UI elements as the action co-coordinator for the command. i.e Buttons, Context Menus, etc.

Not only is this clean and easy, but it also has built in routines to control the IsEnabled state. ie. Button is disabled if you cannot currently delete.


<Button Command="Delete" 
        CommandTarget="{Binding ElementName=users}" 
        Content="{Binding Command.Text, RelativeSource={RelativeSource self}}" />

The "Command" is quite simply a reference to the ApplicationCommands.Delete command, while the "CommandTarget" is the name I gave to my LinqGrid.

The unfortunately long "Content" attribute is the Buttons label, bound to the Command itself. Remember how I said the provided commands have localized names... this is how you access them in XAML. Thankfully, this is also purely optional.

This is all you need to do. When the button is pressed, the Command is fired and our LinqGrid performs the delete, first checking with PreviewDelete for its permission.

Download the sample solution for the full code.

Linq with Xceed's WPF Grid - Part II (and a half)

2007-09-21T09:00:00.000+10:00

An excellent suggestion was raised from my last post (thanks Marcus), which is to make the Linq DataContext a property on the LinqGrid.

This way the LinqRow can access it via it's parent, rather than via the singleton (which is bad practice in my book).

I've put all this code in a solution (includes sample database and all), so I'll let the code speak for itself.

Download Sample Solution

Linq with Xceed's WPF Grid - Part II

2007-09-20T10:58:00.000+10:00

In my first post I showed a handy binding class that wraps around your linq data tables, now I'm going to show some handy changes to Xceed's data grid, to complete the linq binding.

My main problem with the data grid, is there isn't any good place to commit a transaction after editing a row.

To fix this, we can provide our own DataRow class which handles the SubmitChanges() when a row is dirty.

Update: Modified code to use IsDirty flag. Note that IsDirty is reset to false in base.EndEdit().


 class LinqRow : DataRow
 {
   public override void EndEdit()
   {
     if (IsDirty == true)
       DataSource.Context.SubmitChanges();
   
     base.EndEdit();
   }
 }

To make the DataGridControl aware of our new DataRow class, we need to derive and override the GetContainerForItemOverride() method.


 class LinqGrid : DataGridControl
 {
   protected override DependencyObject 
       GetContainerForItemOverride()
   {
     return new LinqRow();
   }
  
   protected override void OnInitialized(EventArgs e)
   {
     base.OnInitialized(e);
  
     if (View == null)
     {
       View = new TableView();
       AdjustHeadersFooters();
     }
   }
  
   private void AdjustHeadersFooters()
   {
     // Show / hide the element base
     View.FixedHeaders.Clear();
     View.Headers.Clear();
  
     // Add a ColumnManagerRow
     DataTemplate columnTemplate = new DataTemplate();
     columnTemplate.VisualTree = 
       new FrameworkElementFactory(typeof(ColumnManagerRow));
     View.FixedHeaders.Add(columnTemplate);
  
     // Add the insertion row
     DataTemplate insertTemplate = new DataTemplate();
     insertTemplate.VisualTree = 
       new FrameworkElementFactory(typeof(InsertionRow));
     View.FixedHeaders.Add(insertTemplate);
   }
 }

Insertion Row

You may have noticed the AdjustHeadersFooters call in the class above, this code removes the GroupBy control and adds in the Insertion row by default. The GroupBy control is cool, but I rarely use it.

Putting it together

Instead of using the DataGridControl in your xaml file, simply use the LinqGrid.

In my next few posts I'll discuss hooking up the delete, sorting and filtering, and how to use the linq queries instead of the whole table.

Linq with Xceed's WPF Grid

2007-09-20T09:52:00.000+10:00

Binding a read only Linq data table to Xceed's WPF data grid couldn't be easier, but things get a little trickier when you want insert/delete/update functionality.

The general recommendation (from Xceed's samples) is to implement a derived IBindingList collection class to provide the behaviour yourself.

Sounds like a lot of work to do something simple, especially if you want to bind to a lot of different objects - so here's a generic one I've prepared.

BTW: I haven't seen any articles on the web regarding best practice with this, so I'd love any comments if there is a better way.


  class LinqList<T> : BindingList<T>
  {
    ITable table = null;
    
    public LinqList(ITable table)
      : base(table.OfType<T>().ToList())
    {
      this.table = table;
    }
    
    protected override object AddNewCore()
    {
      T item = (T)base.AddNewCore();
    
      if (table != null)
        table.Add(item);
     
      return item;
    }
     
    public override void EndNew(int itemIndex)
    {
      base.EndNew(itemIndex);
    
      if (itemIndex >= 0)
        DataSource.Context.SubmitChanges();
    }
  
    protected override void RemoveItem(int index)
    {
      if (table != null)
        table.Remove(this[index]);
     
       base.RemoveItem(index);
    }
  }

My first code draft uses the BindingList as a base class (to avoid implementing everything myself) and takes a Linq data table as a constructor parameter.

Notice the OfType() call in the constructor, this provides basic support for inheritance with Linq, but more on that in a later post.

I use a singleton to store my Linq data context (DataSource.Context) to call SubmitChanges() after adding a new item.


  static class DataSource
  {
    static DataSource()
    {
      DataSource.Context = new MyDataContext();
    }

    static public MyDataContext Context { get; set; }
  }

Binding to Xceed

Simply bind an instance of this class to your data grid, and you're almost set!

See Part II for some changes that need to be made to the Xceed grid, to make this binding easier.