How to Calculate Code Churn using TFS

Your source control repository can provide a lot of valuable insight into your project. In particular, it can highlight source files that are being edited far too often. Files that are being changed on a regular basis (or by many different developers) indicates there might be one of these problems with your code:

• lots of bugs
• too many responsibilities (failure to adhere to the “Single Responsibility Principle”)
• not extensible enough (failure to adhere to the “Open Closed Principle”)

Counting changes to source control files is often called “code churn”. It’s usually defined as the sum of all added, modified and deleted lines. Most source control systems will have some way of letting you get at this information, and I might post about how to extract it for other systems at a later date, but for now, here’s two approaches you can take if you’re using TFS.

Using the TFS API

The TFS API allows you to get details of each changeset individually. Unfortunately the lines added, deleted and modified aren’t included (or at least I can’t find them), but on the whole you’ll find that simply counting the number of modifications to each file is good enough at identifying trouble areas. I also like to count how many changes each user has made.

Here’s a simple class that demonstrates how to source control statistics from TFS using the API. You pass the URL of your collection (e.g. http://myserver:8080/tfs/MyCollection) into the constructor. Then to get the churn or user statistics you need to specify the path within that collection that you want to examine (e.g. $/MyProject/). You’ll see that I am filtering out only changes to the files I am interested in (e.g. only C# files), and I perform a regex on the path of each item, so changes to the same file in different branches are counted together. You’d need to customise that for whatever branching strategy you are using. I’ve also made it cancellable, as this can take a long time to run if you have a long history.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text.RegularExpressions;
using System.Threading;
using Microsoft.TeamFoundation.Client;
using Microsoft.TeamFoundation.VersionControl.Client;

namespace TfsAnalysis
{
    class TfsAnalyser
    {
        private readonly VersionControlServer vcs;

        public TfsAnalyser(string url)
        {
            var collection = ConnectToTeamProjectCollection(url, null);

            vcs = (VersionControlServer)collection.GetService(typeof(VersionControlServer));
        }

        public TfsAnalyser(string url, string user, string password, string domain)
        {

            var networkCredential = new NetworkCredential(user, password, domain);
            var collection = ConnectToTeamProjectCollection(url, networkCredential);
            
            vcs = (VersionControlServer) collection.GetService(typeof (VersionControlServer));
        }

        /// <summary>
        /// Gets User Statistics (how many changes each user has made)
        /// </summary>
        /// <param name="path">Path in the form "$/My Project/"</param>
        /// <param name="cancellationToken">Cancellation token</param>
        public IEnumerable<SourceControlStatistic> GetUserStatistics(string path, CancellationToken cancellationToken)
        {
            return GetChangesetsForProject(path, cancellationToken).GroupBy(c => c.Committer).Select(g =>
                new  SourceControlStatistic { Key = g.Key, Count = g.Count() } ).OrderByDescending(s => s.Count);
        }

        /// <summary>
        /// Gets Churn Statistics (how many times has each file been modified)
        /// </summary>
        /// <param name="path">Path in the form "$/My Project/"</param>
        /// <param name="cancellationToken">Cancellation token</param>
        public IEnumerable<SourceControlStatistic> GetChurnStatistics(string path, CancellationToken cancellationToken)
        {
            return GetChangesetsForProject(path, cancellationToken)
                .Select(GetChangesetWithChanges)
                .SelectMany(c => c.Changes) // select the actual changed files
                .Where(c => c.Item.ServerItem.Contains("/Source/")) // filter out just the files we are interested in
                .Where(c => c.Item.ServerItem.EndsWith(".cs"))
                .Where(c => ((int)c.ChangeType & (int)ChangeType.Edit) == (int)ChangeType.Edit) // don't count merges
                .Select(c => Regex.Replace(c.Item.ServerItem, @"^.+/Source/", "")) // count changes to the same file on different branches
                .GroupBy(c => c)
                .Select(g =>
                new SourceControlStatistic { Key = g.Key, Count = g.Count() }).OrderByDescending(s => s.Count); 
        }

        private Changeset GetChangesetWithChanges(Changeset c)
        {
            return vcs.GetChangeset(c.ChangesetId, includeChanges: true, includeDownloadInfo: false);
        }

        private IEnumerable<Changeset> GetChangesetsForProject(string path, CancellationToken cancellationToken)
        {
            return vcs.QueryHistory(path, RecursionType.Full).TakeWhile(changeset => !cancellationToken.IsCancellationRequested);
        }

        private TfsTeamProjectCollection ConnectToTeamProjectCollection(string url, NetworkCredential networkCredential)
        {
            var teamProjectCollection = new TfsTeamProjectCollection(new Uri(url), networkCredential);
            teamProjectCollection.EnsureAuthenticated();
            return teamProjectCollection;
        }
    }
}

Having got this information, I usually write it out to a CSV file to analyse it offline. It can yield very interesting results. One one project I discovered several files that were being modified on average more than once a week over the lifetime of the project (10 years).

Using the TFS Warehouse Database

There is however a quicker, and potentially easier way to get at the churn statistics, and that is to go direct to the Tfs_Warehouse database yourself. This does mean you need admin rights to access the database. You also lose the ability to differentiate between a regular edit commit, and a merge commit (although you could use the API to get this information). However it provides counts of lines added, removed and modified, and runs much quicker. The inspiration for this technique came from the Code Churn Analyser CodePlex project (and updated to use what seems to be the newer schema in the Tfs_Warehouse database rather than the older TfsWarehouse). I’ve modified their SQL slightly as in our system, Changesets weren’t always connected to WorkItems. Here’s a SQL statement that grabs the code churn:

SELECT 
[FactCodeChurn].CodeChurnSK ChurnId,
DimFile.[FileName] [File]],
DimFile.FilePath [FilePath],
DimFile.FileExtension [FileExtension],
DimChangeset.ChangesetID ChangesetId,
DimChangeset.ChangesetTitle ChangesetTitle,
DimPerson.PersonSK PersonId,
DimPerson.Name PersonTitle,
[FactCodeChurn].LastUpdatedDateTime Date,
[FactCodeChurn].LinesAdded LinesAdded,
[FactCodeChurn].LinesModified LinesModified,
[FactCodeChurn].LinesDeleted LinesDeleted
FROM [FactCodeChurn]
JOIN DimChangeset on FactCodeChurn.ChangesetSK = DimChangeset.ChangesetSK
JOIN DimPerson on DimChangeset.CheckedInBySK = DimPerson.PersonSK
JOIN DimFile on FactCodeChurn.FilenameSK = DimFile.FileSK
WHERE DimFile.FileExtension = '.cs'    

I decided to use LINQPad to process this information, as it provides nice LINQ to SQL strongly typed objects that greatly speed up development. Once again I filtered out files I wasn’t interested in and used a regular expression to group together changes to the same file on a different branch. This one is able to count the files that have had the most different developers working on them. I use LINQPad’s “Dump” method to output the results of interest, but they could easily be output to a data file:

void Main()
{
    var churns = FactCodeChurns
        .Where (cc => cc.FilenameSKDimFile.FileExtension == ".cs")
    .Select(cc => new { File = Regex.Replace(cc.FilenameSKDimFile.FilePath , @"^.+/Source( Code)?/", ""),
        User = cc.DimChangeset.CheckedInBySKDimPerson.Name, 
        LinesAdded = cc.LinesAdded,
        LinesDeleted = cc.LinesDeleted,
        LinesModifided = cc.LinesModified
    });
    
    var changes = new Dictionary<string, Stats>();
    var users = new Dictionary<string, HashSet<string>>();
    foreach(var churn in churns)
    {
        Stats stats;
        if(!changes.TryGetValue(churn.File, out stats))
        {
            changes[churn.File] = stats = new Stats();
        }
        stats.Usages++;
        stats.LinesAdded += churn.LinesAdded.Value;
        stats.LinesDeleted += churn.LinesDeleted.Value;
        stats.LinesModified += churn.LinesModifided.Value;
        
        HashSet<string> fileUsers;
        if(!users.TryGetValue(churn.File, out fileUsers))
        {
            users[churn.File] = fileUsers = new HashSet<string>();
        }
        fileUsers.Add(churn.User);
    }
    
    changes.Where(kvp => kvp.Value.Usages > 200)
            .OrderByDescending(kvp => kvp.Value.Usages)
            .Select(kvp => new { kvp.Key, kvp.Value.Usages })
            .Dump("Usages");
    changes.Where(kvp => kvp.Value.LinesModified > 5000)
            .OrderByDescending(kvp => kvp.Value.LinesModified)
            .Select(kvp => new { kvp.Key, kvp.Value.LinesModified })
            .Dump("Lines Modified");
    changes.Where(kvp => kvp.Value.LinesAdded > 50000)
            .OrderByDescending(kvp => kvp.Value.LinesAdded)
            .Select(kvp => new { kvp.Key, kvp.Value.LinesAdded })
            .Dump("Lines Added");
    changes.Where(kvp => kvp.Value.LinesDeleted > 40000)
            .OrderByDescending(kvp => kvp.Value.LinesDeleted)
            .Select(kvp => new { kvp.Key, kvp.Value.LinesDeleted })
            .Dump("Lines Deleted");
            
    users.Where(kvp => kvp.Value.Count > 20)
        .OrderByDescending (kvp => kvp.Value.Count)
        .Select(kvp => new { kvp.Key, kvp.Value })
        .Dump("Users");    

}

class Stats
{
    public int Usages { get; set; }
    public int LinesAdded { get; set; }
    public int LinesDeleted { get; set; }
    public int LinesModified { get; set; }
}

As you can see from my code, I only dump the statistics above a certain threshold – when you’ve got tens of thousands of files and commits, you’ll want to do this.

Hope this proves useful to someone. I think Code Churn statistics are a very quick and effective way of highlighting some of the areas of code that may be suffering from too much “technical debt”.

Visualizing TFS Repositories with Gource

I recently came across the gource project, which creates stunning visualizations of your source control history. It doesn’t include built-in support for TFS repositories, but its custom log file format makes it quite simple to work with.

Since the project I worked on recently hit 1,000,000 lines of code (yes, I know that means it needs to be mercilessly refactored), I thought I would create a gource visualization. My example code below has a regular expression to allow me to filter out the bits in source control I am interested in. It also deals with the fact that we imported our solution from SourceSafe, so it needs to get the real commit dates for early items out of the comments.

public void CreateGourceLogFile(string outputFile)
{
    TfsTeamProjectCollection tpc = new TfsTeamProjectCollection(new Uri("http://mytfsserver:8080/"));
    tpc.EnsureAuthenticated();
    VersionControlServer vcs = tpc.GetService<VersionControlServer>();
    int latestChangesetId = vcs.GetLatestChangesetId();
    Regex regex = new Regex(sourceFileRegex); // optional - a regular expression to match source files you want to include in the gource visualisation
    Regex sourceSafeImportComment = new Regex(@"^\{\d\d/\d\d/\d\d\d\d \d\d:\d\d:\d\d\}");
    int lines = 0;
    using (var writer = new StreamWriter(outputFile))
    {
        for (int changesetId = 1; changesetId < latestChangesetId; changesetId++)
        {
            var changeset = vcs.GetChangeset(changesetId);
            var devEdits = from change in changeset.Changes
                           where
                           ((change.ChangeType & ChangeType.Edit) == ChangeType.Edit
                           || (change.ChangeType & ChangeType.Add) == ChangeType.Add
                           || (change.ChangeType & ChangeType.Delete) == ChangeType.Delete)
                           && regex.IsMatch(change.Item.ServerItem)
                           select change;
            foreach (var change in devEdits)
            {
                DateTime creationDate = changeset.CreationDate;
                var commentMatch = sourceSafeImportComment.Match(changeset.Comment);
                if (commentMatch.Success)
                {
                    creationDate = DateTime.ParseExact(commentMatch.Value,"{dd/MM/yyyy HH:mm:ss}", CultureInfo.InvariantCulture);
                }

                int unixTime = (int)(creationDate - new DateTime(1970, 1, 1)).TotalSeconds;
                writer.WriteLine("{0}|{1}|{2}|{3}", unixTime, changeset.Committer,
                    GetChangeType(change.ChangeType), change.Item.ServerItem);
            }
        }
    }
}

private string GetChangeType(ChangeType changeType)
{
    if ((changeType & ChangeType.Edit) == ChangeType.Edit)
    {
        return "M";
    }
    if ((changeType & ChangeType.Add) == ChangeType.Add)
    {
        return "A";
    }
    if ((changeType & ChangeType.Delete) == ChangeType.Delete)
    {
        return "D";
    }
    throw new ArgumentException("Unsupported change type");
}

Once you have your log file created, just run gource to see an amazing replay of the history of your project:

gource custom.log

Here’s a screenshot:

How to Add a Custom Work Item to a Category In TFS

I recently had to make a custom work item type using the TFS 2010 Power Tools. However, the requirements search in Test Center didn’t return any results for work items of this new type. The problem was, TFS has a concept of work item categories (or “groups”), and my new work item type needed to be in the “requirements” group.

To add a new work item type to a work item category you have to use witadmin. Open a VS2010 command prompt and get the categories like this:

C:\Users\Administrator\Documents>witadmin exportcategories /collection:http://win-gs9gmujits8:8080/tfs/defaultcollection /p:"My Project Name" /f:categories.xml

Then this MSDN article explains the syntax of the categories xml file. I added my custom work item type in like this:

<CATEGORY refname="Microsoft.RequirementCategory" name="Requirement Category">
  <DEFAULTWORKITEMTYPE name="User Story" />
  <WORKITEMTYPE name="My Custom Work Item Type" />
</CATEGORY>

Then you reimport them using the witadmin tool again:

C:\Users\Administrator\Documents>witadmin importcategories /collection:http://win-gs9gmujits8:8080/tfs/defaultcollection /p:"My Project Name" /f:categories.xml

And now Testing Center is able to assign the new work item types as requirements.

How to use the TFS API to retrieve a historical version of your project

I recently had the need to retrieve several old versions of a project from TFS. Fortunately, the TFS API allows you to automate this process.

First you need to connect to the TFS Server:

TfsTeamProjectCollection tpc = new TfsTeamProjectCollection(new Uri("http://tfsserver:8080/"));
tpc.EnsureAuthenticated();

Next we will set up a workspace to use. I check first that the workspace doesn’t exist.

VersionControlServer vcs = tpc.GetService<VersionControlServer>(); //.GetAllTeamProjects(false);
string workspaceName = "temp";
string workspaceOwner = "mheath"; // your user name
// protect ourselves from using a workspace in use for something else\
Assert.Throws<WorkspaceNotFoundException>(() => vcs.GetWorkspace(workspaceName, workspaceOwner), "The workspace already exists");
var workspace = vcs.CreateWorkspace(workspaceName, workspaceOwner);

// use the workspace...

// when we are done with it... 
vcs.DeleteWorkspace(workspaceName, workspaceOwner);

Now we need to map our workspace to a local temporary folder. We can use the server path to select only a subset of the code in the project (for example a single branch):

string localPath = @"D:\TFS\Temp";
string requestPath = "$/MyProject/Dev/Source";

// protect ourselves from accidentally overwriting something important
Assert.IsFalse(Directory.Exists(localPath), "directory should not exist before this test is run");
Directory.CreateDirectory(localPath);

workspace.Map(requestPath, localPath);

Finally we are ready to make a request. We use a VersionSpec to specify which version we want. I am using a DateVersionSpec here, but you can use a ChangeVersionSet or a LabelVersionSet if you prefer. The ItemSpec specifies what to get. We want everything in our request path including subfolders, so recursion is turned on:

var version = new DateVersionSpec(new DateTime(2009, 11, 24));
var item = new ItemSpec(requestPath, RecursionType.Full);
GetRequest getRequest = new GetRequest(item, version);

GetStatus status = workspace.Get(getRequest, GetOptions.GetAll);
Assert.AreEqual(0, status.NumWarnings);
Assert.AreEqual(0, status.NumFailures);

And that is it – it is usually quite quick to retrieve as well (certainly much faster than SourceSafe was!).

Why you should use DVCS for Personal Projects

For a long time it bugged me that there wasn’t an easy way for me to use version control for my personal projects. I have over 100 small applications sitting around on my computer. Most of them are just test apps that will probably never be visited again. But others are more useful utilities, or perhaps future open source projects currently in incubation. Some of them are work related, others purely for personal enjoyment or learning. Often I found myself wishing that I could have the benefits of source control, so I could back out of changes that broke something.

Pre Distributed Version Control Systems

In the days before DVCS (or, to be more accurate, before I knew about DVCS), at different times, I tried all the following approaches:
  
Store projects on my company’s VCS. This usually involved asking permission to have some space on SourceSafe or TFS for my personal projects. Whilst this has the benefit of meaning that I can share my work with others easily (and it is guaranteed to be backed up), there is the hassle of getting this set up in the first place, plus the fact that some of these projects are very shortlived, while others are “skunkworks” ideas which you don’t want to give publicity to until they are ready for it.

Run a private VCS server on my dev machine. It is possible to install Subversion, SourceSafe, TFS etc servers on your local machine, and use them for VCS. However, as well as using up valuable resources on your personal machine, it has a very poor migration story. If you rebuild your PC, need to quickly copy code onto a USB stick and work on it from home, this option ends up being more hassle than it is worth.

Subversion file-based repository. A few years ago I discovered that you could get Subversion to back up to a file:// path. I thought this would be the answer to all my issues. The reality is, that it was quite fragile, especially since I was storing code on USB sticks, so the drive letter might change. I ended up corrupting my repositories so regularly I gave up on this option pretty quickly.

Make it open source. When CodePlex showed up, I immediately moved several of my projects there. This meant I had access to a free central repository, enabling me to work from different computers if I needed to. The downside is that most of my projects weren’t appropriate for making into open source applications.

Don’t bother with version control and make backup zip files. This ended up being my most common approach. Every now and then I would backup to a zip file. Of course, those backups are few and far between and don’t even exist for most projects. And I almost never had a backup available on those few occasions when I genuinely needed one.

Advantages of DVCS

But all that has changed after I decided to find out what all the fuss was about with Distributed Version Control systems. Whilst the idea seemed a little crazy to me at first (everyone gets a copy of the entire repostory?), the obvious advantages for personal projects won me over pretty quickly. I decided to try out Mercurial, since it seemed to have slightly better support on Windows, although I’m sure Git is just as good. Here’s some of the top advantages to using it on your personal projects:

No Server Required. This is a huge benefit. I don’t need a central server on the internet, or on my company network. If I want to move to another PC, I can just copy the code folder over (or Sync folders using something like DropBox) and it just works.

Version Control Everything. It’s now a no-brainer to put a new test project under version control. It takes only a few seconds to do. If for some reason you decide you don’t want version control anymore, just delete the .hg folder and its gone.

Migrate to a Central Repository Later. When I added NAudio to Codeplex, it already had been in development for several years. However, I have no checkin history up to that point, just a bunch of backup zip files. With Mercurial, you can move to using a centralised repository at any point in the future (whether public or private) and all your checkin history comes along for the ride.

Unconstrained branching strategy. Admittedly, for small personal projects, branching is not often that important. But it can come in handy when are half way through implementing one feature, and then want to work on a different task. Without version control, you have to decide whether to bin the half-finished changes, or to copy them somewhere else and manually merge them back in later. With Mercurial, it becomes trivial to create as many branches as you need. And you can merge directly between any two branches, irrespective of how many intermediate branches were created between them.

Merging divergent copies. Sometimes I have a copy of a personal project at home and at work and have no idea which one is the latest and greatest. Or maybe after backing it up to a few places, I have inadvertently made changes to two separate copies. One membership application I wrote for a youth group 8 years ago turns out to be still in use and they asked me for new features recently. I had to work out which of several copies was the one I should be using. With Mercurial, it is trivial to ensure that one copy is not missing any changes in another.

Little and often checkins. One really nice feature for my open source projects is that I can check in little changes without needing to immediately push them to the central server. This means I can check in little and often, and only do a push to the server once I have tested and made sure my feature is robust.

What was I doing?. Another advantage is that by using a DVCS with my personal projects, if I need to come back to one after a couple of years I can quickly examine the log, looking at diffs to see what I was up to last time I worked on it. This can be handy if I had left it in a state where there were some half-finished new features in progress, and actually I want to discard them and resume from an earlier point.

Trivial rollback – one of the things that scared me about DVCS was the idea that once I had checked in a file, it lives on in the repository forever. So accidentally checkin several megabytes of compiled binaries and you have an unnecessarily bloated repository. But the reality is that issue only exists if you push that to a central repository (and even then there are usually ways of working round the issue). If you haven’t you can just clone to the prior revision and your mistake is gone. I’ll perhaps do a post later on my thoughts about DVCS in the enterprise, where matters like this are quite important.

If you take anything away from this post, it is learn a DVCS and use it wherever you can. You will be glad you did.

Merge-Friendly Code

If you are working a large application where you have to simultaneously support hotfixes, service packs and new features for customers running several different versions, while developing new versions of your software, chances are you have some kind of branching structure in your source control system.

And before too long, you will experience the joy of merging features from one branch into another. Here’s my top six tips for writing code that is easy to merge…

1. Little and Often

The first principle is that it is better to make many small, focused check-ins, and merge them early, rather than checking a vast collection of changes in one hit and attempting a gigantic merge after several months of development. Sadly, this is not always possible, as sometimes a major change has to be kept out of a branch for a long time.

2. Get Your Branch Strategy Right

It is worth spending some time making sure you choose the correct branching strategy. A lot could be said about this, but here’s just two things to consider.

Don’t put two major features into one branch unless you are willing to deploy them together. Although you can “cherry-pick” changesets to merge, the reality is that this only works if your changes are completely isolated from one another. In other words if changeset B depends on something that was in changeset A, then you can’t just merge changeset B into another branch, changeset A has to come along for the ride too.

Also, avoid getting into the situation where you need to merge between two branches that are not directly related. In TFS, these are called “baseless merges”, and they can result in deleted code getting re-inserted.

3. Check in Clean Code

Nothing messes up a merge more quickly than someone making widespread “cosmetic” changes – introducing or deleting whitespace, renaming things, moving methods to a different part of the file, etc. Sweeping changes like this have a high probability of conflicting with someone else’s change.

The solution is of course, to reduce the need for this kind of change by making sure that what you check in is formatted correctly, and follows the appropriate coding standards and naming conventions. Tools like StyleCop, Resharper, and FxCop are all able to help here.

4. Single Responsibility Principle (SRP)

Simply put, the Single Responsibility Principle dictates that every class should have one and only one responsibility. If it has two or more, you should extract functionality into additional classes. Similarly every method should perform one and only one task.

Adhering to this straightforward principle results in many classes, each composed of short methods. Very often merge conflicts are due to two people working on the same file or method, but changing it for very different reasons. But if a class or method has only “one reason to change”, then the chances of two developers working on different features needing to simultaneously change it are greatly reduced.

5. Open Closed Principle (OCP)

The Open Closed Principle states that classes should be open for extension but closed for modification. Or to say it another way, it should be possible to add new features and capabilities to your codebase simply by creating new classes, rather than having to mess with the internals of existing classes. And if you use technologies like MEF, it really is possible to add whole new features without touching a single line of your existing codebase.

Obviously, in any large real-world application, there will always be the need to make some changes to legacy code. But this should be the exception rather than the norm. In fact, the only real reasons to change the existing code are to fix bugs, and to make it more extensible.

If commits to source control of new features consist mostly of changes to existing files rather than the creation of new ones, maybe you need to do some research into how you can better design your classes to conform to OCP.

6. Unit Tests

After performing a merge, it is vital that you are able to determine as quickly as possible if you made any mistakes in the conflict resolution process. A unit test suite with good code coverage is invaluable in helping with this task. Obviously this may need to be complemented with further integration testing and manual testing, but the quicker you can identify problems with a merge, the better.

So those are my top recommendations for avoiding merging nightmares. Anyone got any other suggestions?

Using TFS to find what files a user has got checked out

Posting this as a reminder to self of how to do it…

tf.exe status /user:username

Can also add a "/s:" parameter to specify a TFS server, but seems to default to the right one for me.