We recently wanted to add a manual verification to our TFS builds. We didn’t want someone accidentally clicking the build to production when they meant to build to development. They are right next to one another and look very similar.

I didn’t want to spend a lot of time on it, so I came up with this. It’ll work for now and ensure we manually interact with the queue build dialog.

<Target Name="BeforeEndToEndIteration" Condition="'$(C***)' != 'suck'" >
    <Error Text=
"You did not state your feelings for the C***. Please state this by submitting
 the parameter /p:C***='suck' in the queue build dialog."/>
</Target>

The parameter name realy isn’t “C***”, but I didn’t want to offend any followers of the team with the longest championship drought of any professional sports team. Actually, I bet it is safe to extend that to individual performers too.

In the past few days, and nights, I’ve learned a few things. Most are bummers, but there is light at the end of the tunnel. This is what I feel our build process was like for the past few days, but I think we’re doing much better now and things are actually working. (Cross your fingers)

I’ve tried a few approaches yesterday and here is what I’ve found.

Deploying via Ftp

Initially, I tried to deploy to our servers via FTP. This appears to work well, but the timestamps are all adjusted by the GMT offset. In our case, the files end up on the server looking like they were edited 6 hours in the future. This makes every single file appear different on the next sync since none of the time stamps match. Apparently this is by design from Microsoft. What a bummer. Running the development server on GMT will cause the timestamps in the database on that server to also be off. This option is out.

Deploying via file share

Again, this appears to work well. The first time everything goes. We change a single file and the log sees that only one file has changed, but doesn’t move it. Doh! They come out of TFS as read-only, and copying to a file share will carry over those file attributes. No problem.  

attrib -r \\myServer\myShare\myApp\*.*

Nope! You can’t run attrib on a network share. It has to be a physical disk.

OK. Well I’ll just attrib -r the source files in the build’s working directory. Great! That worked. Let’s try it again. Doh!

Unable to perform the get operation because it is writable.

You’re kidding me! Grr…OK! Fine! I’ll attrib +r after I push. Nope! Not every file in the source directory is suppose to be read only and some of them cannot be read only. AHHH!

Oh yes! Beyond Compare has an attrib -r command in the scripting reference.

ATTRIB
Usage: attrib (+|-) [(+|-) <...>]

Sets (+) or clears (-) the DOS file attributes in the current selection. An attribute set can include archive (a), system (s), hidden (h), and read only (r) attributes. Windows only implementation: Linux version will not recognize ATTRIB.

Sound great right?….that doesn’t work. Even on simple unit test this doesn’t work. I’m at my wit’s end now. There is one more thing to try. There is one more command in the Beyond Compare script reference that might be able to help.

OPTION
Usage:
option stop-on-error
option confirm:(prompt|yes-to-all|no-to-all)

Adjusts script processing options.
· stop-on-error makes the script watch for various error conditions, including file operation errors, and, when one occurs, prompts the user before continuing.
· confirm can use prompt, yes-to-all, or no-to-all to handle confirmation dialogs that occur due to file operations. By default, prompt is used.

Let’s hope this works. I’ll put it right before the sync, cross our fingers and run it through some tests.

It WORKS! Or it appears to so far. Hopefully.

The files on the server still carry the read only flag. That’s not ideal for us, but it will have to do for now.

The past few days have been spent getting Selenium setup and running as part of our CI builds. We only have a few proof-of-concept tests right now, but the power and potential is apparent. We have no automated unit testing right now and very little code in our code behinds so functional testing is as good as we’re going to get. Also, with the amount of AJAX that is currently in our pages, and the additional amount we see in them in the near future, Selenium’s method of control the browser helps us make sure our pages do and continue to behave like we expect.

Getting setup with Selenium was incredibly simple. The Selenium IDE Firefox plugin makes creating your first Selenium test a snap. Download it and install it and you’re off. I’m not going to walk you through creating the tests, because the Selenium web site has a pretty decent movie that’ll take care of you.

The output of the test recorder are HTML files that contain the instructions, aka “Selenese”, to re-execute the tests through both the IDE and Selenium core. Selenese is very simple by design. While we could run the Selenese tests through the core in our Continuous Integration server, I’ve decided to use the IDE to export the C# source for the tests I create. This allows my to put the Selenium test into an NUnit test harness, refactor some of the common setup and config, and also run the tests via Selenium Remote Control(RC). I know the refactoring goes against the Selenese simplicity, but I want to run my tests locally and with different credentials without changing every test. Also, running in RC lets me run the tests without having to watch the pages in the browser fly by and, someone day, concurrently run all the browsers I care about.

Here’s an example of what one of those tests might look like 

using System;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading;
using NUnit.Framework;
using Selenium;

namespace SeleniumTests{

   [TestFixture] public class NewTest {
	private ISelenium selenium;
	private StringBuilder verificationErrors;

	[SetUp] public void SetupTest() {
		selenium = new DefaultSelenium("localhost", 4444, "*chrome", "http://www.google.com/");
		selenium.Start();
		verificationErrors = new StringBuilder();
	}

	[TearDown] public void TeardownTest() {
		try {
			selenium.Stop();
		} catch (Exception) {
			// Ignore errors if unable to close the browser
		}
		Assert.AreEqual("", verificationErrors.ToString());
	}

	[Test] public void TheNewTest() {
		selenium.Open("/");
		selenium.Type("q", "fooberry");
		selenium.Click("btnG");
		selenium.WaitForPageToLoad("30000");
	      Assert.IsTrue(selenium.IsElementPresent("css=a[href=\"http://www.fooberry.com/\"]"));
	}
   }
}

Getting this to run in the Team Build is as easy as getting the regular NUnit tests running. First we would want to parameterize those tests a bit so we can pass different hosts, ports and credentials (if present). That’s not too hard and should be obvious.

We’re trying multiple CI builds. One build just compiles and runs the unit test code, which is currently this single test….baby steps.

[TestFixture] public class TheMostAwesomestUnitTestInTheWorld {
    [Test] public void ValidateOurCoreBeliefs() {
        Assert.That(true, Is.True);
    }
}

The other build compiles and runs both the unit tests and integration tests. We suspect this build will take a while and the sooner we could get feedback about the quality of the build, the better. We could push this build to a separate build agent too, but for right now they are all just backing up in the same queue.

The only missing piece of the puzzle so far is the Remote Control Server. The RC Server is a Java application that has to run on a machine with access to which ever browser you plan to test. Starting the server is simple enough. Just make sure Firefox.exe is in the path. If you forget, you’ll get a nice warning about it.

java -jar selenium-server.jar

It’s up an running and when you run the above unit test, you see the browser window fly by and the server log all the actions it’s performing on the browser.

Great! …if you want to stay logged into your server with the console open, but I didn’t. Luckily, creating a Windows service is pretty easy. There is a great tutorial out there that walks you through how to setup a similar Java application as a service. Just substitute the Selenium RC Server for the FitNesse server. Note that when I tried to install the Windows Server 2003 Resource Kit, on both Vista and Windows Server 2008, I got a compatibility warning. I ignored the warning since I was only using the two files and it worked great.

Now you can start and stop the Selenium RC Server via a Windows service, keep it running when you log out and start automatically when the server starts up.

Have fun.

The TFS diff tool is horrible. It a pretty bare minimum difference between the two versions. The UI is basic.

Nice huh? This line changed, but you’re on your own to know what characters actually changed. While tooling around in the Visual Studio Options dialog I can across the settings that would let me change the diff and merge tools.Check out Tools | Options | Source Control | Visual Studio Team Foundation | Configure User Tools…

I needed a another diff tool, so I downloaded a trial of Beyond Compare and plugged it in there and it worked great! Except, in 15 days it wouldn’t be working so great unless I could convince someone to shell out the $X to get us past the trial. After pinging some of my buddies about what diff tools they use, I decided to try the diff tool in TortoiseSVN. I installed it and added TortoiseMerge to the dialog.

Ran my compare again and Bang! A new diff dialog appears that is much better than before.

That is still a lot of redundant clutter for me, so switching views helps a little with that.

We no longer get the detail about what changed on a line, but it is so much easier to manually identify when the rows overlap, I’m not missing it.

With a understanding of the basic concept of branching, we can cover the branching strategies my team uses and the level of success we’ve been able to achieve with each.

 

Never Branch

Honestly, most of our applications do not branch at all. Most of the applications our team maintains have were developed in Visual Basic 6 and stored in Visual Source Safe repositories a decade ago. We do very little maintenance development on them and no new development. We receive a bug, patch and deploy one or two bugs at a time so there is little need for extra overhead created by branching.

Branch for Maintenance

While we have no applications that follow this method, I feel compelled to mention the simplest method of creating isolation via branching. While continuing development down a single path, as we do with our VB6 application, a large development effort maybe underway and the need to patch the existing version arises. In this case we could create a branch from the label of the previous release, patch, release and merge the patch back to the main path. This was the method of branching mentioned in the previous post.

It is commonly called “Branch for Maintenance” because you branch at the time you need to apply maintenance to and existing piece of software. The reasons we don’t use this method on our team will be highlighted.

Branch for Version

On my current project we have our main source tree or “trunk“; however, we don’t develop against it. Our project has a predictable release schedule with a known set of bugs and enhancements that are to be included in each of these releases. We create a branch for each version we plan to release and commit all changes to that branch. This includes branching for any service pack version(s) we plan on releasing.

The branch allows us to concurrently work on multiple future versions and service packs. This is very valuable during the end of iteration when the app can spend several days, or even weeks, in user acceptance testing. During this time, while the app under user acceptance testing is frozen for new development, the next release can be under heavy development in its own branch. If issues during user acceptance need to be resolved, those patches can be applied to the tip of that version’s branch, and the next version is unaffected until the merge occurs.

We’ve found that as the date to branch the next version approaches, it pays to start merging back to the trunk. This will let the future branch start closer to its previous version. The closer the versions are at the start the fewer merges happen once the final merge is completed.

Some bugs cannot wait for the next release schedule. These bugs get their own service pack release and their own branch. Instead of being branched form the trunk, which may contain changes from other versions, the branch is created from the tip of the released version’s branch. This ensures that the start of the service pack branch is the exact source that is running in production. Once the patch is created, committed and released, we merge those changes back to the trunk and back down to any branches for future versions that might need the bug fix as well.

In addition to working from the branch, we create a build for each branch that will deploy the tip of the branch to the our development and testing environments. This is valuable in that the version in each environment can be switched with ease. We run the build for the version we need, it builds it from its branch and moves it to the correct servers. We can also take our servers back to previous versions if we want to recreate a production bug in a more transparent environment.

The benefits of this branching strategy outweighs the overhead created by the resulting merges. In most cases our merges are very straight foreword.

Branch for Feature

The last strategy we’ve only discussed using on our team, and actually haven’t implemented yet. Before we get into the actual method, an overview of the situation’s issues may help clarify our intended benefits.

One of our applications has several features and bugs being worked on at the same time by several people. In order to get the features and bug fixes to the users as quickly as possible the release schedule depends heavily on when the bugs can be fixed first. Once a high priority feature is completed, the app needs to go production. This means other feature and/or bug fixes may, or may not get included in the next release. Also, on some occasions a production issue comes in and needs to be fixed right away, before anything else goes to production.

As you can imagine, this is a nightmare to keep straight. There were two options for keeping unwanted code from making its way to production. Just as without branching, we could open up the code and comment out the code that can’t go, and then immediately uncomment it once the application releases. What we found was people were afraid to commit any changes to source control until their feature or bug fix was complete to avoid that nightmare. It could take a couple weeks to complete their feature and this means a couple weeks between commits of the code. Again, not something we desire. We want to integrate the changes sooner than a couple weeks.

What we are trying now is to create branches for each of the features/patches under development. Once they are complete we merge the branch back to the trunk. We then build and release from the trunk. This method allows us to pick which features that will be included in the next version. It lets us maintain history as the features develop since commits to the feature’s branch are isolated from other features.

This method is only valuable because of the unpredictable schedule and chaotic combination of features included in each release. It does create more overhead than branch for version since there can potentially be numerous branches all being developed concurrently. This means more merges and potentially more conflicts.

Branch Housekeeping

Source Structure

We struggled with where to store all these branches. We’ve settled on storing them closer to the trunks than we did initially. Here is an example of what our source tree currently looks like:

/MyApp/main/docs
           /libs
           /release.scripts
           /source/core
                  /core.unitTest
                  /domain
                  /domain.unitTests
                  /...etc
           /...etc
      /1.0/docs
          /libs
          /...etc
      /1.1/docs
          /libs
          /...etc
      /2.0/docs
          /libs
          /...etc

The directory libs is a directory to store our third party and external libraries used by the application. This directory is branched along with the source. This allows for different releases to use different versions of the same dependencies. The ultimate goal is to have everything needed to build the application under the main or branch directory. This includes all documentation and all scripts needed to setup the database. If we were really cool we would be able to have the build scripts setup the database based on the scripts in its branch, but we’re not that cool.

Building the Branch

To build each branch in the Branch for Version we create a build that we can instigate either by check in or manually. Be careful; if the build actually deploys to the build servers, only one build should be instigated by a check in. You wouldn’t want the version on the server changing depending on what was the last branch to get a commit. That’s not to say the build can’t happen to validate the tests still pass, but just don’t deploy to the server.

In our build tool you would see builds like:

• MyApp – 1.0
• MyApp – 1.1
• MyApp – 2.0

It works really well. With a button click we can put the next version on the development servers to illicit feedback on features in development and turn around and replace it with the version under user acceptance testing.

A friend at work asked me to describe the branching strategies we use on our team. Our organization is moving to Team Foundation Server from Visual Source Safe and people are starting to have a lot more confidence in branching their code. First, I wanted to go over the basics of branching and why working without isolation can, and have, caused so many headaches.

The most basic utilization of source control is to commit your changes and, hopefully, label the project for each release to your customers. You gain both the history of how the source has changed overtime and configuration information about how the source looked at the time of each release.

For applications with no new development, only service packs or maintenance releases, this works fine. There is little need to have concurrent development and therefore isolation of that development. Where this method becomes problematic is when deep into new development of the next version, you need to fix a bug in the version(s) you’ve already released.

If you don’t have the labels you’ll have a tough time resurrecting the source that was built in the previous release(s). Assuming you can identify the source in the previous build, you have a few options on how to apply the patch. If you are patching the most recent version you could comment out all the new features that aren’t ready to go to production, apply the patch to the main source, label and release the service pack. Then you would uncomment out the code and keep going on new development. This could be a large amount of tedious work. If the deltas that need to roll back are small this might not be a bad option, but in most cases it is just a headache.

The next simplest alternative is to rollback the source the to release label, copy the entire source tree, patch, label, and release. This does provide the isolation we desire and allows new development to continue without impact from the service pack, but now you are presented with a new challenge. You must get the changes from the service pack back to the main source. This will be very manual process where you will look at the files in each source tree, determine which has been changed, and apply the deltas to the main source tree. Not as bad as the other method, but it would be nice if we could maintain that relationship and get some help with bringing our changes back to the main source.

Enter branching. Branching appears to do exactly what we did when we copied the source. It does some fancy magic under the covers, but we can ignore that for now. It will create an isolated “branch” of the source that we can commit changes to free of worries of introducing new development to the service pack, or service pack code to the new development until it is complete.

Branches maintain that logical relationship which provide a huge advantage when it comes time to merge our changes back. At the time of the merge our tools can look at the histories of both the source and target branch and determine if code is being added, removed or changed. It will thusly apply the changes, but in some cases changes will exist to the same part of code in both places. This needs to be resolved manually.

While the merge is usually very smart, especially if the branch hasn’t diverted very far from the source, this doesn’t mean the source is free from errors. It is possible for one class to change in the branch and the other to change in the trunk and when merged back, no conflicting changes are found, but the classes don’t cooperate like that should.

The merge is more than a one way operation. In addition to merging from the branch back to the source of the branch, changes can be merge to the branch. This is useful if a bug fix was already patched in the new development branch and needs to be included in a service pack release.

That covers the very basics of branching and merging. I’ll be back in a bit to explain the strategies our team implements.