Step 4 Add Coverage¶
Now we have our code tested, we want to know how much of the code is tested. It's better to have 10 tests that go through 50% of the code, than having 100 tests that just test 1 method.
This is where code coverage comes in. There are a few code covering frameworks, the best known is NCover, of which the latest versions are not free anymore :-(.
Since the previous scenario steps are with VS2010 and MSTest, I'll use the coverage of MS-test.
Wikipedia article on Code Coverage.
Turning on Code Coverage¶
When setting up the unit tests in VS.NET, a localtestrun.testrunconfig file will be created and added as part of the solution. Double-click this file and find the option Code Coverage option on the left of the dialog. Select the assemblies for which you want to collect code coverage information and then re-run the unit tests. Code coverage information will be collected and is available. To get the code coverage information open the test results window and click on the code coverage results button, which will open an alternative window with the results.
I just made a company rule that for code coverage to be ran via ccnet, the testsettings file must be named : CodeCoverage.testsettings,
with a specific base name(cover_me) and no timestamps appended. Just to make things easier for me. If you want MS-Test to run coverage, just pass the testsettings as an extra argument :
1<exec program="${mstest_exe}" >
2 <arg value="/testmetadata:${mstest_metadatafile}" />
3 <arg value="/resultsfile:${mstest_unit_test_file}" />
4 <arg value="/testlist:UnitTests" />
5 <arg value="/testsettings:CodeCoverage.testsettings" />
6</exec>
There is a catch : Ms-Test from VS2010 does not produce XML anymore, see this post for a solution. You really need the dll from VS2008 for it to work, the VS2010 has another interface sadly enough. So best to digg up your DVD of VS2008. Below is an updated version of the program, which also removes the Lines elements from the coverage result file, making it a lot smaller to merge (size reduction up to 66%). This updated version fails the build if a certain module/assembly is below the threshold. Should you do not want the build to break, just enter 0 for the threshold :-)
Putting it all together¶
Now we have all the needed info, so let's start with setting it all up. Do we want to run code coverage in the CI build or not? In my opinion this is on the edge. To keep this setup as clean as possible, we'll add another CCNet-project for this solution : a QA one. QA stands for Quality Assurance.
Tip it's best to instruct the build to break if coverage goes below a certain threshold, otherwise there is no real point in doing coverage at all. Also keep a pragmatic approach, demanding 100% coverage is extreme. Example : how does one test network outage? You'll probably have some error handling code, that you tested once or a couple of times by pulling out the network cable. But I've not seen a way to test these kind of scenario's automated together with other tests.
The new project definition in CCNet.config¶
1 <cb:scope ProjectName="ProjectX">
2 <cb:define ProjectType="_QA" />
3 <project name="$(ProjectName)$(ProjectType)" queue="Q1" queuePriority="801">
4 <workingDirectory>$(WorkingDir)$(ProjectName)$(ProjectType)</workingDirectory>
5 <artifactDirectory>$(WorkingMainDir)$(ProjectName)$(ProjectType)$(ArtifactsDir)</artifactDirectory>
6
7 <scheduleTrigger time="06:30" buildCondition="ForceBuild" name="QA_Scheduled">
8
9 <labeller type="defaultlabeller" />
10
11 <sourcecontrol type="vsts">
12 <workspace>$(ProjectName)</workspace>
13 <project>$/$(ProjectName)/Main</project>
14 <cb:vsts_ci/>
15 </sourcecontrol>
16
17 <tasks>
18 <nant>
19 <cb:nant_common/>
20 <cb:nant_target_qa />
21 </nant>
22 </tasks>
23
24 <publishers>
25 <cb:common_publishers />
26 </publishers>
27
28 </project>
29 </cb:scope>
I usually schedule this in the morning, so when you enter the office, you know that the QA build is broken or not. If broken you know what to do :-)
As you see, there is a new Nant target : nant_target_qa
This has the following layout :
1<cb:define name="nant_target_qa">
2 <targetList>
3 <target>clean</target>
4 <target>compile</target>
5 <target>cover</target>
6 </targetList>
7</cb:define>
The Build Script¶
This involves a bit more than just putting the 6 mentioned lines above into a Nant target. Reason : if any test fails, we still need to process that coverage file into xml, so it an be merged. If we do not do this, we would only have coverage info when all tests are ok. This is how we do this in NAnt :- do not let the tests fail the build :
failonerror="false" - store the result of the tests in a property :
resultproperty="testresult.temp" - proces the coverage file :
exec - if the tests failed, fail the build :
fail
The cover target¶
1<target name = "cover" description="runs the tests with coverage" >
2<!--
3 company rule : code coverage settings must be set via the file CodeCoverage.testsettings
4 with the following NamingScheme : baseName="cover_me" appendTimeStamp="false" useDefault="false"
5-->
6 <if test="${file::exists('CodeCoverage.testsettings')}">
7
8 <exec program="${mstest_exe}" failonerror="false" resultproperty="testresult.temp" >
9 <arg value="/testmetadata:${mstest_metadatafile}" />
10 <arg value="/resultsfile:${mstest_unit_test_file}" />
11 <arg value="/testsettings:CodeCoverage.testsettings" />
12 <arg value="/testlist:UnitTests" />
13 </exec>
14
15 <property name="TestsOK" value="false" unless="${int::parse(testresult.temp)==0}"/>
16
17 <property name="DataCoverageFilePath" value="${RuWi::FindFile('cover_me','data.coverage')}" />
18 <property name="TurnCoverageFileIntoXml_exe" value="C:\Tools\TurnCoverageFileIntoXml\TurnCoverageFileIntoXml.exe" />
19 <property name="MS_Test_CoverageFile" value="MSTestCover.xml" />
20 <property name="WantedCoveragePercent" value="75" />
21
22 <fail message="No data.coverage found in cover_me folder" unless="${string::get-length(DataCoverageFilePath)>0}" />
23
24 <echo message="DataCoverageFilePath : ${DataCoverageFilePath}" />
25
26 <exec program="${TurnCoverageFileIntoXml_exe}" >
27 <arg value="${DataCoverageFilePath}" />
28 <arg value="cover_me\Out" />
29 <arg value="${MS_Test_CoverageFile}" />
30 <arg value="${WantedCoveragePercent}" />
31 </exec>
32
33 <fail message="Failures reported in unit tests." unless="${TestsOK}" />
34 </if>
35</target>
Code section¶
CCNet.config¶
1 <cruisecontrol xmlns:cb="urn:ccnet.config.builder">
2 <!-- preprocessor settings -->
3 <cb:define WorkingDir="D:\WorkingFolders\" />
4 <cb:define WorkingMainDir="D:\ArtifactFolders\" />
5 <cb:define ArtifactsDir="\Artifacts" />
6
7 <!-- TFS settings for a CI build -->
8 <cb:define name="vsts_ci">
9 <server>http://tfs-server:8080/tfs/default/</server>
10 <username>cruise</username>
11 <password>**********</password>
12 <domain>tfs-server</domain>
13 <autoGetSource>true</autoGetSource>
14 <cleanCopy>true</cleanCopy>
15 <force>true</force>
16 <deleteWorkspace>true</deleteWorkspace>
17 </cb:define>
18
19 <!-- all projects will have at least these publishers -->
20 <cb:define name="common_publishers">
21 <merge>
22 <files>
23 <file>MStest_UnitTestResults.xml</file>
24 </files>
25 </merge>
26 <xmllogger />
27 <statistics />
28 <modificationHistory onlyLogWhenChangesFound="true" />
29 <artifactcleanup cleanUpMethod="KeepLastXSubDirs" cleanUpValue="2" />
30 <artifactcleanup cleanUpMethod="KeepLastXBuilds" cleanUpValue="25000" />
31 <!-- email the breakers on a broken build and when it's fixed -->
32 <email from="CruiseControl@TheBuilder.com"
33 mailhost="TheMailer.Company.com"
34 includeDetails="TRUE">
35 <groups/>
36 <users/>
37 <converters>
38 <ldapConverter domainName="Company" />
39 </converters>
40 <modifierNotificationTypes>
41 <NotificationType>Failed</NotificationType>
42 <NotificationType>Fixed</NotificationType>
43 </modifierNotificationTypes>
44 </email>
45 </cb:define>
46
47 <!-- the Nant build arguments for a CI build -->
48 <cb:define name="nant_CI">
49 <executable>c:\Tools\nant\bin\nant.exe</executable>
50 <nologo>true</nologo>
51 <buildTimeoutSeconds>1800</buildTimeoutSeconds>
52 <buildArgs>-D:useExtraMsbuildLogger=true -D:isCI=true -listener:CCNetListener,CCNetListener -D:configuration=Debug</buildArgs>
53 </cb:define>
54
55 <!-- the nant targets for a CI build -->
56 <cb:define name="nant_target_CI">
57 <targetList>
58 <target>clean</target>
59 <target>compile</target>
60 <target>unit_test</target>
61 </targetList>
62 </cb:define>
63
64<!-- the nant targets for a QA build -->
65 <cb:define name="nant_target_qa">
66 <targetList>
67 <target>clean</target>
68 <target>compile</target>
69 <target>cover</target>
70 </targetList>
71 </cb:define>
72
73 <!-- end preprocessor settings -->
74
75 <!-- Projects -->
76 <cb:scope ProjectName="ProjectX">
77 <cb:define ProjectType="_CI" />
78 <project name="$(ProjectName)$(ProjectType)" queue="Q1" queuePriority="901">
79 <workingDirectory>$(WorkingDir)$(ProjectName)$(ProjectType)</workingDirectory>
80 <artifactDirectory>$(WorkingMainDir)$(ProjectName)$(ProjectType)$(ArtifactsDir)</artifactDirectory>
81
82 <labeller type="defaultlabeller" />
83
84 <intervalTrigger name="continuous" seconds="30" buildCondition="IfModificationExists" initialSeconds="30" />
85
86 <sourcecontrol type="vsts">
87 <workspace>$(ProjectName)</workspace>
88 <project>$/$(ProjectName)/Trunk</project>
89 <cb:vsts_ci/>
90 </sourcecontrol>
91
92 <tasks>
93 <nant>
94 <cb:nant_CI/>
95 <cb:nant_target_CI />
96 </nant>
97 </tasks>
98
99 <publishers>
100 <cb:common_publishers />
101 </publishers>
102
103 </project>
104 </cb:scope>
105
106 <cb:scope ProjectName="ProjectX">
107 <cb:define ProjectType="_QA" />
108 <project name="$(ProjectName)$(ProjectType)" queue="Q1" queuePriority="801">
109 <workingDirectory>$(WorkingDir)$(ProjectName)$(ProjectType)</workingDirectory>
110 <artifactDirectory>$(WorkingMainDir)$(ProjectName)$(ProjectType)$(ArtifactsDir)</artifactDirectory>
111
112 <scheduleTrigger time="06:30" buildCondition="ForceBuild" name="QA_Scheduled">
113
114 <labeller type="defaultlabeller" />
115
116 <sourcecontrol type="vsts">
117 <workspace>$(ProjectName)</workspace>
118 <project>$/$(ProjectName)/Main</project>
119 <cb:vsts_ci/>
120 </sourcecontrol>
121
122 <tasks>
123 <nant>
124 <cb:nant_CI/>
125 <cb:nant_target_qa />
126 </nant>
127 </tasks>
128
129 <publishers>
130 <cb:common_publishers />
131 </publishers>
132
133 </project>
134 </cb:scope>
135
136 <!-- End Projects -->
137 </cruisecontrol>
Build Script¶
1<project default="help">
2 <property name="solution" unless="${property::exists('solution')}" value="ProjectX.sln" />
3 <property name="configuration" unless="${property::exists('configuration')}" value="Debug" />
4 <property name="CCNetListenerFile" unless="${property::exists('CCNetListenerFile')}" value="listen.xml" />
5 <property name="msbuildverbose" unless="${property::exists('msbuildverbose')}" value="normal" />
6 <property name="CCNetLabel" unless="${property::exists('CCNetLabel')}" value="0.0.0.0" />
7 <property name="WantedCoveragePercent" value="75" />
8
9 <property name="mstest_metadatafile" value="${string::replace(solution, '.sln', '.vsmdi')}" />
10 <property name="mstest_unit_test_file" value="MStest_UnitTestResults.xml" />
11 <property name="mstest_CoverageFile" value="MStest_Cover.xml" />
12 <property name="TurnCoverageFileIntoXml_exe" value="C:\Tools\TurnCoverageFileIntoXml\TurnCoverageFileIntoXml.exe" />
13
14 <property overwrite="false" name="msbuildlogger" value="C:\Program Files\CruiseControl.NET\server\MSBuildListener.dll" />
15 <property overwrite="false" name="versionInfofile" value="VersionInfo.cs" />
16
17 <!-- custom scripts -->
18 <script language="C#" prefix="RuWi">
19 <references>
20 <include name="System.Xml.dll" />
21 <include name="System.dll" />
22 </references>
23 <imports>
24 <import namespace="System.Text" />
25 </imports>
26 <code>
27 <![CDATA[
28 [Function("UpdateVersionFile")]
29 public static bool UpdateVersionFile(string inputFile, string newVersion, bool debugMode)
30 {
31 bool ok = true;
32 try
33 {
34 System.IO.StreamReader versionFile = new System.IO.StreamReader(inputFile, System.Text.Encoding.ASCII);
35 string line = "";
36 System.Text.StringBuilder result = new StringBuilder();
37 string searchPatternVersion = @"(\d+\.\d+\.\d+\.\d+)";
38 string searchPatternAssemblyProduct = string.Format(@"AssemblyProduct\({0}(.*?)\{0}", "\"");
39 string replacePatternAssemblyProduct = string.Format(@"AssemblyProduct({0}(Debug)${1}1{2}{0}", "\"", "{", "}");
40
41 while (!versionFile.EndOfStream)
42 {
43 line = versionFile.ReadLine();
44
45 if (System.Text.RegularExpressions.Regex.IsMatch(line, searchPatternVersion) && (line.Contains("AssemblyFileVersion")))
46 {
47 line = System.Text.RegularExpressions.Regex.Replace(line, searchPatternVersion, newVersion);
48 }
49
50 if (debugMode && System.Text.RegularExpressions.Regex.IsMatch(line, searchPatternAssemblyProduct))
51 {
52 line = System.Text.RegularExpressions.Regex.Replace(line, searchPatternAssemblyProduct, replacePatternAssemblyProduct);
53 }
54
55 result.AppendLine(line);
56 }
57
58 versionFile.Close();
59
60 System.IO.StreamWriter updatedVersionfile = new System.IO.StreamWriter(inputFile);
61 updatedVersionfile.Write(result.ToString());
62 updatedVersionfile.Close();
63 }
64 catch (Exception ex)
65 {
66 ok = false;
67 Console.WriteLine(ex.ToString());
68 }
69 return ok;
70 }
71 ]]>
72 </code>
73 </script>
74
75 <script language="C#" prefix="RuWi">
76 <code>
77 <![CDATA[
78 [Function("FindFile")]
79 public static string FindFile(string folderName, string searchPattern)
80 {
81 var f = System.IO.Directory.GetFiles(folderName, searchPattern, System.IO.SearchOption.AllDirectories);
82 if (f.Length > 0)
83 {
84 return f[0];
85 }
86
87 return "";
88 }
89 ]]>
90 </code>
91 </script>
92
93 <!-- end custom scripts -->
94
95 <target name="help" >
96 <echo message="Removed for keeping the file shorter." />
97 </target>
98
99 <target name="clean" description="deletes all created files">
100 <delete >
101 <fileset>
102 <patternset >
103 <include name="**/bin/**" />
104 <include name="**/obj/**" />
105 <include name="${mstest_unit_test_file}" />
106 <include name="${mstest_CoverageFile}" />
107 </patternset>
108 </fileset>
109 </delete>
110 </target>
111
112 <target name="adjustversion" description="Adjusts the version in the version.info file">
113 <if test="${not file::exists(versionInfofile)}">
114 <fail message="file: ${versionInfofile} which must contains the version info was NOT found" />
115 </if>
116
117 <echo message="Setting version to ${CCNetLabel}" />
118
119 <property name="debugMode" value = "False" />
120 <property name="debugMode" value = "True" if="${configuration=='Debug'}" />
121 <if test="${not RuWi::UpdateVersionFile(versionInfofile,CCNetLabel,debugMode)}">
122 <fail message="updating file: ${versionInfofile} which must contains the version info failed" />
123 </if>
124 </target>
125
126 <target name="compile" description="compiles the solution in the wanted configuration" depends="adjustversion">
127 <msbuild project="${solution}" >
128 <arg value="/p:Configuration=${configuration}" />
129 <arg value="/p:CCNetListenerFile=${CCNetListenerFile}" />
130 <arg value="/v:${msbuildverbose}" />
131 <arg value="/l:${msbuildlogger}" />
132 </msbuild>
133 </target>
134
135 <target name="unit_test" description="runs the unit tests" depends="compile">
136 <if test="${file::exists(mstest_metadatafile)}">
137 <exec program="${mstest_exe}">
138 <arg value="/testmetadata:${mstest_metadatafile}" />
139 <arg value="/resultsfile:${mstest_unit_test_file}" />
140 <arg value="/testlist:UnitTests" />
141 </exec>
142 </if>
143 </target>
144
145 <target name = "cover" description="runs the tests with coverage" >
146 <!--
147 company rule : code coverage settings must be set via the file CodeCoverage.testsettings
148 with the following NamingScheme : baseName="cover_me" appendTimeStamp="false" useDefault="false"
149 -->
150 <if test="${file::exists('CodeCoverage.testsettings')}">
151
152 <exec program="${mstest_exe}" failonerror="false" resultproperty="testresult.temp" >
153 <arg value="/testmetadata:${mstest_metadatafile}" />
154 <arg value="/resultsfile:${mstest_unit_test_file}" />
155 <arg value="/testsettings:CodeCoverage.testsettings" />
156 <arg value="/testlist:UnitTests" />
157 </exec>
158
159 <property name="TestsOK" value="false" unless="${int::parse(testresult.temp)==0}"/>
160 <property name="DataCoverageFilePath" value="${RuWi::FindFile('cover_me','data.coverage')}" />
161
162 <fail message="No data.coverage found in cover_me folder" unless="${string::get-length(DataCoverageFilePath)>0}" />
163
164 <echo message="DataCoverageFilePath : ${DataCoverageFilePath}" />
165
166 <exec program="${TurnCoverageFileIntoXml_exe}" >
167 <arg value="${DataCoverageFilePath}" />
168 <arg value="cover_me\Out" />
169 <arg value="${mstest_CoverageFile}" />
170 <arg value="${WantedCoveragePercent}" />
171 </exec>
172
173 <fail message="Failures reported in unit tests." unless="${TestsOK}" />
174 </if>
175 </target>
176 </project>
177Code for converting coverage to XML¶
1using System;
2using Microsoft.VisualStudio.CodeCoverage;
3
4namespace VS_CodeCoverage
5{
6 class Program
7 {
8 static int Main(string[] args)
9 {
10 string[] Arguments = null;
11 var obc = Console.BackgroundColor;
12 int returnValue = 0;
13 bool ShouldBreakBecauseOfNotEnoughCovered =false;
14
15 try
16 {
17 Arguments = Environment.GetCommandLineArgs();
18
19 if (Arguments.Length != 5)
20 {
21 Console.BackgroundColor = ConsoleColor.Blue;
22 Console.WriteLine("Usage : {0} DataCoverageFilePath CoveredFilesPath ResultXmlFilePath wantedCoverage", Arguments[0]);
23 Console.BackgroundColor = ConsoleColor.DarkGreen;
24 Console.WriteLine(" {0} In\\LTREMRUBEN\\data.coverage Out d:\\codecover.xml 75", Arguments[0]);
25 Console.BackgroundColor = obc;
26 return 1;
27 }
28
29 string DataCoverageFilePath = Arguments[1];
30 string CoveredFilesPath = Arguments[2];
31 string ResultXmlFilePath = Arguments[3];
32 decimal wantedCoverage = Convert.ToDecimal(Arguments[4]);
33
34 CoverageInfoManager.ExePath = CoveredFilesPath;
35 CoverageInfoManager.SymPath = CoveredFilesPath;
36
37 Console.WriteLine();
38 Console.WriteLine("Converting {0} ...", DataCoverageFilePath);
39 var coverage = CoverageInfoManager.CreateInfoFromFile(DataCoverageFilePath);
40
41 var CoverResult = coverage.BuildDataSet(null);
42
43 System.IO.MemoryStream CoverResultStream = new System.IO.MemoryStream();
44 CoverResult.WriteXml(CoverResultStream);
45
46 Console.WriteLine("Initial Size in bytes : {0:0,0}", CoverResultStream.Length);
47 CoverResultStream.Position = 0;
48
49 Console.WriteLine("Cleaning up xml info ...");
50 System.Xml.XmlDocument CoverResultXmlDoc = new System.Xml.XmlDocument();
51 CoverResultXmlDoc.Load(CoverResultStream);
52
53 var LineInfos = CoverResultXmlDoc.SelectNodes("//Lines");
54
55 foreach (System.Xml.XmlNode lineInfo in LineInfos)
56 {
57 lineInfo.RemoveAll();
58 }
59
60 var SourceFileNameInfos = CoverResultXmlDoc.SelectNodes("//SourceFileNames");
61 foreach (System.Xml.XmlNode SourceFileNameInfo in SourceFileNameInfos)
62 {
63 SourceFileNameInfo.RemoveAll();
64 }
65
66 CoverResultXmlDoc.PreserveWhitespace = false;
67 CoverResultXmlDoc.Normalize();
68 CoverResultXmlDoc.Save(ResultXmlFilePath);
69
70 Console.WriteLine("--> Compressed Size in bytes : {0:0,0}", new System.IO.FileInfo(ResultXmlFilePath).Length);
71 Console.WriteLine();
72 var moduleInfos = CoverResultXmlDoc.SelectNodes("//Module");
73 foreach (System.Xml.XmlNode moduleNode in moduleInfos)
74 {
75 string moduleName = moduleNode.SelectSingleNode("./ModuleName").InnerText;
76 var BlocksCovered = Convert.ToDecimal( moduleNode.SelectSingleNode("./BlocksCovered").InnerText);
77 var BlocksNotCovered = Convert.ToDecimal(moduleNode.SelectSingleNode("./BlocksNotCovered").InnerText);
78
79 decimal pct = (BlocksCovered / (BlocksCovered + BlocksNotCovered)) * 100;
80
81 Console.WriteLine("Module {0} : pct {1:0.0000}", moduleName, pct);
82
83 if (pct < wantedCoverage) ShouldBreakBecauseOfNotEnoughCovered = true;
84 }
85 Console.WriteLine();
86
87 if (ShouldBreakBecauseOfNotEnoughCovered)
88 {
89 returnValue = 3;
90 Console.WriteLine("! ! Coverage not high enough, wanted set to {0:0.00}", wantedCoverage);
91 }
92 else
93 {
94 Console.WriteLine("Done.");
95 }
96
97 }
98 catch (Exception ex)
99 {
100 returnValue = 2;
101 Console.WriteLine(ex.ToString());
102 }
103
104 return returnValue;
105 }
106 }
107}