Testing Definitions

After a recent project meeting, I realised that the participants had differing interpretations of what is involved in each phase of testing. I compiled the definitions below to ensure the team are aligned.

A Unit Test is a test that ensures that a single class or routine does what it is meant to without needing to cross class or system boundaries in order to execute. This can be represented using the ACID acronym:

Atomic
Consistent
Isolated
Durable

• A test that verifies multiple pieces of functionality is not atomic.
• A test that changes results based on external factors (such as user interaction) is not consistent.
• A test that accesses a database, file system or interacts with other classes is not isolated. Neither are tests that need to be run in a specific order.
• A test that fails when specific data isn't available or breaks when other classes in the application change (not including the class under test) is not durable.
  

Integration Testing is the combined execution of two or more classes or components. Integration testing can start as soon as there are two classes to test and should continue until the entire system is complete.

Regression Testing is the repetition of previously executed test cases for the purpose of finding defects that previously passed the same set of tests.

System Testing is the execution of the software in its final configuration, including integration with other software and hardware systems. It tests for security, performance, resource loss, timing problems, and other issues that can't be tested at lower levels of integration.

Acceptance Testing is confirmation by a key user, through trial or review, that the software meets mutually agreed-upon requirements.

Load Testing is the execution of the software with varying levels of throughput to determine at what point the response time degrades or fails.
Recovery testing is the process of testing how well a system can be recovered following a hardware failure or other catastrophic problem.

Update Default Visual Studio 2008 Project Templates

To update existing Visual Studio project templates, follow the manual instructions here. Should you wish to change the default templates, they can be found at:

C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\ProjectTemplates

I wanted to set TreatWarningsAsErrors to true by default for all class library projects. First precaution, I took a backup copy of the original zip file prior to unzipping the files.

Next step, the line below was added to the to the debug and release PropertyGroup sections of the extracted csproj file.

<treatwarningsaserrors>true</treatwarningsaserrors>

Then, I recompressed all the extracted files and copied the zip back to the ProjectTemplates directory, overwriting the original zip.

Finally, it is necessary to regenerate the templates as Visual Studio seems to load them from:

C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\ProjectTemplatesCache

To regenerate the default templates, run devenv.exe /installvstemplates from a Visual Studio command prompt.

Disable Shutdown Event Tracker

I have recently read a couple of blog posts that promote using Windows Server 2008 as a Workstation.  To avoid potential integration issues, I always think it's a good idea to build applications on an environment that mirrors test and production as closely as possible.  So when developing applications with differing deployment targets, for instance, IIS, BizTalk or even the Compact Framework, it's important to have different virtual Windows Server 2003 development images available.  There are other benefits to this approach which I'll list in a future post.

However, one annoyance with using multiple development environments is the shutdown event tracker which forces you to enter a reason for each system shutdown or restart.

To turn off the Shutdown Event Tracker in Windows Server 2003 or 2008, follow the steps below:

1. Open the Microsoft Management Console
2. Add the Group Policy snap-in
3. Under Administrative Templates expand System
4. Set Display Shutdown Event Tracer to Disabled

Software Construction Analogies

Software construction is often compared with building construction.  In fact, Steve McConnell devotes a whole chapter to software metaphors in his seminal work, Code Complete 2. 

 

In recent times, I've taken note of some of the analogies used by colleagues and noticed one directly related to construction:

 

If you only have a hammer, everything looks like a nail.

 

Maslow's Hammer has been stated and paraphrased many times over, especially in software development circles.

The concept is thus: individuals who are incomplete in their knowledge or training of solutions propose the same type of solution to every problem they encounter. They opt for the more familiar solution to one that may be more effective yet with which they are unskilled.

In closing, another colleague of mine recently used, perhaps invented, an analogy that provided a new take on the 'sledgehammer to crack a nut' idiom.  He considered the introduction of Windows SharePoint Services to solve a rudimentary Workflow problem to be the equivalent of 'using a cannon to kill a mosquito'.  Now who could possibly argue with that?

The Quality Reduced Product

I revisited the concept of the quality reduced product today when a product manager insisted that the software development team take a really low cost option.  I reminded the product manager that the typical steps in delivering a product in less time and for less money result in lower quality.  Whilst the product's end user willingly consented to this trade off, my warnings regarding the impact on the team were largely ignored.

The heading of this post is taken from the seminal book for Software Managers, Peopleware: Productive Projects and Teams by Tom DeMarco and Timothy Lister.  In the chapter Teamicide they list the sure-fire ways to inhibit the formation of teams and disrupt project sociology.  Quality reduction of a product is one cause.

... [quality] concessions are extremely painful to developers as their self-esteem and enjoyment are undermined by the necessity of building a product of clearly lower quality than they are capable of.  An early casualty of quality reduction is whatever team identification the group has been able to build.

This is also discussed in Rob's post Nine Things Developers Want More Than Money, he reminds managers that being setup to succeed is key to developer motivation. 

Developers want to build software that not only works, but is maintainable; something they can take pride in. This is not in-line with product development’s goals, which are for developers to build software that works, and nothing more.

The first thing to go when time is tight is quality and maintainability. Being forced to build crap is one of the worst things you can do to a craftsman. Delivering a project on-time but knowing it’s a piece of crap feels a heck of a lot like failure to someone who takes pride in what they build.

In conclusion, whilst the effects of quality reduction are intangible in the short term, the long term effect on a team's engagement and motivation can be catastrophic.

Stakeholders Defined

This post regarding stakeholders is again inspired by and contains excerpts from the book Software Systems Architecture: Working with Stakeholders using Viewpoints and Perspectives.

Software systems are not just used, they have to be built and tested, operated, repaired and of course paid for. Each of these activities involves a number of people, each with their own requirements and interests. The people are collectively referred to as stakeholders. Understanding the role of each stakeholder is fundamental to understanding the role of the architect in the development process.

The IEEE Standard 1471 on architecture description defines a stakeholder as a person, group, or entity with an interest of concern about the realization of a system's architecture.

I personally have a checklist I like to run through when a new project involving my development team is initiated. I like to make sure that all the necessary teams, such as support or infrastructure, are correctly engaged prior to implementation. From the perspective of a software architect, Nick Rozanski and Eoin Woods succinctly classify these stakeholders according to their roles and concerns as below.

 

Stakeholder	Role
Acquirers	Oversee the procurement of the system or product. In my experience, these stakeholders are often referred to as 'the business' and are usually the most important stakeholders, providing or authorizing funding. Their goals are usually value for money and efficient expenditure of resources during delivery and operation.
Assessors	Oversee the system's conformance to standards and legal regulation.
Communicators	Explain the system to other stakeholders via documentation and training materials. Analysts communicate the requirements to the developers whilst technical authors create manuals for the users and administrators.
Developers	Construct and deploy the system from specifications or lead the teams that do this.
Maintainers	Manage the evolution of the system once it is operational Their main concerns focus on documentation, instrumentation or change control. In my experience this is often the responsibility of the development team.
Suppliers	Build and supply the hardware and infrastructure on which the system will run.
Support	Provide support to users for the product or system when it is running.
System Administrators	Run the system once it is deployed. They focus on concerns such as monitoring, business continuity and disaster recovery and scalability.
Testers	Test the system to ensure that is suitable for use. Testers act as the conscience of the development team, systematically testing the system in order to establish whether it it is suitable for deployment and use. Unlike developers, testers do not have a sense of ownership of the implementation and with their specialist knowledge and experience means they can perform a more thorough and objective job of evaluating the system than other stakeholders.
Users	Define the system's functionality and ultimately make use of it.

Most development projects should include representatives from most if not all of these stakeholder groups though their importance varies from project to project. I totally agree with authors when they state that not considering a view from each class of stakeholder will lead to problems in the future.

Software Architecture Concepts

I'm currently reading Software Systems Architecture: Working with Stakeholders using Viewpoints and Perspectives. I already think this is a book I should have read years ago! In chapter 2, the authors, Nick Rozanski and Eoin Woods break down the widely accepted definition of software architecture by the Software Engineering Institute (SEI) at Carnegie-Mellon University in Pittsburg:

The architecture of a software intensive system is the
structure or structures of the system, which compromise software elements, the externally visible properties of those elements, and the the relationships
amongst them.

I have seen this definition quoted many times but rarely examined in closer detail. Rozanski and Woods do however analyse structures and externally visible properties.

They state that there are two types of system structure, static and dynamic. Static structures of a software system define its internal design-time elements and their arrangement whilst dynamic structures of software system define it's runtime elements and their interactions.

Externally visible properties also manifest themselves in two ways, the externally visible behavior (what the system does) and quality properties (how the system does it). The externally visible behavior of a software system defines the functional interactions between the system and it's environment. A quality property is a nonfunctional property of a system such as performance, security, or scalability. These are commonly referred to as the "ilities".

So the key take away is that it's the role of the architect to derive the static and dynamic structures for each candidate architecture and understand the extent to which they exhibit the required behaviors and quality properties.