How to deliver real value using Iterative and Incremental Development

The benefits of Agile development approaches in which developing software applications iteratively and incrementally is an important and inevitable principal are questionable if deliverables produced during the project do not introduce any business value.
If a development project doesn’t deliver business values incrementally iteration by iteration then it’s executed in a value-neutral setting in which

• requirements are treated as equally important
• code delivery is as important as the delivery of an usable system
• implementing largest number of requirements possible become more important than delivering desired outcome of the project

In another word, the development team iteratively implements the requirements rather than iteratively solve problems. Although project members feel that they are iterating successfully, they never make the transition from iteratively addressing the technical risks to iteratively delivering usable versions of the system that provide realizable business benefit.
This will raise the question that how we exactly have to develop requirements iteratively to secure delivery of the actual desired outcome of the project. To answer this question we need to define the best possible deliverable which assures the stakeholders that development is progressing toward creating a system that is aligned with their actual requirements. That deliverable is nothing but an executable that its features represent a collection of end-to-end behavioral threads of system usage. In a simpler word, the output of each iteration should be an executable that possesses three very important characteristics:

• Should cover a set of the actual “behaviors” of the system that are most important to the stakeholders at that point of time
• Should keep the confirmed “behaviors” that are implemented in the previous iterations
• All its implemented features composing desired “behaviors” should be working integrated smoothly

You might’ve noticed that I put the world behavior in the quotes. What I am trying to emphasize here is that it’s not important how many features you implement. What’s important is that the implemented features form desired behaviors of the system and do something useful for the business. Another word, only those features that are really needed have to be implemented. Statistics data such as CHAOS report, as I’ve pointed out in another article at http://www.iasahome.org/c/portal/layout?p_l_id=PUB.1008.62, show that a high percentage of implemented features in many projects are never used, even in those projects that are declared successful.
Now that we have set the ground rules and defined what it is that our projects iteratively have to deliver, let’s take a look at the ways of doing that.
To achieve the value delivery, we need to have a way to work around the end-to-end behavioral threads (independent threads of system usage) or what is called scenarios. “Credit Check” and “User Registration” are examples of scenarios. They are independent threads of system usage that tell stories about how for example a user registers and what other things might happen along the way.
The obvious question here would be “How to find the best scenarios and in what order they have to be implemented?”
As I mentioned above, scenarios are stories and finding scenarios is about the requirement gathering skills that is far beyond the scope of our discussion here. But as the principals of Iterative development state, scenarios do not have same level of priority. With that in mind, at every iteration we need to find those scenarios that have the following characteristics:

• They are addressing the most important goals of stakeholders at that point of time
• They are addressing the most pernicious risks in the project

Project risks threaten the project's ability to deliver the desired outcomes in a timely and cost-effective manner. Technical risks can be mitigated by choosing scenarios that force the confrontation of the risks. By mapping the risks to scenarios that mitigate them, iterations can be planned to ensure that every iteration reduces project risk and increases the business value realized from the solution.

Relationship between Risks, Scenarios, and Desired Outcomes

So, for example if in our earlier example of user registration, the user information needs to be registered in a LDAP database, the development team need to find risks that are mapped to this scenario, for example limited knowledge of technical team in working with LDAP databases, and rate those risk. If the level of their threat is high then they have to be implemented before other less important risks.
I must mention that addressing risks is not less important than implementing important scenarios for stakeholders. What if a scenario which imposes a big risk is not important when you are in the third iteration and when it comes close to the end of the development a change in business strategy makes it an important scenario? This is a rare case of course. But it can become a counterproductive situation in a project. A smart architect keeps his/her eyes on those scenarios and related business scenarios and as the iterations pass by he/she tries to decide whether the stakeholders’ attention is turning toward those scenarios. Some architects assign a very small team (with One or Two members) to work around some of those risks simultaneously so if their related business scenarios become more important they don’t impose a big danger.
The widest used technique for modeling scenarios is Use Case Modeling. Use cases structure and group requirements by user goals and business value. A use case contains a set of flows, which describe the interaction between the system and its actors. The set of flows provide a map of how the system can be used to meet the goals of the use case and a structured way of describing the set of end-to-end scenarios needed to drive the iteration planning and the development of the software.
A use case captures a set of scenarios, each of which is described by one or more of its flows. All use cases contain a basic flow, which describes the set of normal, "happy day" scenarios. They also contain a number of alternative flows to describe variations, extensions, and exceptions to the basic flow. These can be combined with the basic flow to describe all the other possible scenarios. So, a scenario can be considered to be defined by a use case's basic flow plus zero or more of its alternative flows.
Besides, almost all of the case tools that I have seen out there have a tool for drawing and managing use cases. It seems that it has become a unified way of defining and managing scenarios.

Summary

Nothing but an executable code depicts the actual progress of a development project. But not every executable does that unless it implements end-to-end behavioral threads that deliver most important requirements of business. This doesn’t mean that there shouldn’t be any documentation. It has to be, but as much as it’s required.


-- This post has been published as an article on the ITtoolbox web site as well.

Choosing the Appropriate Software Development Process Framework

Hi folks,

Here is another published article of mine on IASA's web site. I hope can keep it up.
This is the direct link and this is a link through IASA's content repository.

I also figured out that you can not post any comment on the IASA's forum unless you are a member. So please do it here whatever it is.

Overcoming Requirement Challenges with IID

Recently I've allocated some of my time to writing articles for IASA's web site. Here is one: Overcoming Requirement Challenges with IID.

Considerations of creating a successful SDP

Hello

I’ve recently joined a BPO (Business Process Outsourcing) provider company which gives me an excellent opportunity to put my knowledge of SOA and SaaS in action. So I guess that’s what is going to shape my future posts here.
Well, here is one.
SDPs (Service Delivery Platforms) are playing almost the same role for delivering Software as a Service (SaaS) as Operating Systems do in desktop applications’ development and deployment. Rather than requiring each application to create the full stack of subsystems needed for it to run, an operating system provides an infrastructure through which general purpose services are reused. The following picture depicts the natural and ongoing process of extraction and generalization of functionality from application into frameworks and from there into the core platform components which leads to the improvement of economies of scale.

Figure 1: Borrowed from Microsoft's Architecture Journal

There would be the same concept in various levels offered by SDPs. There are different factors that can be used to specify the level of success of an SDP. What I mean by the Level of Success is SDP’s effectiveness and scalability, and the ability to provide highly reusable services – for example through an SDK - that will make the implementation and maintenance of SaaS-delivered applications less intensive.
Observation of existing SDP offerings seems to indicate that two most important factors are:

• Services breadth: the completeness of the platform; in other word, the support for different stage of SaaS-delivered application life cycle (following picture)
• Services depth: the degree of sophistication of the services it provides.
  

Figure 2: Borrowed from Microsoft's Architecture Journal

Hence there are two aspects that SDP implementers (mostly traditional hosters) and ISVs (Independent Software Vendors) who develop and deploy the service should take under consideration:

• Different Application Archetypes; Business applications can be classified in different archetypes based on their characteristics and requirements. Two examples of these archetypes are OLAP and OLTP. Each of these application families has its own constraints and characteristics. For example OLTP will optimize for low latency, whereas latency for OLAP systems is not as important. The infrastructure to implement and support each is significantly different.
  The point is that SDP’s effectiveness is pretty much dependent on the archetype served. The more knowledge of the application an SDP has, the greater its ability to increase the efficiency of running and operating it, and the greater the degree of sharing.
• Patterns and Frameworks used in design and development; no matter what archetype an application is bound to, it can follow a pattern in design or development or it can use a framework to implement some of its services. An example of common, standard and widely adopted application infrastructure framework is Microsoft’s Enterprise Library.
  I would say a valuable SDP provides an SDK including documentation, samples and even some basic tools for ISVs enabling them to develop their software using known patterns and frameworks. This way the SDP has a much increased ability to automate common procedures and offer more advanced operational management capabilities. Thus, finer-grain tuning, customization and troubleshooting will be available.
  Additionally, hosters can offer a higher range of differentiated services with different monetization schemes. For instance, the hoster knows that all applications will log run-time exceptions. So basic run-time exception logging can be offered in the basic hosting package, and advanced logging, notification and escalation could become a premium offering. Notice that with this approach the ISV application doesn’t change, because all the logic resides on the SDP side.
  

Figure 3: Borrowed from Microsoft's Architecture Journal

MSA

You might've heard of MSA (Master of Science in Analytics) by now.
It’s an intensive 10-month professional graduate degree program designed by Institute for Advanced Analytics at North Carolina State University that focuses exclusively on the tools, methods, and applications of analytics and is designed to educate professionals with sophisticated technical skills necessary to navigate and analyze the masses of data that organizations are collecting.
The objectives of the program are:

• provide students wit an understanding of basic concept and methodologies in the analysis of massive data sets
• Show how these methods are applied to a variety of complex problems facing organizations, using real-world problems
• Give students a sense of the broader context, such as security, privacy and ethical issues in the use of personal and confidential data

What makes this program unique is its emphasis on real-world, business-focused analytics. Comparing this program with other business related programs you'll realize that its aim is to provide the talent capable of leveraging world-class business intelligence systems. For example typical MBA degrees include limited instruction in statistics or advanced degrees in Data Mining don’t address critical and contextual issues such as data quality and integration, privacy, security and enterprise-wide decision making.
This endorses what the course designers believed that “Competing on analytics in corporations, government agencies and educational institutions is becoming a must”.

What has mostly caught my attention (and the reason I made this post) was that this program is about how to apply mathematic to get what you are looking for. Those who, like me, have studied applied mathematics and liked it and dealt with pure-math professors know what I mean.

If you like to participate and be one of the first graduates of this program, you better hurry. For more information you can take a look at the program’s website at NCSU.

What your project success is driven by?

What would be your answer to the above question? Use-Case driven, Test-Driven, Scenario-Driven, or perhaps Feature-Driven.
People often talk about these drivers as the only forces steering projects and shaping project plans. But in fact these mechanisms are used for defining and managing projects’ scopes. I believe without Iterative and Incremental Development (IID) approach you won’t have the means to implement a practical solution that users and stakeholders can take advantage of. My main reason is latent in the definition of Stakeholder and Stakeholders’ role in the success of a project.Stakeholder in “Use Case Modeling” book of Kurt Bittner and Ian Spence is defined as an individual who is materially affected by the outcome of the system or project(s) producing the system. Hence, one could draw the conclusion that the best impetus for developing a system is its stakeholders’ feedback and their acceptance of the solution.They are the primary source of requirements, constraints, and risks for the project. They supply the funding and audience for the project and will make the decision whether the project is worthwhile.In my opinion, IID is the right approach to get stakeholders involved. You need to get their approval at the end of each iteration to be able to move to the next one. That empowers you to revise your plan and improve your development process.You can also embrace change requests – which the risks they impose increase as you get closer to the end of the project - from the outset of the project.
Utilizing IID you can suppress “Change Prevention Process” anti-pattern the goal of which is to prevent new requirement being added to the project or existing requirement expanded upon. Another word, sticking to the original plan and requirements and using it as an excuse to stop users from changing them. I would say that's a common issue in all projects avoiding IID. Of course, in order to avoid falling into "Never Ending Change Requests" pitfall all fundamental changes need to be detected and addressed before architecture is solidified.

To summarize, your project has to be Stakeholder Driven.

The lawyers who say NO!

A while ago I was reading an article at Dr.Dobb’s Journal from Scott Ambler (see his profile at IBM) here. I suggest you guys to read it as well if you haven’t yet.

It actually points out a hidden impediment which is an obstacle to many of software development teams that try to exercise IID (Iterative and Incremental Development); the lawyers who say NO!
Before I continue I must remind you that what you read here is my personal opinion and you might find it incorrect or disagree with. Well, that’s what the comments are for. I believe as a reader, it’s your obligation to share your opinion with the writer and other fellow readers.

“Lawyers who say NO!” is a metaphor for those people who approve the project’s funding which can be the customer itself, those who audit project’s compliance with what is planed business-wise, and those real lawyers who make the contract with customers. The message this article is trying to send is no matter how much you, as a software specialist, try to fix your process and development methods in order to mitigate the risks and achieve the goals, there will be non-technical obstacles that can make your efforts worthless. Unless everybody involved in the project has the same understanding of what has to be done and how (of course with their own point of view and different levels of details).

Customers and project investors need to understand the fact that that a close cooperation with the development team is the key role they can play in the success of a project and eventually getting a better ROI from the product.
Moreover, having a right governance process with which development methodology can be aligned and can resolve the potential clashes between technical and business views is crucial. Because business success partially rests on successful delivery of software products, business executives need to understand how their investments in information technology and software development are paying off. They demand visibility and accountability. That's where the governance shows up. I say partially because user experience is more than a good looking and bug-free software application.

So as you can see delivering a successful project requires a right collaboration process in place. Note that this is in addition to having a suitable Development Methodology and Governance Process.