MEASURING I.T. PERFORMANCE

Summary of previous module:

Last time, we discussed the importance of aligning an IT strategy with the corporate strategy.
We reviewed methods, structures, processes, and people issues that can facilitate IT alignment.
We also discussed that accounting for IT as an overhead function, making it difficult for senior managers to assess IT’s contribution.

MEASURING I.T. PERFORMANCE

In this module, we explore measurement systems that help answer the question:

“Is my IT function performing well?”

What are some general guidelines for strategic measurement systems?

Balanced Scorecard Approach

What are some valid ways to assess IT performance?

Benchmarking

What measures can be used to assess the productivity and quality of software development and support--the most expensive and most difficult IT activity to assess?

Function Points

BALANCED SCORE CARD APPROACH

by Robert Kaplan and David Norton, HBR, 1992, 1996

Problem the articles address:

Exclusive use of Financial Measures such as NPV, Payback, and IRR can lead to poor decisions focused on the short run.

Example: Financial assessment of total outsourcing:

Vendors buy IT assets (up to $300 million)--why won’t a bank even give you a loan on IT assets?
Vendors postpone payments until latter part of the contract
Vendors may purchase company stock
All this looks great from a NPV standpoint--what is missing from this picture?

“Best objectives can be badly achieved”

Reduce time to market: Improve management of new product innovations or Release only new products with minor differences from old products

Reduce production costs: Better set-up times or Producing more standardized, low margin products

Faster customer service on phone: Hang up quicker Develop an IT system that pulls all relevant customer documents on line

BALANCED SCORE CARD APPROACH

No single measure can provide a clear performance target.
Balanced Scorecard is a set of measures that gives managers a fast but comprehensive view of the business. Four categories of measures:

Customer perspective: How do we look to out customers?

Lead time: customer order to delivery times
Quality: defect rates
Performance: customer satisfaction surveys

Internal perspective: What must be excel at?

Productivity: output/input
Delivery: projected verses actuals
Employee Turnover

Innovation and learning perspective: Can we continue to improve and create value?

New Product delivery times
Adoption of new technologies
Employee Education

Financial perspective: how do we look to our stockholders?

NPV, ROI, Return on Capital, Stock prices

The how:

1. Translating the Vision

How do you translate a mission statement like “provide superior service to targeted customers”?

Need to develop scorecard measures for the specific products and services for the specified and articulated target segments.

2. Communicating and Linking

Communicating & educating
Setting Goals--Ask SBUs, then employees to set own objectives in accordance with corporate scorecard
Linking Rewards to Performance Measures--but what can be the unintended consequences of this?

Example: UK Heath Care system rates hospitals on fastest patient care

3. Business Planning

-- need to integrate strategic planning with budgeting & resource allocation.

In Earl paper, we saw the approaches to IT strategy tended to focus on one or the other

Method approach rated high on strategy vision, low on implementation
Aministrative approach rated low on strategy vision, too focused on budgets & resources
Organizational approach rated high on both.

4. Feedback and Learning

Adapting to changes
Adapting to learning, like unintended consequences

BENCHMARKING AS A STRATEGY FOR MANAGING CONFLICTING STAKEHOLDER PERCEPTIONS OF INFORMATION SYSTEMS

by Lacity & Hirschheim

How can senior managers assess IT performance?

They want to be convinced they are getting the best value for their money.
It is very difficult to develop absolute, objective, measures of IT performance.
Benchmarking is a way of assessing relative performance.

Benchmarking: Measuring internal IT performance against other companies’ IT performance

FOUR SOURCES OF BENCHMARKS

1. Informal Peer Comparisons

Pro: IT managers trust peers

Con: Other stakeholders don’t perceive information as objective because it’s too easy to filter out “bad” news.

2. Informal Outsourcing Queries

Pro: Quick and generates a number

Con: Other stakeholders don’t perceive information as objective because it’s too easy to filter out “bad” news; Vendors don’t put much effort into the informal bid.

3. Formal Outsourcing Evaluations:

Pro: Most rigorous and valid way to assess performance

Con: expensive, disruptive

4. Benchmarking Services

Benchmarking services collect data, normalize it, create reports which plot the client’s performance against a selected reference group of the client database.

Pro:

Exhaustive assessment against hundreds of companies
Perceived as objective by other stakeholders
Less disruptive

Con: Some IT managers select benchmarking services to obtain a good report card

“If you want a bunch of graphs that say you are pretty good you can buy those”

WAYS TO SABOTAGE/BIAS BENCHMARKS

Select a reference group of marginal performers
Select technical benchmarks which have no meaning to other stakeholders
Select benchmarks which focus on wrong IT values: Cost vs. Service
Undermining the Data Integrity of the Benchmarks
Designing Reports that Misrepresent the Data

OBTAINING VALID BENCHMARKS

Align performance expectations for the IT portfolio based on business contribution
Select a family of measures suggested by the IT strategy
Stakeholder participation in the selection of benchmarks
Select the reference group which represents the stiffest competition possible
Select reference group which means the most to other stakeholders
Industry
Size of Organization
Geographic Region
Use benchmarks not as a report card but to identify improvements.

BENCHMARKS FOR VARIOUS I.T. FUNCTIONS

A CLOSER LOOK AT FUNCTION POINTS

Application Software Development and Support is the hardest area to manage--estimates range as high as 80% of all systems fail because they are late, over-budget, and lack functionality.

There is a need for valid measures of software productivity and quality

“If you can’t measure it, you can’t manage it”

We need software measures that:

1. Estimate project completion time
2. Estimate project cost
3. Determine the value of the project to user
4. Calculate analyst & programmer productivity
5. Estimate system quality
6. Independent of technology
7. provide a benchmark against other companies

PRODUCTIVITY & QUALITY MEASURES IN GENERAL

Productivity = Output/Input

Quality: Number of defects per unit of output

In systems, inputs are easier to measure: total man-hours, total costs, etc.

What do we use for output?

Traditionally, we have used lines of code (LOC)

LINES OF CODE

Example: on average it takes 10 man-hours to develop one line of code

What does this mean? How do we interpret this number?

The averages vary with the programming language

It motivates non-efficient coding (i.e. the more the better)

It isn’t calculated until later stages of SDLC, thus a poor indicator of time & budget

Users cannot understand it very well

FUNCTION POINTS

Developed by Albrect of IBM in 1979

Function points estimate software size based on the functions delivered to users in the form of outputs (reports), inputs (data entry screens & forms), queries, files (such as a customer database), and interfaces to other systems.

Function points becomes our measure of output

Productivity will be function points/man-hours

Quality will be number of defects per 1,000 function points.

We will see that function points meets our requirements for sound software metrics:

Additional benefits:

overhead rate associated with counting function points is low:
It takes about 5 minutes to count the function points associated with what it took you a month to program.