Quality Management | By Dave Nielsen | minute read
Senior executives use Project Scorecards, also known as Balanced Scorecards, to ensure project activity aligns with the strategies and visions of the organisation. The scorecard is a little like putting the reader in the driver's seat of a car. They need a view through a clear windshield to determine the direction the project is headed in and instrumentation such as the speedometer, tachometer, oil pressure gauge, and water temperature gauge to ensure the car is performing correctly and not in danger of breaking down or crashing. By the way, the reason these scorecards are often referred to as "balanced scorecards?" Previous to their introduction, executives had only a view of the financial performance of operations or projects. A need was identified for a more "balanced" view of activities, one that would include measurements of other aspects of performance.
Project Scorecards should satisfy two project requirements: the need for a vehicle to communicate project performance and health to busy executives and the need to compare performance across multiple projects.
The tips and tricks described in this article will draw upon the project management best practices described in the PMBOK (Project Management Body of Knowledge). You can acquire these best practices by taking a quality PMP course, or PMP exam preparation training.
Balanced Scorecard (BSC)
Balanced Scorecards, or BSC, were devised and introduced by David P. Norton and Robert S. Kaplan in 1992 to correct the one-dimensional view of organisational performance provided by measurement instruments of the past. Performance was measured in terms of finances up to that point and the downside of measuring financial performance was that it lagged behind other elements of performance. For example, an organisation that undertook a $5M project to capture a 10% increase in market share worth $4M per year would look great if it were completed for just $4.5M, but ROI could not be measured until at least the first year after project completion.
Norton and Kaplan proposed measuring organisational performance in 3 additional areas: Customer, Internal Business Processes, and Learning and Growth to gain a more balanced view of organisational performance. Norton and Kaplan believed that an organisation performing well in these areas could expect to perform well financially as well. Performance measurement in these areas would also allow executives to be proactive in correcting problems that would lead to poor financial performance, rather than reacting once the organisation experienced the poor performance.
Balanced Scorecards were meant to measure performance in all areas of the organisation, not just project performance. A project could fall under any one of the 4 areas of performance, or even under several, but will only count for one facet of organisational performance. The use of BSC by the organisation will definitely influence the design of the project scorecard, but the project scorecard cannot duplicate the format of the BSC because the information available does not satisfy the requirements of the BSC.
Key Performance Indicators (KPI)
Key Performance Indicators, or KPIs, is an acronym frequently used in conjunction with Balanced Scorecards. Balanced Scorecards use 5 or 6 metrics in each of the 4 areas of organisational performance as measurements. The 5 or 6 metrics could be any one of thousands which have been measured, but choice is constrained by the nature of the area (financial, customer, business processes, learning and growth), the nature of the organisation, and the nature of the processes and tools in place to capture the metrics. These metrics are called Key Performance Indicators, or KPIs.
This article is more concerned with giving you practical advice on composing your project scorecard, and choosing the metrics to support it, than in the theory of balanced scorecards and KPIs and for that reason I will avoid the use of these acronyms as they don't directly help you build your scorecard.
Goals and Objectives
Keep the quality tenet "quality is a product's ability to meet customer requirements" in mind when selecting the metrics to base your scorecard on. The first question a senior executive might ask regarding your project is "What strategic goal or objective will this project help me meet?" The answer to this question should be found in your Business Case and/or Project Charter. There should be one or two over arching objectives described in those documents which answers that question. Strategic objectives will be those targets or goals that your organisation will be able to meet as a result of your project delivering its goals and objectives.
Your project objectives might be to produce an engineering order and configuration system that will store up to 100,000 historical software orders and up to 10,000 configurable software packages, process a customer order and produce a software operating system for the customer's central office telephone switch. Your organisation's strategic objective might be to increase market share 10% by offering customers the ability to perform configuration management of their software using your new software system. Note that it is not the project manager's responsibility to achieve a 10% increase in market share, simply to produce the software order and configuration management system that senior executives believe will enable them to achieve that goal.
Choose the high level project goals and objectives which support the organisation's strategic objective and identify them with that objective. For example, you could make the strategic objective the main bullet and the project objectives sub-bullets:
- Increase the Company's market share by 10%
- Introduce an engineering order and configuration system which will:
- Store up to 100,000 historical software orders
- Manage up to 10,000 configurable software packages per order
- Process customer orders and build an operating system
- Manage software library rules governing configuration
- Manage marketing rules
- Identify the software configuration of each customer's switch
The main bullet should be a strategic objective and no more than one page/slide worth of supporting project goals and objectives should accompany it.
Overall Project Performance
The overall project performance indicator should be derived from 3 or 4 indicators of the project's health. These are: performance to budget, performance to schedule, performance to scope, and quality. The overall project performance indicator is a subjective evaluation of project health; there is no single metric which can be used.
Red, yellow, and green are commonly used to indicate overall project health with red being used to indicate a project that is performing poorly and in need of intervention from sponsors, or steering committee to put it back on the rails. Yellow is used to indicate a project that is not performing to established standards but which can be recovered using resources currently available to the project. Green indicates that the project is performing well.
Regardless of which means you use to indicate overall project performance, you should use performance indicator of the project area which is performing the most poorly to indicate overall project health. If performance to schedule is yellow then overall performance cannot be green.
Performance to Budget
Earned Value Management (EVM) provides valuable and universally accepted metrics for measuring a project's performance to budget. The objective is to determine whether you have accomplished the project work you planned for the budget you planned. Budget will include the cost of all goods, services, resources (human and non-human), and administrative services consumed by the project. There are several different ways of mining the metrics needed to perform these measurements. The simplest is the MS Project file for the project, which can track all these. MS Project shows money being spent at exactly the same rate as activity completion. The budget for the C++ programming is exactly 72% consumed when the C++ programming is 72% complete. This may not be what your audience is looking for.
Check for financial reports being issued by your finance organisation that measure budget consumption for your project. These reports are likely to be ones read by senior executives and issuing a financial report of your own which does not align with the one from the finance department could lead to hours spent reconciling the two reports. Some things to consider when you gather your metrics:
- Are there any time tracking tools used by the finance department to track labour costs?
- When does the finance department consider labour budgets spent? When the cost is incurred? When the cheque is written? When the cheque is cashed?
- When does the finance department consider the cost of materials incurred? When the materials are purchased? When they are delivered? When they are used? When the cheque is issued?
Use reports from the finance organisation to base your reports on and explain how you used them, if possible. If financial reports based on the project are not available to you, learn the methods used by finance for their reports and make your metrics conform to your organisation's financial reporting methods.
There are several different EVM performance indicators which use financial metrics. The Cost Variance is the simplest. It simply compares the Actual Cost of Work Performed to the Budgeted Cost of the Work Performed (CV = BCWP - ACWP). A negative amount indicates a project that is over budget, a positive amount indicates a project that is under budget. Earned Value is actually the EVM term for the Budgeted Cost of Work Performed (CV = EV - ACWP).
The Cost Performance Index, or CPI, is another indicator of a project's financial health. The CP is an absolute value in currency while the CPI is a comparator and can be used to compare a project's performance to budget in one period against another period, or against another project. The CPI is calculated by comparing the variance to 1, with 1 being exactly on budget. The formula for calculating the CPI is BCWP/ACWP. A CPI greater than 1 indicates a project that is under budget, a CPI less than 1 indicates a project that is over budget.
Burn rate is a third indicator used to indicate project performance to budget. The burn rate of the project is simply the rate at which the project budget is being spent. Faster than the plan? Slower than the plan? Or exactly to plan? The burn rate is the inverse of the CPI so the formula for calculating the burn rate becomes: Burn Rate = 1/CPI. A burn rate of greater than one means your project is consuming budget faster than planned and will exhaust the budget before all the work is completed. A burn rate of less than one indicates your project is consuming budget slower than planned and will complete before the budget is exhausted.
Performance to Schedule
Performance to schedule is the measurement of how quickly the work of the project is being done. Is it being done on time? Ahead of time? Behind time? Your project may be performing well to schedule but performing poorly to budget so the indicators used to measure cost performance cannot be used to measure schedule performance. Your project may be ahead of schedule because you spent budget on overtime work to achieve your head start.
Your MS Project file will be your only source for the metrics you need to measure schedule performance. The file should capture schedules in terms of time - hours, days, weeks, or months. The base measurement for schedule performance is Schedule Variance, or SV. SV can be calculated using monetary units (as dictated by EVM), or in units of time as long as you stick with one measure and use it consistently throughout the project. The EVM formula for calculating SV is BCWP - BCWS (Budgeted Cost of Work Performed - Budgeted Cost of Work Scheduled). You can use monetary units, hours, days, weeks, or months as units of measure for BCWP and BCWS. A positive SV indicates a project ahead of schedule and a negative SV indicates one that is behind schedule.
Schedule Performance Index, or SPI, is the schedule equivalent of the CPI and is calculated with the formula SPI = BCWP/BCWS. An SPI of greater than 1 indicates the project is ahead of schedule and an SPI less than 1 indicates the project is behind schedule.
Scope can be viewed through two different lenses. The alignment of the project to the original set of deliverables identified for it and the amount of time or cost of the approved changes to the scope. Show your project's performance to scope by showing the planned deliverables for the project, the deliverables created, and the cost of the new features approved for the project. Limit the list of deliverables to key deliverables and indicate whether they are planned or built.
Changes to scope can be shown as deltas to the project's planned scope. Show additional features and functions and their related costs and also show features and functions that were removed from the plan with their costs.
Quality performance can be measured in a variety of ways. Your principal source of quality metrics should be the issue or trouble tracking tool used to record bugs found by the QA (Quality Assurance) group. This tool should be capable of producing just about any report you care to ask of it and be able to break the recorded issues down according to the following categories:
- By severity
- By cause (software bug, data, question, etc.)
- By area or application
- By status (open, interim, fixed, closed, etc.)
- By owner (software development, database administrator, etc.)
- Re-opened tickets
There are two metrics that will be of interest when QA activities are underway: the number of tickets opened in a period and the number of tickets closed in a period. You may also want to report on the number of errors found per 1K lines of code, or any other indicator of the quality of the development work. Just remember that the ultimate goal of Quality Management is to produce a product that fits the clients requirements so reporting on a large volume of opened tickets should not be a cause for alarm. Many more tickets being opened than are closed in a period, or a large volume of tickets being re-opened may be cause for alarm.
The value of your scorecard will very much depend on the accuracy of the metrics it consists of. Use discretion when choosing metrics to report. Only choose those which you can verify. Take care in the capture and storage of the information you use; start by keeping your MS Project file accurate and up to date.
Your data probably won't be 100% accurate no matter how careful you are so inform your audience of your assessment of the data accuracy and of how the data is captured and retrieved for the scorecard report.
Do not attempt to create a scorecard report using only text. The report won't be read. Use a media that best supports the metric being reported. You may want to stick with the car dashboard/windshield analogy in which case a traffic light showing red (stop), yellow (proceed with caution), or green can be an effective visual aid to the overall project performance indicator. A speedometer showing a range of CPIs with the needle pointing to the current CPI is also a great visual indicator (if a little gimmicky).
Bar graphs are an ideal means to show historical information such as trends. Showing the project's current CPI or SPI tells the viewer if the project performed at, under, or over for the reporting period but showing a rolling window of 10 reporting periods worth of CPIs or SPIs tells the viewer the same thing, plus whether performance is improving or degrading. Trend lines will make the picture clearer and a line at the 1.0 index will show where the project should be performing.
Scatter diagrams are useful for showing the causal relationship between two variables, one of which is under the control of the experiment. The control, or independent variable is plotted along the horizontal axis and dependant variable is plotted along the vertical axis. This type of chart is useful in showing the results of a process change on a quality metric.
There are many off the shelf tool sets that will take the metrics you wish to report and turn them into a great visual show. You can use one of these tools and experiment with it's features, choosing a combination that suits your audience, or you can use the visual aids you have available to you to custom make your own.