Using ROI to Evaluate Project Management Training

Case Studies | By Gina Abudi | Read time minutes

Business workshop and presentation

Return on Investment (ROI) is a monetary measurement that is used to evaluate the efficiency and effectiveness of an investment made by an organisation. Investments take many forms, financial, human capital, equipment, and training programmes, to name just a few. This article will focus on the use of ROI and the Phillips ROI Methodology™ to measure the effectiveness of a project management training programme completed within XYZ Law Firm. The 5 levels of evaluation which will be reviewed within the case study are:

  • Level 1: Reactions (the "smiley" sheet): Did participants like the training they received?
  • Level 2: Learning: Are participants confident that they have learned something from the training programme?
  • Level 3: Behaviour/Application: Are participants able to apply what they learned in the training programme back on the job?
  • Level 4: Results/Business Impact: Did the training show improvement in efficiencies, productivity, profits, costs, reduced turnover?
  • Level 5: ROI: Did the training programme show a positive ROI?

Why the Interest in ROI for Project Management Training Programmes?

The challenges surrounding training and its effectiveness within the organisation have become more complex over the years. Today, the challenge is even more significant for learning and development professionals. Return on Investment (ROI) as a tool for evaluating project management training is becoming an expectation of senior executives within organisations. In today's tight economy with reduced resources and tighter budgets, learning and development professionals are finding it increasingly necessary to show the monetary value of the project management programmes they are bringing to the organisation.

Today, the success of project management training programmes is measured by the financial contribution of the programme to the organisation. It is not surprising then that ROI measurements for project management training programmes are often requested. With such a large focus on project management in all kinds of organisations, there is an increasing desire to show the monetary benefit of investing in project management training programmes which often reach a wide variety of employees and represent a significant investment for organisations.

The case study example below will follow the Phillips ROI Methodology™ and encompass the four major phases of the methodology:

  • Evaluation Planning
  • Data Collection
  • Data Analysis
  • Reporting

Case Study Background: XYZ Law Firm

XYZ Law Firm was interested in developing a project management mindset within the firm. There was support from the managing partners and funding was expected to be approved. The goal of the training was to ensure that completing projects at the law firm was not a severe drain on resources. There was much anxiety behind any project which needed to be completed and the champions of the training programme, which included a few of the partners, attorneys and the paralegals, were confident that with some basic project management training, this anxiety could be controlled and projects done within the firm would be more easily managed. The whole concept of "project management" scared individuals.

The managing partners of the law firm specifically wanted to be updated on project status and they often felt they were unsure what projects were being worked on and how they were progressing. A significant goal was to ensure implementation of best practices and standard processes and increase knowledge sharing.

Overall the goals of the training programme included:

  • Increased likelihood of successful projects
  • Ability to implement strategic plans into action
  • Improved monitoring and controlling of projects
  • Proactive risk management
  • Improved time management and teamwork
  • Efficient utilisation and tracking of resources
  • Standards around status reporting
  • Implementation of best practices and standard processes

Some of the projects that the law firm worked on included:

  • IT/infrastructure projects
  • A variety of projects for their corporate clients, such as mergers and acquisitions and major contract negotiations
  • Annual recruiting projects, which included recruiting from law schools worldwide
  • Long range capital strategic projects

Many such projects took longer than originally planned for, went out of scope and went over budget and resource commitment. This was certainly a problem for the client projects the law firm worked on as such projects were under fixed price agreements with the client. If the project went off track, it would impact the profitability for the law firm for that particular engagement.

Phase 1: Evaluation Planning

Based on the goals of the law firm, a three day customised basics of project management training programme was developed. Prior to the start of the programme, data was collected on past project initiatives, specifically around:

  • Budget
  • Time
  • Scope
  • Resource utilisation
  • Quality

Much of this data was tracked, although it was not in one central location and required conversations with many members of the law firm, from managing partners through to paralegals and administrative assistants.

In the first phase of the project, we developed a plan around the data we would be collecting. The plan included the following components for Level 1-4:

  • Level 1: Satisfaction survey to participants
  • Level 2: A test administered to participants to measure the learning from the 3-day programme
  • Level 3: Action plan follow-up and questionnaires including information on how the participants were applying the skills they learned. Level 3 questionnaires also went to managing partners and senior attorneys to get their feedback and perception of the skills being applied
  • Level 4: Collection of all data for business impact

We also developed an ROI Analysis Plan to include the items we intended to measure against to show improvement based on the 3-day training programme. The data items to be collected at level 4, which are the impact measures targeted for improvement included:

  • Improved performance on projects, including improved quality of end result
  • Efficient utilisation and tracking of resources
  • Increased percentage of successful projects (on time, within budget and within scope)
  • Implementation of best practices and standard processes

We set the ROI (hurdle) rate at 20%.

Phase 2: Data Collection

A pilot group of 20 junior attorneys, paralegals and administrative assistants were selected to go through the programme first and measure the benefits of the training. In this way, an isolation factor, an important component of doing an ROI study, was available to compare against a similar group of attorneys, paralegals and administrative assistants who would work on projects also, but would not initially attend the 3 day programme.

The following data was collected from the programme:

  • Level 1: 95% of the participants were satisfied overall with the 3 day customised programme
  • Level 2: 100% pass rate on the exam (85% or higher was considered passing). It should be noted that the exam was not an overly difficult exam and as long as the attendees participated and paid attention in the 3 day class they would be able to pass the exam. Frankly, the real measurement was in the Action Plans that were produced
  • Level 3: 100% of the participants submitted Action Plans and completed follow up questionnaires. Action plans were completed after the class and followed up on within 6 months. The action plan included information on:
    • Participant's goals (individual improvement efforts)
    • How participants intended to meet those goals:
      • Action steps to be taken
      • Support and/or resources needed
      • Timeline for completion
    • How participants will know if they are successful in meeting those goals
    • How participants will evaluate that success
  • After the 6 month time period, participants were asked to complete a Continuous Improvement Plan for moving forward. This follow up to the Action Plan included:
    • Results/accomplishments to date
    • Next steps to accomplish
    • Timeline for doing so and support needed
  • Level 4: Data collected from participants, managing partners and senior attorneys on the impact to the business. The data collected from level 4 showed a 40% increase in the number of successful projects. Successful projects went up from 70 successful out of every 100 (smaller projects included in the 100 along with more strategic projects) to 98 successful out of 100.
    • 40% increase in the number of successful projects:
      • On time, within budget and scope
      • Projects met set quality standards
      • Resources tracked and effectively allocated
      • Standard processes implemented for all projects
      • Lessons learned captured and best practices shared

Phase 3: Data Analysis

In converting the data collected to a monetary value, we found that cost savings (programme benefits) were as follows:

  • More efficient use of resources: $285,000
  • Projects on time, within budget and scope: $250,000
  • Increase in better tracking of client projects: $450,000

Programme benefits are measured in level 4: Business Impact. To convert to a monetary value, a dollar amount was put to every resource based on salary/time/fringe, etc. If projects did not go out of scope, no additional resources were needed, which was a cost savings. More efficient use of resources also meant that projects had fewer people assigned to them. The $450,000 in increase from better tracking of client projects was the amount that would have been lost from the bottom line had those projects that were measured not been completed successfully.

The total cost of the programme was $320,000. This included planning for the ROI study, food and beverage for participants in the programme, facilitator costs, development of the customised programme, time out of the office for training, salary/fringe of participants, etc. Any cost associated with the programme or planning for the programme was included in the total costs. The biggest component of the cost was salary/fringe of participants, especially since the attorneys and paralegals are billable resources.

Intangible benefits, those benefits that did not have a dollar value associated with them, were also considered. They included:

  • Reduced anxiety over involvement in and managing projects
  • Improved teamwork and reduced conflicts
  • Satisfaction of managing partners and senior attorneys over status reporting on projects

These intangible benefits were able to be tied back to the programme and certainly contributed to the success of the projects.

During phase 3, we calculate the ROI of the project management training programme, using the ROI formula:

ROI = Net Programme Benefits / Programme Costs x 100

  • ROI = $665,000 / $320,000 x 100
  • ROI = 2.08 x 100
  • ROI = 208%

For every $1.00 invested, $2.08 is returned after the costs are recovered. This was significantly over the 20% ROI set at the beginning of the project during the planning of the study.

Phase 4: Reporting

A report of the results of the study was presented to the managing partners of the law firm. Components of the study were presented to the participants and others in the law firm. The project was considered a success and it was decided that the 3 day customised project management training programme would be rolled out firm-wide. Additionally, a half day programme was created for the managing partners of the firm to provide them training around how to support project initiatives. A future focus was to develop a portal to house all project information. That was a "to do" for the IT/Application Development department and would be a key strategic project for them.

Summary

For training and learning departments, the PMO, or lines of business; showing the ROI of the project management training programmes conducted within the organisation, whether in-house developed programmes or vendor-provided programmes, enables you to show the value the programmes have on the business in monetary terms that executives can understand, respect and champion.

The six types of data collected during the Phillips ROI Methodology™ process (see list below) allow consistency in the measurement of project management training programmes, thereby bringing credibility to the process:

  1. Reaction and value of programme
  2. Learning and confidence
  3. Application and implementation
  4. Business impact
  5. ROI
  6. Intangible benefits

To begin measuring your project management training programmes, you might start with a brand new programme or review the current programmes that are in place, following the criteria for measuring project management training programmes discussed earlier in this article. If you review a current programme, gather data on the initial purpose of the programme when it was first developed and implemented. Does it still meet that purpose? What value do you perceive the individuals taking the project management training programme are achieving? How much has been invested in the programme since the beginning? Should this programme remain in place "as is," or should it be updated, or possibly be eliminated from your offerings in its entirety or in part to accommodate new needs among your project managers? An ROI study will enable you to look at the value of your current programmes and enable you to better plan new programmes.


References

  • Phillips, J. and Phillips, P. (2007) Show Me the Money: How to Determine ROI in People, Projects and Programs, San Francisco: Berrett-Koehler Publishers, Inc.
  • Phillips, J., Phillips, P., Stone, R., and Burkett, H. (2007) The ROI Field Book: Strategies for Implementing ROI in HR and Training, Massachusetts: Butterworth-Heinemann/Elsevier.

Gina Abudi has over 15 years consulting experience in a variety of areas, including project management, leadership, and development of strategic learning and training programmes. She blogs at: https://www.ginaabudi.com/


Recommended read: Predicting the ROI of Change, by ExecutiveBrief.

What's Next?

You may also be interested in