Questions About ROI in Today's Climate

by Jack J. Phillips

The following responses are provided to questions posed to Dr. Jack Phillips by MagazynHR.

1. Jack, as the father of the ROI Methodology, how do you assess the general position/ condition of business nowadays?

JJP:  In general, the global recession affected almost every organization in every country. While some survived it quite well, others did not, leaving them in a severe economic situation.  From all indications, the recovery will be slow, taking perhaps two to three years before most countries will be back to business as usual.  
This recession has had three significant impacts on the use of ROI:

a) The use of the ROI Methodology has intensified during this global recession, as organizations of all types have used this methodology to decide which programs to eliminate, which to keep, and which to fund in the future. b) The recession has inspired a resurgence in the need for ROI capability in organizations. We have conducted a record number of ROI certifications in the last two years. c) As more organizations come out of the recession, key executives are demanding ROI up front, before a project is implemented. Because there is a need to avoid wasteful spending and unnecessary expenses (to keep the organization lean), executives are asking for a forecast of ROI in advance. Collectively, these consequences will mean that ROI will be a way of life in the future, even in organizations where it has not been previously pursued.

2. How did you conceive the idea of ROI?

JJP: The ROI Methodology had its beginnings in 1973, as I wrote my thesis for a master’s degree in decision sciences at Georgia State University. I conducted a major study of the ROI on a cooperative education program at Lockheed Martin. This study was initiated by senior executives who were asking for the value, even in those days. From that beginning, the methodology has been improved, adding a system, a step-by-step process, and standards along the way. Our first book on this methodology was published in 1983, and this became the first book for training evaluation in the USA. For more details on the history of the ROI Methodology, a document has been provided and is available to requestors at www.roiinstitute.net.

3. When ROI is used to assess people management effectively, in what types of organizations is it most effective, and in which branches?

JJP: Historically, ROI is most effective in private businesses where there is an important concern for results and accountability.  It has been most popular in organizations where there is a specific requirement for efficiency and intense competition for funds for projects.  It is also very popular in fast-changing industries that are constantly striving to be the best they can be.  
    Within an organization, it is more popular with the human resources area.  Within this area, learning and development teams are using it more than other HR functions.  This is because in many organizations, learning and development is considered to be a disruptive process, taking people away from work.  Consequently, there is pressure to show the value of learning and development.  Beyond the human resources area, ROI is probably most used in technology and marketing.  In marketing, it has been used less for meeting and events, but this usage is gaining in popularity.   Quality processes are using it moderately.  The parts of an organization less likely to use it would be public relations, government relations, and the communications function.

4. Are there any business models where ROI is the most effective or fits best?

JJP: When the business model focuses on results that are measured clearly, it works much better. When a culture of measurement and accountability is in place, ROI is easily implemented. This occurs more in private businesses, which for the most part have been subjected to greater accountability than public sector organizations, nonprofits, and non-governmental organizations. This is not to suggest that ROI does not work in the non-business type of organizations; in fact, most of our growth is in that area. More than half of our work is now in the public sector, NGOs, and nonprofits. Those organizations are now focused on results and accountability more than in the past, and it will take a long time to change the culture to support ROI.

5. What kinds of differences in using ROI do you observe in different parts of the world?

JJP: The good news is that ROI works in any culture or country.  We are now enjoying implementation in 58 countries, with Lebanon becoming our latest to fully implement through a partnership arrangement.  There are some differences, however, from one country to the other.  The principal difference is the trust that occurs in organizations and is often a factor in the culture.  In some cultures, individuals are willing to provide accurate and reliable data, and there is a focus on accountability, as in Germany, for example.  There are other organizations where the individuals do not want to accept this level of accountability, and will not provide adequate or reliable data.  In third-world countries, ROI becomes more difficult because of the lack of infrastructure and focus on measurement and accountability.  It is, however, working in some of those countries, as we have experienced in Indonesia.  
    The success of the ROI implementation in a country often depends on the size of the country (smaller countries seem to be more successful), the capabilities of our partners, and the support of the government.  In those countries where the government has been a sponsor or supporter, implementation occurs more rapidly, as in the case of Ireland.  It helps if businesses, governments, and educational institutions can work together to implement ROI.

6. In the area of people management, ROI is most famous for measuring training results. Is this really the most powerful area in which to use it?

JJP: You are correct.  The ROI Methodology is most famous for measuring results of learning and development programs.  However, there are many other possibilities that make it powerful in other areas.  For example, technology commands a much larger budget than learning and development, yet it is unusual to see a consistent, structured process like the ROI Methodology in place to measure the impact of technology.  Also, across the marketing function, which includes meetings and events, the expenditures are quite large, but the evaluation processes used are very inconsistent and sometimes not very credible.  The ROI Methodology can help.  In the green movement and sustainability initiatives, there is almost no consistency in measuring the success of those types of projects, yet the success of these projects is often questionable.  The methodology is desperately needed in those areas and can have a tremendous impact.  Fortunately, we have books and case studies in these areas.

7. Is it really possible to isolate the training effects?

JJP: Yes – that’s the short answer – it is possible, not only for training, but to isolate the effects of any process or program.  Whether it is a new technology initiative, an HR program, a meeting or event, a system, or a policy – the effects of the process can be isolated.  There are several ways to do this.  Perhaps our best contribution to this field is to show the different ways in which this can be accomplished.  There are very research-focused approaches, such as the experimental versus control group, trend-line analysis, and forecasting, which is more of a mathematical model.  Then there are methods that involve estimates from the most credible sources, such as participants, managers, experts, and customers.  These can be very accurate and credible, particularly if a step is taken to remove the error from the estimate.  
    In our methodology, we use estimates only as a fall-back position when nothing else works.  We then ensure that we collect it from the most credible source, and we collect the data in a very non-threatening, unbiased way.   Finally, we adjust for the error of the estimate.  The important point is that when approached properly, even the default position (estimates) is very credible and will be accepted by top executives, even the chief financial officer.  
    We have accumulated a tremendous amount of research showing the accuracy of these types of estimates, but we prefer not to use them.  If we must use them, however, we are prepared to defend them.  In the ROI methodology, one of the standards is to always isolate the effects of the program. Without it, the study will not be credible.

8. Is ROI only for mature organizations?

JJP: No. ROI can work for any type of organization. There are some advantages of using ROI to a growing organization that has not yet reached maturity. This allows the organization to build in some very effective processes rather than wait until the organization is mature and somewhat resistant to change. The ROI Methodology requires a change in our thinking. It requires a change in the way in which we initiate programs and projects. It changes the way in which we develop objectives, pushing them to Level 3 and Level 4, application and impact. It changes the way in which we secure commitment to provide data. It changes the way in which we share the responsibilities for measurement and evaluation, and it changes the way in which we use the data to drive improvement and significant shifts. Change is often more difficult for more mature organizations to accept than one that has not reached its maturity.

9. How can organizations prepare to implement ROI? Should it prepare to implement ROI?

JJP: Preparation is the key.  Preparation starts with building capability, the first step.  Every member of the HR team (where the ROI Methodology is used) should have some ROI training.  A few individuals should develop very essential competencies, and this is achieved through ROI certification.  This is a week of training, plus completion of a project to reach the designation of Certified ROI Professional (CRP).  Others will need one- or two-day workshops to understand this process.  When the team understands ROI, the resistance is lowered.  
    The second step to preparation is to conduct studies and use the results to make improvements. This shows executives and sponsors the value of the ROI Methodology.  Studies should be conducted at the application, impact, and occasionally at the ROI level.     The next step is to develop tools and templates, and use them to keep costs and resources at a minimum.  Certified ROI Professionals have access to a variety of tools and templates at no charge.   
    Finally, the team must brief managers and show them how the Methodology is being used.  The ROI Methodology is accepted by the CFO and CEO, as well as other executives who are seeking accountability at this level.  Communicating to them the actions being taken and the reasons behind them is a requirement.

10. What are the most frequent barriers to implementing ROI in people management, and how can we overcome them?

JJP: The number-one barrier is the fear of the results, particularly at the ROI level.  There is a concern that if the program is not working, the ROI study will reflect unfavorably on those who have developed, implemented, or facilitated the program. (This can be overcome if a proactive approach is taken to pursue ROI and not wait for the request.  This should be pursued in the spirit of process improvement, not performance evaluation.)
    The second barrier is perceived complexity of the process.  In reality, this is a very straightforward, logical process.  The ROI Methodology does not contain many statistics or significant finance and accounting concepts.  It is business evaluation and consequently is not very difficult.       
    The third barrier is the perceived costs for the ROI process. Costs can be managed, adding only a small amount to the expenditures for measurement and evaluation.   A best practice profile of the use of the ROI Methodology can be attained for about 3%-4% of the learning and development or HR budget, depending on the function being evaluated.
    The fourth barrier is the time.  Individuals often do not have time for more evaluation; they are too busy implementing programs. (It will take additional time, but the time can be shared among others.  This is important enough that additional time must be allocated to this, even if it means taking time away from issues such as design, development, and delivery.
    The fifth barrier is that people do not know how to do this. They do not see the connection, or do not have the skills or knowledge to make it work. (This can be overcome with workshops, books, guides, tools, templates, and case studies, all of which are available with major publishers and through the ROI Institute website {www.roiinstitute.net}.)

11. What is the most difficult part in the implementation of the ROI Methodology?

JJP: The most difficult part of implementation is to overcome all of the fears of additional accountability from all stakeholders.  This is essentially a change management process, and to change habits means that we must work with the team.  There is a fear of the outcomes, and we have to alleviate those fears through education, involvement, and communication.  Above all, ROI must be pursued in the spirit of process improvement.

12. There is an opinion that ROI is very time-consuming and is not worth using to evaluate smaller programs or projects. Is that true?

JJP: The use of ROI can be time-consuming, but the time can actually be managed by taking shortcuts, using tools and templates, and sharing the responsibility. Because of this, it should be used sparingly, with only a small number of programs taken to the ROI level. Our best practice recommendation suggests that 100% of programs should be evaluated at the reaction level (Level 1), 80% at the learning level (Level 2), 30% of the programs at the application level (Level 3), 10% at the impact level (Level 4), and 5% at the ROI level (Level 5). This means that 30% of the programs are evaluated at application, one-third of those are taken to business impact, and half of those are evaluated at the ROI level. These targets keep resources at a minimum, and focus only on those programs that should be evaluated at those levels. For business impact and ROI, the programs selected for evaluation are very important, strategic, expensive, and consume many resources.

13. Are there any alternatives to ROI?

JJP: That is an interesting question. First, some people advocate the Kirkpatrick levels, which are actually levels that are definitions and are included in our process – that is not alternative, but the level is already built into ROI. There are others who suggest a case method approach developed by Brinkerhoff, which is essentially collecting data on the most successful individuals, usually at Level 3 and sometimes at Level 4. This approach is already built into the ROI Methodology. We also collect data from those that were not successful so we can understand how to improve the results.
At the ROI analysis level, there is no significant alternative to ROI. Depending on the country, some people claim they would like ROI analysis, but it is viewed as too complex or not credible. Consequently, it is not frequently used. The ROI Methodology has been evolved and developed over two decades, and is now the most-used evaluation system in the world. Over 4,000 organizations are using it, including the United Nations, which has become the largest client of the ROI Institute for three years in a row.

14. What is the difference between ROI and training evaluation at Level 4 of Kirkpatrick’s model?

JJP: First, the Kirkpatrick levels are included in the ROI Methodology as the first four levels.  In the 1980s, we added a fifth level, ROI, which is the ultimate level of evaluation. The Kirkpatrick levels are just definitions – there is no system, no process, and no standards.  It is a very elementary view of evaluation.  While some review the Kirkpatricks’ levels before pursuing ROI, it is not necessary because everything in the Kirkpatrick levels is built into the ROI Methodology.

15. I want to ask about ROE. What does that really mean?

JJP: Unfortunately, the term ROE has been used in the human resources, learning and development, and the meetings and events fields. In that context, some users, including the Kirkpatricks, call it return on expectations. For most definitions, it is really a reaction measure, basically asking the client, “Did we meet your expectations?”, i.e. are you satisfied with the program? We suggest calling it reaction if that’s what it is. If it is something else, it can be easily defined as learning, application, impact, or ROI. We suggest that those terms should be used as they are more generally accepted.
The problem is amplified by the Kirkpatricks’ use of the term ROE, suggesting it is an impact measure. That is not the case. Using a vague term does more harm than good when we present results to senior leaders. They clearly understand what the measure means when they look at the definitions, so we suggest not using the term. The accounting profession thinks of ROE as “return on equity”, and we are actually deceiving them as well. I suggest avoiding that term altogether, and use the terms reaction, learning, application, impact, and ROI. These will resonate quite well with all stakeholders. (We have recently published two articles on this issue and they are on our website. They provide more detail about the return on expectations, and how it is flawed concept.)

ROI AND HR

16. ROI in human resources is still not a very popular concept in Poland. What is the best method to sell the idea there?

JJP: The same situation exists in other countries. The HR executives have often avoided the ROI process because they fear the outcome. Also, they cling to the concept that HR is essential, and therefore it should not necessarily be evaluated at the business level. A third concern is that the HR team is often waiting for management to request ROI. This is not a sound process, because waiting for such a request causes loss of control of the ROI implementation. Then we are on an executive’s timeframe and agenda – not the most comfortable place to be. Perhaps the best way to convince the HR team is to show the benefits of ROI. The ROI Methodology is helping clients in a variety of ways, including the following: • Aligns programs to show business needs • Shows contributions of selected programs • Earns the respect of senior management/administrators • Builds staff morale • Justifies/defends budgets • Improves support for programs • Enhances design and implementation of programs • Identifies inefficient programs that needs to be redesigned or eliminated • Identifies successful programs that can be implemented in other areas

It will also require educating the HR executives and team to understand ROI, and how it can make a difference. It is important to alleviate fears and dispel the myths of ROI. (There is a document on the ROI Institute website called, “The 25 Most-Frequently Asked Questions.” It may be helpful to review the issues captured in this article in addressing this issue.)

17. What are the most common obstacles reported by HR people during ROI implementation?

JJP: The most significant obstacles are the time to develop capabilities with ROI, and the shift in thinking about HR programs.  The use of ROI basically requires moving HR from an activity-based process to a results-based process, and that often changes many parts of the process.  ROI implementation represents a significant change program.  It will take additional time, resources, and the involvement of HR executives.  It is critical for the HR executives to be involved; if they are not, the ROI implementation may not work.

18. Jack, in your opinion, are there management areas that cannot be measured?

JJP: No. Anything can be measured – any process, initiative, or activity. A phenomenon can be measured, even if it is a perception. At times, there is a concern that perception data is not credible, but it can be if it is collected from the most credible sources and steps are taken to ensure quality and quantity of data. For example, customer satisfaction and job satisfaction are two very important perception data categories. If it can be measured, then it can obviously be evaluated at all five levels.
The difficulty with this process is converting the data to monetary values. We take the position that anything can be measured and can also be converted to money. However, the conversion process may leave a value that is not credible, or it took too much time to convert the data to money. We define an intangible as a measure that we cannot convert to money because it is either not credible or took too many resources to do it. We are now making progress in measuring functions such as public relations, change management, government relations, research and development, communication, and many other initiatives. Any functional area can be measured, even if it is simply a new policy implemented.

19. Are there situations where we should not implement ROI?

JJP: Very small companies will usually not have the resources or the concern to measure the impact and ROI of various projects and programs.  Although they may need it, it is not to the point where they are concerned about measurement issues. Therefore, small organizations do not often pursue this.  We do, however, recommend that small businesses use some of the concepts of the ROI Methodology to ensure that the projects are aligned to the business and have the proper focus.
    At the individual program level, programs that are very inexpensive and involve only a few individuals may not require ROI.  This often eliminates compliance or regulatory programs - even short e-learning programs are often excluded.  If a project or program is absolutely necessary for the job, i.e. a software tool or new skills, then it should not necessarily be measured at the ROI level.  The important point is to take a sensible approach to the use of ROI.

20. We sometimes hear that ROI puts too much emphasis on effectiveness rates and leaves no place for the human aspect. What do you think about that?

JJP: We agree that the ROI calculation alone may do just that.  However, if you remember our process, we capture six types of data- reaction, learning, application, impact, ROI, and the intangibles.  These data types are arranged in a chain of impact, as people react, learn, apply, and have an impact.  It is the impact measures that are converted to money and are compared to the costs of the program to generate ROI.  It is all about the people - what they have accomplished, how they are reacting, learning, and using the process.  It is a balanced set of data that fully accounts for the people side of the process. This makes ROI useful, intriguing, and flexible enough to use in so many different types of programs. We should never just evaluate a program solely on an ROI calculation.

HR

21. What recent trends have you observed in the field of human resources?

JJP: HR is changing dramatically. It used to be a very activity-oriented process, with many transactions involving collecting information and processing forms. It was a necessary activity. These days, many of the transactions are performed using technology, being shifted to other parts of the organization, or are outsourced altogether. What remains in HR are coordinators or managers who manage processes, or consultants who advise operating managers on HR issues. We have moved from an activity-based process to a results-based function where we are leveraging our expertise through a consulting process.
HR continues to be de-centralized, providing support and services closer to their clients. The days of a larger, central HR function are gone.
HR executives are being forced in some cases to show the value of what they do. The very progressive HR managers and executives are taking a proactive role in showing the value of all HR initiatives. After all, the largest single expenditure in many organizations is people. If HR cannot show the contribution of those people in credible ways, the function is in jeopardy of losing budget. As some experts say, the HR team may not continue to have the right to manage HR if they cannot do it with accountability and business contribution in mind.

22. What competencies should HR specialists have?

JJP: Because of the changes outlined in the previous question, this requires some new HR competencies.  These include competencies around problem-solving, consulting, negotiations, measurement, analysis, and communication.  We can no longer join the HR team because we like people; we must join the team because we want to show that people make a difference in the organization, and we are willing to help make it happen.  If we really believe that people are our most important asset, we must show the return on that asset. That is the financial ROI that can be developed with the ROI Methodology.

To view all the questions and responses, please click here.

Article Details

Questions About ROI in Today's Climate

Published in 2011 in MagazynHR Poland

Read Article


© 2014 ROI Institute
PO Box 380637
Birmingham, AL 35238-0637
205.678.8101 phone
contact us

.