The Myths of Return on Expectation

by Jack J. Phillips and Patti P. Phillips

Beware This Vague and Nebulous Term

Learning and development professionals routinely create new terms and jargon. While meaningful to them, the obscure vernacular is often confusing to key clients and executives. The most recent addition to the vocabulary is return on expectation (ROE). Let’s examine this term and relate it to what we know about current measurement systems.

What Is It?

What is ROE? Is it a number, a concept, or an objective? Some people suggest it is a number. We recently saw a report of a learning program that delivered an impressive 85.2 ROE. This gets attention, even from executives who are accustomed to the acronym “ROE.” However, business vernacular defines ROE as return on equity. This standard accounting measure indicates the return of shareholder investment in a company. Unfortunately, ROE in the above example was calculated as follows: Using a scale of one to 100, clients of the learning and development function rated their level of program satisfaction. The average score was 85.2; hence, the ROE. This was presented as data reflecting the impact of the program.

In reality, this calculation of ROE represents reaction data, Level 1 in the classic evaluation frameworks posited by Kirkpatrick and Phillips. Presenting reaction data using a familiar business measure presents an illusion of something that it is non-existent, reflecting unfavorably on learning and development. Finance and accounting staff chuckle at what they see as a misuse of the classic ROE acronym.

ROE could be a concept, suggesting that client expectation is being met along a variety of measures, such as usefulness, relevance, and value. Taking this measure of ROE is simply a matter of asking clients if they are satisfied with the program – a Level 1 reaction measure.

Perhaps ROE is an objective. Some suggest that ROE is based on achieving objectives or certain outcomes. If the outcome is productivity, quality, or sales, for example, the measure becomes results or impact, Level 4 under both Kirkpatrick and Phillips evaluation frameworks. If this is the case, why not call the outcome results or impact? If ROE represents an objective where the client sets an expectation about what participants should do, then the results represent behavior change or application (Level 3 on both Kirkpatrick and Phillips frameworks). If the client suggests that participants acquire certain knowledge or skills, the objective is the classic learning objective, Level 2 in both Kirkpatrick and Phillips frameworks.

Vague definitions leave decision-makers little basis for their decisions. However, definition is a minor issue when compared to how this ROE is developed.

How to Develop the L&D ROE

The learning and development definition of ROE is vague and its development follows an ill-conceived path. Some say the client develops the ROE entirely. This approach has two flaws. First, according to many learning professionals, managers and executives (clients) who request programs do not always know how to articulate specific measures of success. Clients may want the program to be “very effective”…what does that mean? Or “we want best-in-class managers”…again, not clear or definitive. Leaving this process entirely to the client often presents nebulous, misguided, or misunderstood expectations.

The other flaw in allowing the client to set the expectation is that the expectation may be impossible. Suppose that a client says, “I want 150% ROI!” Now the expectation is an ROI calculation. What would you do with that? The client may say, “We want to improve our sales by 100% in six months, something we have never achieved, but…”That may not be possible. Suppose the expectation is: “We want zero unplanned absenteeism in our call center” –not a realistic goal. The point is that having the client set the expectation sometimes yields an unachievable target.

The next approach is negotiation with the client toward an appropriate expectation. This becomes the return on the negotiated expectations (RONE). While this may be the best approach because it can yield specific, appropriate and realistic measures, why not classify the expectation in one of the classic five levels of evaluation rather than throw out another nebulous term?

We were recently involved in a situation where ROE went astray. A European broadcasting company spent millions of dollars on a leadership program. When the leadership development team attempted to define the expectation so that they could use the concept of return on expectation, the CEO stated, “My expectation is effective leader behavior. There is no need for you to collect data; I will tell you if my expectations are met.” Based on the limited parameters stated by the CEO, the team collected no follow-up data. The CEO was fired midway through the project and the new CEO asked about the status of the program. The leadership development team explained that they were measuring the program using thereturn on expectation as defined solely by the previous CEO. Now they were caught in an embarrassing situation as the new CEO facetiously suggested that the former CEO be brought back to evaluate the program. Frustrated, the CEO stopped the program and fired the leadership development team, stating that they had wasted a great deal of money. This extreme case demonstrates the risk of working from a nebulous expectation that is understood by one long-gone executive.

The Role of the CFO

Perhaps there is no more important influence on funding for learning and development than that of the Chief Financial Officer (CFO) and the finance and accounting team. Today the CEO expects the CFO to show the value of non-capital investments, which requires the finance and accounting team to be involved in our work. At the same time, many HR functions are now reporting up through the CFO, adding pressure to show value. Given the importance of this function, it is helpful to ensure that measures used to gauge learning’s success get their approval.

Don’t Mess With Finance and Accounting

The word “return” comes from the accounting field, most notably referring to the return on investment (ROI), a financial concept defined as “earnings divided by investment.” In the context of learning and development, ROI is net monetary benefits from the program divided by the cost times 100. This yields the ROI percentage. ROI positions learning as an investment. The accounting field also uses the word “return” in many other expressions, such as “return on equity” (ROE), “return on assets” (ROA), and “return on capital employed” (ROCE).

The concept of ROE raises a red flag to accountants, as it references fundamental financial terms. Compounding the confusion around measures of “return” are variations of return on expectation. These include return on anticipation (ROA), return on inspiration (ROI), return on information (ROI), return on involvement (ROI), return on client expectation (ROCE), and return on event (ROE). Even worse are return on training (ROT), and return on people (ROP). Some professionals have even used the concept of return on objectives (ROO), suggesting that this is a completely different process from measuring the success of objectives at different levels.

Figure 1 compares misused terms and the accounting perspective. The issue here is twofold. First, use of the word “return” piques the interest of finance and accounting, because it is the basis for many of their common measures. Second, misusing terms with which finance, accounting, and top executives clearly identify decreases our credibility. From their perspective, we are unwilling to show the actual value of what we do in terms that they understand. We are substituting a new term, hoping that they will see value in this.

Figure 1- Misused Financial Terms Term Misuse CFO Definition

ROI Return of information Return of intelligence Return on investment ROE Return on expectation Return on events Return on equity ROA Return on anticipation Return on assets ROCE Return on client expectation Return on capital employed ROP Return on people ? ROR Return on resources ? ROT Return on training ? ROW Return on web ? ROO Return on objectives ?

Who’s the Real Client?

Identifying the real client for a learning and development program is often a murky issue. The client funds the program and has the option to invest in other initiatives. This client is interested in the value of learning and development as expressed in terms that they understand, often leading to business impact measures and ROI.

For example, in a large, multinational organization, the centralized learning and development function develops programs used by the different business units. Each business unit has a learning and development advisor who serves as a liaison with the corporate university. From the corporate university perspective, the client is the learning and development advisor – their principal contact. From this perspective, the client is another learning professional. What this individual may view as value could be different from the business unit head who ultimately provides funding, allowing individuals to be involved in the program. This client is paying for the program through transfer charges and absorbing associated administrative and travel expenses.

In reality, the business unit head is the real client. If you ask that client about expectations from learning and development, you will receive a different description than the one you might get from the advisor. The difference comes from their perspectives. The learning and development advisor essentially sees this as their program. They have asked the corporate university to conduct the program, and they assume some ownership from that request. It follows that if the program does not deliver value, it could reflect unfavorably on them. This fear of results often forces them to use a vague measure that no one understands. It presents an easy way out and avoids the risk of the program not delivering the value that the business unit head desires.

Focus on Business Contribution

Most executives want to see alignment with the business needs, and learning and development success expressed as a business contribution. In a recent survey sponsored by ASTD of Fortune 500 CEOs, top executives weighed in on the types of data that matter to them. The number-one measure CEOs want to see is the connection of learning and development to the business (Level 4 Business Impact). Ninety-six percent of CEOs responding to the survey want to see these data; however, only eight percent receive the data. In the same study, seventy-four percent of the top executives desired to see the ROI from learning and development. Only four percent are seeing it now.
The gap in what CEOs want and what they receive presents a challenge. Learning leaders must meet the expectations of executives who ultimately fund learning and development functions. Without their commitment and funds, learning and development would not exist as a formal process. The term, technique, or process used to measure success must be defined by contribution meaningful to the real client.

Back to Basics

An easy way accomplish business alignment is to consider objectives at multiple levels. Learning objectives are developed with performance-based statements, and sometimes include a condition or criterion. However, from the client perspective, these objectives represent only learning; there are other important levels of objectives. Application objectives (Level 3) clearly define what the participants should do with what they learned. Examples of Level 3 objectives include: • At least 99.1 percent of software users will follow the correct sequence after three weeks of use • The average 360◦ leadership assessment score will improve from 3.4 to 4.1 on a 5-point scale in 90 days • Ninety-five percent of high-potential employees will complete all steps in their individual development plans within two years • Sexual harassment activity will cease within three months after the zero-tolerance policy is implemented • 80 percent of employees will use one or more of the cost-containment features of the healthcare plan in the next six months • By November, pharmaceutical sales reps will communicate adverse effects of a prescription drug to all physicians in their territories.

Impact objectives specify what the application will deliver in terms of business contribution. These Level 4 impact measures communicate the consequence of application, usually defined in the categories of output, cost, and time. Examples of Level 4 objectives include: • The Metro Hospital employee engagement index should rise by one point during the next calendar year • There should be a 10 percent reduction in overtime for front-of-house managers at Tasty Time restaurants in the third quarter of this year • After nine months, grievances should be reduced from three per month to no more than two per month at the Golden Eagle tire plant • Sales expenses for all titles at Proof Publishing Company should decrease by 10 percent in the fourth quarter

Impact objectives connect the program to the business. In some cases, ROI objectives are set and expressed as a benefit/cost ratio, and ROI as a percent.

Defining expectations and developing objectives that link to meaningful business measures positions any learning and development program for results that resonate with all stakeholders, including the real client.

The Final Word

There is no need for another ambiguous term that creates more confusion. Return on expectation, return on anticipation, return on client expectation – generate meaningless measures and risk misunderstanding among the clients funding our programs. These terms do little to satisfy executive interest in learning and development programs, when their focus is business contribution to the organization. Learning leaders must step up to the challenge and avoid the temptation to grasp trendy jargon or techniques that sound appealing, but do little to demonstrate the real value of learning and development.

The Authors

Dr. Jack J. Phillips, a world-renowned expert on accountability, measurement and evaluation, is co-founder of the ROI Institute. Phillips provides consulting services for Fortune 500 companies and major global organizations and presents at conferences throughout the world. His work has been featured in the Wall Street Journal, Businessweek, Fortune magazine, and on CNN and a dozen other television programs. ASTD gave him its highest award, Distinguished Contribution to Workplace Learning and Development for his work on ROI.

Dr. Patti P. Phillips is President and CEO and co-founder of the ROI Institute, Inc., the leading source of ROI competency building, implementation support, networking, and research. A renowned expert in measurement and evaluation, she helps organizations implement the ROI Methodology in over 40 countries around the world. Together, Jack and Patti have edited or authored over 50 books. They can be reached at info@roiinstitute.net.