The cost/benefit formula is:Looking before, during, and after you leap:
Cost analysis and Return on Investment (ROI) for distance education
(Click here to download ROIDE.rtf file)
Kent L. Gustafson
Lynne Schrum
The University of Georgia
Introduction
Distance education (DE) has been implemented in one form or another in every state and province in the US and Canada and in many countries around the world. Reasons for implementing DE range widely, including:
• meeting the needs of learners who otherwise might be underserved (Australian DE programs for school-age children in remote locations)
• creating new markets (advanced degree programs via web-based courses in higher education)
• reaching large numbers of learners with varying interests, backgrounds, and needs (computer training for teachers via the Internet)
• quickly distributing time-sensitive information (new product training via teleconferencing for sales personnel in the computer industry)
• creating a sense of community for geographically dispersed learners (discussion listservs to support university courses serving commuter populations)
• addressing the scarcity of qualified instructors (environmental monitoring and remediation training by a single expert to a national and ever-changing audience)
• increased consistency and quality of instruction (short course by a department of human services on interpreting state regulations for new social workers)
• providing "just-right," "just-in-time" training (systematically designed and tested course on changing tax regulations).
While each of these reasons may have its own social, moral, or legal justification, several of the above examples (and many others) would lend themselves to some form of cost and benefit analysis. An analysis can take different forms and might be limited to costing alone or might involve more complete analysis, possibly including calculating Return on Investment (ROI). It is generally advisable to do costing and ROI analysis: before a project is started; while it is underway, and at its conclusion (or at least after it has been fully implemented).
Consider the example of a DE project that will be looking at ROI at all three of these times that is designed to provide executives a way in which to complete an advanced degree while maintaining their workload and travel schedules.
Today many business organizations face a difficult and costly situation: retaining competent workers while encouraging them to pursue MBA degrees so as to successfully compete for higher positions in the company. Currently, many companies sponsor a valuable employee to return to school; however, they typically lose the services of that employee for the two years of schooling. To address this challenge a large financial corporation recently joined with a large southeastern university College of Business’s MBA program to use distance learning technologies to create a combined model uniquely suited to their high level workers’ needs.
This corporation was prepared to invest significant resources in the design and development of this tailored program with the understanding that it would become a model to assist many of their current and future executives. Once the concepts and coursework had been designed, the corporation assumed it could be replicated for others. Faculty were invited to participate in the creation of this integrated, multiple-technology, combined format, online MBA program. The project was designed for a cohort of 54 successful executives, to complete the MBA in an intense two-year format. Substantial resources were devoted to a technological support team to assist faculty, design materials, and support students. These costs, although substantial, were considered to be "start-up" with the pay-off coming from retention of quality employees.
A plan has been developed to look carefully at all costs at the end of the first year, in order to gauge the true costs. For example,
• Were participants able to continue their full job responsibilities as intended?
• Were estimates of incidentals (phone, fax, materials) too high or too low?
• Were peripheral costs (to family or health, for example) too costly?
At the end of the first cohort’s two years, an extensive evaluation of all costs is planned. Similarly, the costs to the department and faculty will be calculated. The determination of the continuation of this experiment will be based on many factors, but accurate financial data will be a significant consideration .When thinking about doing cost and benefit analysis of distance education programs it is generally easier to examine costs than benefits. However, even the relative lack of solid costing data for many DE efforts is disturbing. If the promised rosy scenarios of a future overflowing with individually tailored, high quality, widely available DE, are to become a reality, better costing and benefit analysis will have to take place. Society simply will not continue to increase its expenditures on DE unless at least some meaningful data become available to justify the costs. This is particularly true in industry where the very existence of training departments may well depend on their ability to demonstrate their contribution to critical business objectives and corporate bottom line (Phillips, 1998; Mager, 1996). We believe educators also would be well served by carefully analyzing the estimated and actual costs and (to the degree possible), benefits of DE initiatives (Gustafson & Watkins, 1998).
Fortunately, some forms of analysis are relatively easy to do while others require a bit more expertise and effort. This chapter describes a number of common approaches to cost and benefit analysis and describes how they might be used in specific settings.
Types of Cost and Benefit Analysis
Although there are many different ways to conduct cost and benefit analysis, we will discuss only five that are particularly useful when planning distance education programs. Keep in mind that the specific terms we use are sometimes used in different ways by other authors. Therefore it is necessary to understand the underlying concept being described and not assume, for example, that all writers on ROI mean exactly the same thing. Five analytic methods are described including: full costing; cash flow per unit of time; break-even analysis; cost/benefit ratio (benefit/cost), and return on investment.
Full Costing
Full costing by itself does not attempt to directly relate cost to benefits, but is advised for all DE projects to determine their feasibility over their lifespan and is essential if more complete analysis is planned. At a minimum, full costing in education should take into account:
• instructor compensation (salaries and benefits)
• travel and per diem for instructors
• support staff compensation (salaries and benefits)
• initial program development
• program revision
• equipment and software acquisition
• maintenance and upgrade of equipment and software
• consumable program materials
• delivery (e.g., satellite or internet charges)
• other operating costs (telephone, postage, supplies, publicity, etc.)
• facilities (optional)
In business settings one also needs to add what is likely to be the largest single costs; compensation for those receiving training. Additional costs might include: trainee travel and per diem; lost productivity; and need for replacements workers while some are in training, and training for those who will supervise the trainees if new policies or procedures are being implemented in the workplace.
When calculating full costs it is necessary to project the useful life of the DE program before major changes are likely to occur due to changes in the nature of the instruction, new technology, or changed business conditions. Thus, for example, a university might project the full cost of a DE program over five years on the basis that beyond that time conditions are likely to be so different that any projections are meaningless. If this is done, the residual value of any hardware, software, and materials can be subtracted when determining the full cost over five years. The case study presented later in which we describe the statewide DE network for training teachers illustrates the importance of calculating the full cost of equipment upgrades and program revisions to determine whether a program can remain viable.
Cash Flow per Unit of Time
Calculating estimated cash flow per unit of time can be useful for multi-year planning as well as for monitoring and managing a DE program once implemented. This involves projecting on a month-to-month or quarterly basis what expenditures will occur during that time. Any revenues can also be projected on the same timeline to determine how much money will be required to support the program, particularly in its early stages when there are likely to be no revenues. A typical DE program will show very heavy expenditures during start-up as the instruction is developed and infrastructure created. Only after it is implemented will revenues begin to flow and it may take some time for them to meet or exceed continuing expenditures on a monthly basis.
The results of cash flow analysis may be reported in various forms including tables, graphs, and charts. Spreadsheet software often has available several options for how to display results so they may be tailored to the preferences of specific audiences. A typical summary graph might look like Figure 1. It indicates there will be a negative cash flow until month eight of the project after which there is a positive cash flow. However even when the positive cash flow point is reached, the project has not reached the break-even point. (Break-even analysis is discussed below.) One of the important uses of cash flow is to determine whether the organization can fund the project until it reaches the positive cash flow point.
Insert Figure 1 about here
Break-even Analysis
Break-even analysis (sometimes called payback period) takes data analysis one step beyond cash flow to determine when cumulative revenues from a project match or exceed cumulative costs. Unless one desires to include the time-value of money (essentially this is the interest on the money invested and the effects of inflation), break-even analysis is relatively simple to calculate. The break-even point is when the sum of all the monthly costs is equal to the sum of all revenues to that date. Different organizations may have different criteria for judging how long is too long before the break-even date occurs, but projects that exceed 30 to 36 months deserve extra scrutiny before deciding to proceed. The results of break-even analysis can be displayed in various ways with cumulative cost/revenue curves being one easily understood format (See Figure 2). In Figure 2, the break-even point is reached at about 11 months.
Insert Figure 2 about here
From a risk perspective, the further away the break-even point, the greater the risk the project has of failing to reach that point. This is partially because unanticipated events can invalidate the assumptions upon which projects are made (Murphy’s Law is alive and well in DE projects) and also because even small upward deviations in costs or downward deviations in revenues from projections often will dramatically extend the break-even date.
Calculating Benefits
None of the three analytic methods described above directly takes into account any monetary benefits of the DE program other than revenues. Of course to an educational institution, revenues may be the only monetary benefit that can be determined. However, businesses, and in some cases educational institutions, may be able to determine other financial benefits to be derived from a DE program. Benefits can be thought of by considering delivering training via DE versus by some other method (e.g., DE versus flying trainees to a central site for training), or by comparing DE to no intervention. In either situation, benefits to which dollar amounts can be attached must be identified in order to calculate a cost-benefit ratio. Among the more common monetary benefits to consider are:
• saved wages of trainees due to shortened and/or flexible training times
• reduced number of replacement workers for those in training
• saved instructor time
• saved travel and per diem for trainees and/or instructors
• reduction in lost productivity
• increased productivity (e.g., sales or services)
• increased quality or less defects
• reduced operational costs
• increased safety or less worker’s compensation costs
• lower absenteeism
• increased worker flexibility (e.g., ease of reassignment).
When comparing corporate training via DE to more traditional methods such as "stand-up" lectures there is some evidence to suggest that DE is at least as effective while reducing trainee’s learning time (Charron & Obbink, 1993; Garrison & Shale, 1990; Mason & Kaye, 1990; Moore & Thompson, 1990) . The resulting cost savings can be substantial, especially when large numbers of trainees are involved. When trainees travel to a central site for traditional training, costs are incurred for compensation for both training and travel days and per diem expenses. These can be substantial when there are many individuals traveling great distances. The alternative, sending instructors to multiple sites, also has personnel and travel costs that likewise can be calculated. Lost days of productivity can be converted to costs in various ways: lost sales; overtime for other employees, and the need for additional employees to do the work of those in training.
Increases in productivity can usually be converted to dollar amounts when tied to increased products or sales or reduced numbers of employees to provide a given amount of services. Reduction in the amount of rejected products, or those requiring rework, can be converted to dollar amounts as can reductions in the amount of product or service recalls or customer support required. Similarly, the value of less waste of resources, less loss of employee time off the job due to accidents, and lower absenteeism can often be estimated with some degree of confidence. The last item on the list, increased learner flexibility is more difficult to quantify. However, in some situations the value of having employees who can fill in temporarily on other jobs during vacations or illness, or who can readily take on new positions due to company growth or personnel turn over can be estimated by upper management and included in a cost benefit and ROI analysis. (For example, lab technicians might also be trained to draw blood or assist on an EKG to expand the flexibility of staff assignments). For specific examples of how to attach dollar amounts to various types of data see the two books of case studies by Phillips (1994, 1997) and the sample data included in the instructional module by Decker and Campbell (1996).
Cost/Benefit Ratio
Cost/Benefit Ratio (CBR) is often calculated as Benefit/Cost Ratio (BCR) with the latter being similar to ROI. Both use exactly the same values with the only difference being which figures are placed in the numerator and which in the denominator of the ratio. As one might expect, the cost/benefit ratio is calculated by dividing the full cost of the initiative by its total monetary benefits.
The cost/benefit formula is:
Alternatively, the formula for benefit/cost ratio is:
The size of the resulting value indicates the size of any monetary benefit to the organization. When examining the BCR ratio, unless the value is equal to or greater than 1 the benefit is worth less than it costs. The larger the value above 1 the greater the benefit with a value of 2 or more generally being desired by managers who are looking for ways to rapidly increase productivity or profitability. From a risk analysis standpoint, the closer the BCR value is to 1 and the longer the project takes, the greater the risk. For example, a BCR of 1.5 over three years is much less attractive than the same BCR over 18 months since the opportunity for things to go wrong, or external events to overtake the project, is greater for a longer project.
Return on Investment (ROI)
Calculating ROI is identical to BCR except that costs are subtracted from benefits before dividing by total costs to obtain the rate of return. This subtraction of costs from benefits is necessary so that only additional (net) benefits beyond costs are considered since they represent the growth in value beyond the original money invested. Thus, net benefits equal total benefits minus total costs.
Thus, the formula for ROI is:
If this seems confusing, think about putting $100 in the bank. If after one year you took out all the money and it amounted to $105, then your rate of return was 5%. To obtain this value you had to subtract your initial investment from the amount withdrawn before calculating your rate of return. The same logic applies to calculating ROI.
Return on investment is often calculated on a yearly basis even though the expected benefits may occur over several years. There are many different ways to treat multiple year projects with the simplest (and least accurate) being to calculate the overall ROI and divide it by the number of years to obtain the yearly rate of return. For more detailed information on conducting sophisticated ROIs, see Carnevale and Schultz (1990).
What constitutes a reasonable rate of return varies for different organizations just as it does for individuals. Corporations commonly seek an annual return rate of 20% or more before launching new projects. However, the importance of non-tangible benefits or how a project fits long-term strategic plans can have a major impact on the decision. Certainly every proposed DE project will not require an ROI of 20% or larger to be initiated since there are many other important reasons for doing so.
Intangible Benefits
Intangible benefits (sometimes called soft data) may be far more important for some DE projects than their monetary benefits. When planning an analysis, a variety of potential benefits which either cannot be easily measured or to which monetary value cannot be easily or objectively assigned should be identified and considered during the decision-making process. Among the more common intangibles are:
• employee morale
• student satisfaction with the program
• instructor ratings
• widely held belief that "something must be done"
• "political" pressure can be responded to
• sense that it is the "right thing to do"
• desire to test out a new technology before committing to it
• "experiments" to test the feasibility of strategic plans
• development of more "sophisticated" learners who are able to use new
technologies and ways of learning in the future.
Although some may try to assign dollar values to some of these intangibles, Phillips (1994) recommends that often this is undesirable since it may taint the credibility of the more objective benefits data. He recommends dividing the data into "hard" and "soft" categories and simply reporting the soft items as factors to keep in mind during any discussion of the results of the analysis.
Intangible benefits analysis might be particularly appropriate for an institution as it considers the creation of a distance learning program. This example involves a small community college in the northeast — where congestion impacts the travel and learners struggle to make it to classes. Serving a wider audience is not the immediate goal, but staying current and meeting the needs of their learners is. Their plan was to begin with a pilot program by providing developmental support for three classes by interested faculty, and to offer them support. A consultant was brought in to raise the issues for all the faculty, and provided a modest amount of time for planning.
The financial commitment included a quarter time effort for three development individuals, and money for hardware and software. Since this initial small DE project will not expand the number of students served, the financial investment may never be returned, nor will more money likely come into the institution. To understand this institution’s actions, we have to look beyond financial remuneration to the intangible benefits resulting from the project. This institution is looking for enhanced professional development for their faculty, greater satisfaction of their learners by offering some of their classes electronically, and the potential to be on the cutting edge if they are suddenly faced with reduced state funding and the need to expand their student population.
Confidence Estimating
Although an ROI calculation might be done for only one set of values and their accompanying assumptions, it may be desirable to experiment with a variety of "what if" scenarios based on different assumptions to determine their impact on the results. For example, if one of the assumptions is that the cost of developing web-based modules will be $5,000 each, what would be the effect of that cost increasing to $6,000? Or, if the assumed rate of student retention across courses is projected to be 80%, what is the impact if it turns out to be only 60%? Sometimes what appears to be a small difference in a variable can have a major impact on the results. Understanding the interaction of the cost and benefit variables at this deeper level adds considerably to one’s understanding of what is necessary to make a DE project financially successful.
Using a variety of "what ifs" to test the impact of each cost variable may promote better project planning and management. For example, using a piece of commercial software rather than something in the public domain may have little impact on total costs, but have substantial benefits to learners and module developers. Similarly, if the relative weight of each benefits variable is known, as the project is implemented the actual numbers can be compared to those projected to see if the project might need to be expanded, contracted, or perhaps even terminated before its completion.
Presenting decision-makers with a variety of analyses based on different values may be an excellent way to assure they carefully consider their options. It also draws them into "owning" the project since they played a role in determining the assumptions upon which the decision was made to proceed. Spreadsheet programs are ideal for calculating results based on different assumptions and can display the results in easily read charts, graphs, and tables.
Before, During, and After Analysis
Although most of the discussion so far has talked about performing cost and benefit analysis before commencing a project, as the title of this chapter implies, it is also desirable to do analysis during and after the project. As actual cost figures begin to become available they can be compared to those from the original estimate, and if they differ widely, a new cost and benefits analysis can be conducted. In some cases, cost overruns (there never seem to be under-runs) may be able to be accommodated if the benefits too are more than anticipated. In other situations, increased costs may call for terminating a project early. Enroute benefits data may become available if, for example, a series of DE courses is to be developed and implemented one at a time. Initial enrollment numbers and the retention rates of those completing each course and signing up for subsequent courses might be of particular importance to an educational institution. Or, a corporation might examine the impact of training received to date on sales or other benefits measures for those who have received the training to determine whether to continue to train others, modify the design, or perhaps even abort the effort.
Analysis after a project is complete, or at a previously agreed upon date after implementation (e.g., one or two years), can confirm the benefits are in fact being obtained; or perhaps that they are not. In either event, the organization should want to know periodically what the results have been. As reported in the two books of case studies of training interventions by Phillips (1994, 1997) the results can be very positive. However, there are no doubt many cases with the opposite results that will likely never see the light of day.
This leads us back to a point we made earlier, but which bears repeating. The lack of almost any cost and benefit data from DE efforts is most disappointing and if those in the field fail to address this shortcoming, the long-term prospects for its continued expansion are seriously diminished. We further believe that a critical first step in planning any DE project should be for all stakeholders to agree on what they are, and how to measure, all costs and benefits as part of the overall project plan.
Consider the ways in which examining ROI before, during, and after the implementation of a statewide distance learning project might have saved the project, or at least provided more feedback to stakeholders. This statewide distance learning system was unique, in that it included state sponsorship and start-up funding, creation of an independent office and board of directors, and the integration of three types of networks within one office. The goal was to provide ongoing educational opportunities for both rural and urban citizens.
For seven years the system struggled. Prior to the beginning of the system a quasi-needs assessment was conducted, however, a significant demand for the programming was not established. No effort was made to evaluate the consumers’ satisfaction or willingness to pay real costs of delivering courses to remote corners of the state. Because the state was losing money throughout the seven years, no funding was committed to upgrades of technology as the years passed. Additionally, the ensuing years brought dramatic changes that also challenged the system: other states implemented complex networks, telecommunications gained rapid societal acceptance, online education experienced monumental growth and development, and distance learning grew dramatically.
Had those involved planned for and conducted some form of ROI before they began, they might have been able to give a more accurate estimation of true costs of establishing and operating the system. Given a realization that the break-even point might never be reached, perhaps state funders would have been able to identify important non-financial reasons for supporting the system, or perhaps made a determination to scrape the entire project. Instead, this project became a political football, limping along until it was barely serving the rural and underserved population. Staff was frustrated and demoralized, and the notion of distance education earned a very bad, and perhaps undeserved, reputation throughout the state. Significantly, it may be a long time before taxpayers and legislators will be ready to offer similar DE opportunities to teachers or any other groups in the state. In this instance, lack of planning and failing to collect and analyze any cost and benefit data have cast doubt on the whole concept of learning at a distance.
References
Carnevale, A. & Schultz, E. (1990). Return on investment: Accounting for training. Supplement to the July issue of Training and Development Journal, S1-S32.
Charron, E., & Obbink, K. (1993). Long-distance learning: Continuing your education through telecommunications. The Science Teacher, 60(3), 56-60.
Decker, C. & Campbell, C. (1996). Determining the cost effectiveness of training (A self-contained instructional module). ERIC Clearinghouse, ED 394 045.
Garrison, D. R., & Shale, D. (Eds.). (1990). Education at a distance: From issues to practice. Malabar, FL: Robert E. Krieger Publishing Company.
Gustafson, K. & Watkins, K. (1998). Return on investment (ROI): An idea whose time has come — again. Educational Technology. 38(4), 5-6.
Mager, R. F. (1996). Morphing into a 21st century trainer. Training, 33(6), 47-54.
Mason, R., & Kaye, T. (1990). Toward a new paradigm for distance education. In L. Harasim (Ed.), Online education: Perspectives on a new environment, (pp. 15-38). New York: Praeger.
Moore, M. G., & Thompson, M. M. (1990). The effects of distance learning: A summary of literature. (Vol. #2). University Park, PA: American Center for the Study of Distance Education.
Phillips, J. (1994). In Action: Measuring return on investment. Alexandria, VA: American Society for Training and Development.
Phillips, J. (1997). Measuring the return on investment: Volume 2. Alexandria, VA: American Society for Training and Development.
Phillips, J. (1998). The return on investment (ROI) process: Trends and Issues. Educational Technology. 38(4), 7-14.
Figure 1. Example of a Monthly Cost/Revenue Cash Flow Graph
Figure 2. Example of a break-even curve