Minutes of the U.S. Advisory Commission on Public Diplomacy September 2010 Official Meeting
Remarks
Washington, DC
September 28, 2010
Commissioners Present:
Bill Hybl, Chairman
Lyndon Olson, Vice Chairman
Penne Korth Peacock
Jay Snyder, Commissioner
John Olson, Commissioner
Presenters from the LBJ School of Public Affairs
University of Texas at Austin, Texas:
Dr. Ken Matwiczak
Ms. Amanda Dillon
Ms. Katherine Zackel
P R O C E E D I N G S
MR. HYBL: On behalf of the Commission, we want to welcome all of you here, great afternoon. I’m Bill Hybl and I don’t think I have any sound. Okay. Chairman of the U.S. Advisory Commission on Public Policy. With me today are the Vice Chairman, Lyndon Olson, Ambassador Penne Korth Peacock, Commissioner Jay Snyder and Commissioner John Osborn.
The Commission is a bipartisan panel created by Congress, clear back in 1948 to formulate and recommend policies and programs that carry out U.S. Government public diplomacy. The Commission’s focus for 2009 and 2010 has been on efforts to measure U.S. public diplomacy programs and initiatives.
Measuring the effectiveness of public diplomacy programs is difficult but critical to promoting U.S. foreign policies abroad, managing programs effectively, and indicating to the public diplomacy professionals what works and certainly what is valued. The Department of State has long been engaged in evaluation efforts and we have met with some of its staff to understand those particular efforts. We commend these efforts, but we also believe more could be done.
To this end, the Commission entered into a property agreement last year with the University of Texas, Lyndon B. Johnson School of Public Affairs. This collaboration took the form of a two-semester policy research project involving approximately 15 LBJ School graduate students and one professor. The Commission believes that the LBJ was best equipped to undertake this cooperative project due to its outstanding track record in the field of public policy research.
The Commission asked the LBJ School to do three particular things. Number one, develop a public diplomacy model for assessing performance, a quantitative tool that can be used to determine the effectiveness of the Department of State public diplomacy initiatives, vis-a-vis, stated goals and objectives. Number two, review the Department of State’s current public diplomacy evaluation efforts and three, make recommendations for future evaluation efforts.
I’m not going to go into too many of the details. as Professor Matwiczak can. Professor Ken Matwiczak is here with us today with two of his associates to brief us on the report, copies of which hopefully all of you have at this juncture, including the DVD.
I would ask you hold any questions to Ken or to members of the Commission until we have finished this particular presentation.
Ken, you have the floor.
MR. MATWICZAK: Thank you, sir. Good afternoon, everybody, my name is Ken Matwiczak and I’m a senior lecturer at the LBJ School of Public Affairs, University of Texas at Austin, and it’s our pleasure to be here and tell you a little bit about the work we’ve been working on for about the last year or so.
With me are two of our recent graduates from the LBJ School, Amanda Dillon and Katherine Zaclkes and they’ll also be sharing the presentation this afternoon.
So with that, what we’ll be talking about as we go through is a little bit of background of how we came to be, how the project came to be, what we did throughout the project, how we went about our work, what we came up with, a notional model for measuring public diplomacy efforts, etc., and then a demonstration of how that model might be worked or might be used. You’ve got a copy of our handouts and our slides, also on your chair, and a copy of the handout that Amanda will be using later on. So with that, we’ll get started.
Affectionately, for those that don’t want to butcher my name, around the LBJ School I go by Dr. Mat or Professor Mat, saves everybody a lot of trouble. And for those of you that have been watching the news at all, the University of Texas, Austin campus is open again. Those of you that are in the dark, there was a shooting at the main library on campus at the University of Texas at the main library. The shooter came in with an AK-47, shot a few things up, didn’t hurt anybody but himself and shot himself, and there is only the one shooter, so that’s all we know and that’s all anybody knows, and we talked to people back on campus, too. But the LBJ School is on the far diagonal opposite side of campus, so all our people are safe up there.
What we’d like to do is spend, hopefully an hour that’s not going to be too boring, this afternoon, explaining what we did and what we came up with as a result of our work. And what it’s good for and what you can do with it from this point on. And then demonstrate how it might be used by the Department of State.
To begin with –– oops –– it’s like the Gecko with the –-
In the fall of 2008, Dan Firestein, who is an alum of the LBJ School of Public Affairs, came back to school. He was, at that time, the deputy executive Director of the Advisory Commission of Public Diplomacy, and he was familiar with the curriculum and the type of work that we do. We have two masters’ degrees at the LBJ School, the master of public affairs and the master of global policy studies. Each of those curriculum require the students to take a two-semester policy research project course that German referred to.
That policy research project course is taken by first year public affair students and second year global policy study students. In that course we take a team of anywhere from 10 to 16 students, and they become a consulting team for a real world client on a real project, a real question of importance or a real issue. And we contract with that client to answer their questions and perform certain services for it. And we’re under the sponsorship of that client. Well, Mr. Firestein was familiar with the work that we could do and the type of research that the students were capable of doing, so he suggested that we might want to consider engaging with the Commission to come up with a way to measure the successfulness of public diplomacy efforts, which really was kind of a state of disarray at the time. There’s no real hard way to do that. So we thought we’d try to take that project on, so that over the next spring we negotiated with the Commission with Mr. Chan, the Executive Director with the Commissioners on what exactly we could do and what they were expecting out of the project.
Here’s what we agreed to try to do as Chairman Hybl pointed out. In the end, what we wanted to come up with was a model that could be used for measuring public diplomacy efforts. Now why did we get involved with this in addition to having a lot of work going on with public diplomacy? Some of you may not be familiar with how extensive our work in public diplomacy is. Our former dean is the current Deputy Secretary of State, Mr. Steinberg, Jim Steinberg was our dean. And so we have that linkage in public diplomacy background there. But in addition to that, part of the work I do as a senior lecturer and the research I do is on performance measurement, and we’ve done several projects in the past on measuring performance in government entities and how you go about organizing that work in the interest of governance, in the interest of communicating with the public, etc. So there was a mutual attraction between public diplomacy measurement and our work with performance measurement in the past. So then what we wanted to try to do is leverage that work we had done before and apply it to public diplomacy efforts and how it –– and show how it could be applied to public diplomacy efforts.
So in September and August 2009 is when the project actually started. We enrolled 16 students in the course, one of them only stayed a semester because she had to go off and do some other work, but we ended up with 15 students, following through two semesters. In these project courses they are typically projects that are run, managed and conducted by the students themselves, under the guidance and supervision of a faculty member. So they actually do all the planning of the work that needs to be done. We bring in 15 students from varied backgrounds. In the student’s case here, I think most of them could spell public diplomacy when they started the course. I’m not sure how much more they knew about that, so we had to do some training. We had to do training on performance measurement, evaluation, etc.
And to explain how we went through that initial phase of the project, the research we did, until we came up with a model, I’d like to ask Katherine Zackel to talk you through that portion of the project.
MS. ZACKEL: At the beginning of the project, we came up with an original timeline for our research. First, during the period of September through January, we were going to begin background research using Department of State sources and also extensive literature review. We also planned to develop and implement a fishing survey and a later in-depth survey to get feedback from the public diplomacy professionals.
We planned trips to embassies to do case studies and work with the public affairs officers in the field. We also planned to begin developing a conceptual model to measure the effectiveness of public diplomacy.
From January to July, we planned to further develop the model, to collect data from the Department of State to populate the model and also go out into the field and test the model and then get feedback from those public affairs officers that we would be working with. And then finally we would present a final conceptual model and report to the Commission.
We faced several challenges in meeting this original timeline. Our greatest challenge was that we began work in late August of 2009 and we did not receive funding until March of 2010. We also did not receive access to any kind of comprehensive data from the State Department regarding current or past public diplomacy programs. We also received just limited access to the Department of State personnel within Washington, D.C. and in the field. Because of these challenges, it was difficult to survey professionals, collect data, receive feedback, or even study the current measurement tools that were out there.
We did receive support and guidance from the Commission and the Commission staff. We also met with Therca Montgomery, the director of the Evaluation and Measurement Unit. We met with Rick Ruth and Robin Silver from the ECA Office of Policy and Evaluation, and we also able to rely on some personal contacts. Some of the students had interned in embassies or within Washington, D.C. for the Department of State and others had other professional contacts from previous experiences.
Because of these challenges, our approach was modified. We interviewed several public diplomacy professionals. Karen Hughes, the former Under Secretary came to speak with our class and give us contacts regarding the strategic goals of public diplomacy programs. We also met with Carl Chan, Gerald McGlauflin and David Firestein of the Commission staff. Commissioners Peacock and Osborn both came to our class to speak with us and answer any questions we may have. And finally, we had two diplomats in residence at the University of Texas, Craig Engle and Bill Stewart were able to give us their perspective from the field.
Performing the survey was difficult because we didn’t have access to Department of State personnel. We did reach out to those personal contacts that I mentioned earlier, and we also reached out to academia. We found academics who previously served for the Department of State and we also found some who were focusing their research on public diplomacy. So we were able to get the survey out, but not to as many people as we originally had hoped.
We did do an extensive literature review, relying on publicly available sources and we did do an extensive review of public diplomacy activities, primarily using embassy websites.
Our analysis of the current measurement tools that are used by the Department of State showed a lack of coordination and a lack of a unified effort. There seemed to be some duplication of efforts, yet at the same time, there was not consistency in measurement across time, across locations or across different types of public diplomacy programs.
We also found that a lot of these measurement tools focus on outputs, rather than outcomes. Outputs tend to be more descriptive data, such as how many people attended a given public diplomacy event, or how many events occurred within a given month at a given embassy. We focus more on outcomes, which we found somewhat lacking in current measurement, and that would be related to the actual strategic goals of public diplomacy. In the end we found no strong correlation between the public diplomacy programs that go on in the field, the cost of those programs and the effectiveness in meeting strategic goals of the United States.
Through our research we developed a conceptual model, called the public diplomacy model for assessing programs. We began by developing three outcomes. These were understanding, influence and favorability. Understanding is the foreign audiences’ and foreign publics’ ability to comprehend United States policies and United States culture. Influence is our power to effect change in those foreign publics and favorability is our ability to seek their approval or admiration.
We developed metrics to help measure these outcomes and several suboutcomes and we used these metrics to design a model to be able to link the programs that we do out in the field to the strategic goals of the United States.
And to go into more detail about the model, I’m going to turn it back to Dr. Matwiczak.
MR. MATWICZAK: Much of the work that Katherine talked about was self-motivated, self-generated. Because of the limited access we had to Department of State resources, so late in the project, we weren’t able to travel, we weren’t able to conduct surveys, etc., because of the limited funding and the limited access. So the model that we’re about to talk you through or describe to you is a notional model. This is what could be used, or it could provide a framework for further work in measuring public diplomacy efforts. So anything we talk about here, the three outcomes –– the outcomes being the so what or what effect did these measures have, did these efforts have, as opposed to how many of these did we do. The outcomes measure are more directed towards the effect of those. Even these three outcomes are suggested based on the research and the interviews and the survey results that we got back during the initial phase of the project. So we used this as a basis for developing a model for measuring performance. From these we developed metrics, etc., that again, are notional, we think might be a way to measure some of the efforts.
For those of you that are familiar with multicriteria decision-making models, multi-attribute utility models, the framework for measuring performance that we adapt is essentially the same as MAU and uses the multi-attribute utility approach, in modeling approach. For those of you that aren’t familiar with it, don’t worry about the big words, I’ll try to explain our thinking and our rationale for the type of model that we put together.
So what I want to do for the next few minutes is spend some time talking about what the model is, what it looks like, what elements are in the model, how it can be used in the strategic planning process to measure performance and make decisions as we go through. When I finish my talk, then I will turn it over to Amanda who will actually bring up the spreadsheet that we do use and show you how simple it is to use to put in measures of performance that you’re interested in, to set various parameters in the model, interpret the outcome and use the model to do decision making, etc.
So with that, the spreadsheet, the model itself, is based on a simple Excel spreadsheet. We wanted to come up with a way that was portable, accessible and easily understood by all potential users. We didn’t want to make any proprietary software, we wanted this to be publicly available so that almost anybody at any level within the public diplomacy world would be able to use this model, make their own decision making, whether it be at the embassy level in the field or at the State Department level for strategic decisionmaking. So we kept it simple using an Excel spreadsheet.
The basic organization of the model within this spreadsheet then, and you have a framework on your seat there, is a hierarchical structure, a tree structure, working with top level outcome measures that are broken down into subsequently more detailed levels beyond that. And Katherine will demonstrate them –– I mean Amanda will demonstrate that a little bit when she explains how the model might be used. But the first page of the second handout, not the PowerPoint one, is a breakout of one element of one of the outcomes. The understanding outcome, how well those –– the public, general public in the target area understand the message that we’re trying to communicate with the PD efforts. And from there we broke it down into subsequent smaller measures, which we thence broke that into what are the physical things that will inform those suboutcomes. So basically it’s a hierarchical model of broad outcomes, broken down subsequently into smaller measurable bites.
We take those measurements and we translate them to a common scale, so that we can compare how we are doing with respect to generating an understanding of our efforts, or our information that we’re trying to disseminate, versus how much influence that is having, measured vis-a-vis, as the chairman mentioned, vis-a-vis, some strategic goals. So our success in accomplishing these strategic goals are all measured on the same scale and we came up with a common scoring scale, if you will, to use on that. And it’s all measured against the expectations as established during the strategic planning process. Because we have an easily communicated hierarchical structure with visible measures broken down into subsequently smaller measures available and visible to everybody, all the scorings out on a common scale, it becomes an easy tool to use to communicate strategic goals, how we measure success of those strategic goals, etc.
We’ve done this approach to measuring performance in several organizations in state government in Texas. One with a local river authority in Austin, the LCRA, another with the Texas comptroller’s office, one section of that, and we’ve done another project with our local metropolitan transit authority. We’ve also used this approach in trying to model the performance of the budgeting system within the State of Texas government. That one didn’t go so well yet, they’re still working on that. But this approach has been used and the biggest benefit that those users got out of this approach was right here. This is the biggest benefit of the model is to get the strategic planners talking to each other, saying what is important, what are our priorities, how do we know when we’re successful, define successful, etc. If you can accomplish this, the rest of this is easy and it falls straight out.
So the simplest spreadsheet model, with hierarchy. So what does this model have in it? Well, as mentioned, the hierarchy. We take the outcomes and we break them out into subsequent measures. The types of measures that we’re talking about are the outcomes, are the so-what or the effects. Increased understanding of the U.S., increased influence of the U.S. Government in the opinions of the elites in the foreign country that we’re targeting, etc. So the so-whats are the what –– so-whats and the outcomes, the impacts. Depending on the outcome that we’re looking at, we may want to target our efforts at the general populous. You may want to target your efforts at the elite decisionmaker to the country or even the government itself. Public diplomacy effort may have that target, the option to include or exclude any target audience is built into the model.
The policy of area of interest. Are we trying to influence their opinions about environmental issues in the country? Security issues in the country? Or just educate them about the U.S. culture in general, so cultural issues or cultural areas. What are the policy areas of interest that we’re trying to influence? And how do we measure whether or not those efforts are successful? Is it just how many people attend or is it a combination of how many people attend, how many people subsequently change their opinion by a vis-a-vis a survey? Change their opinion about us. And we heard testimony in the previous Commission hearing in July from the offices within the State Department that are doing this type of work. Again, there is no unified framework, there isn’t a unified structure. This is a vehicle for getting those people to talk to each other also. So these are the elements that make up the model. The outcomes, who are we targeting, whether we try –– the message we’re trying to communicate, and how do we know whether or not that message is successful.
Within the model then, the decisionmakers can establish priorities on every single one of these elements. Is understanding much more important than influencing somebody in this country? And you can change this priority, depending on what you’re trying to measure, whether it’s comparing programs at the mission level, or comparing missions in the region level, comparing embassy efforts at the regional level, or at the national level comparing regions, and the success of PD efforts within the regions. So depending on what your goals are, you can establish different priorities for different elements –– all these different elements of the model in there. And you can use these priorities and change these models because there’s a spreadsheet to do "what if" analysis, what we call sensitivity analysis. What if this was a higher priority? Would that influence the decision I’m going to make about this program later on? So we can do some sensitivity analysis.
Incumbent upon the decisionmakers are to sit down and define what they expect out of an effort, out of the public diplomacy effort. Force the decision makers to think, how do I know I’m successful? What do I consider to be a success? And at the ideal level, I’d be perfectly happy if we achieve this level of performance and we got this result. A hundred percent of the population that we surveyed after we did this radio broadcast totally understand, from survey results, what the U.S. is trying to accomplish in Afghanistan. That might be the ideal level. Then again, I am going to be on the opposite end of the spectrum, I would be thoroughly upset if we didn’t get at least this percent of the population. So what’s the least acceptable, the minimum level of effort? And what we’re going to do then is measure the success of these efforts, relative to some ideal standard, or relative to some least –– minimally acceptable level of performance on it. And the model allows you to simply put that in there. Change your standards if you like and watch the effect of changing those expectations also.
The final element that we can include in the model is some characterization of how much risk the decisionmaker is willing to take, vis-a-vis, giving additional resources, or just inherent in the operation itself, in the effort itself, might be a riskier effort to write an editorial than it is to send a broadcast or something in a particular area of the world. How much risk is that decisionmaker willing to take to accomplish that level of performance as a result? And again, this is changeable. Some people are very averse to taking risk with respect to certain other componentcy efforts. Others are willing to take a lot of risk. Put yourself in the situation of Donald Trump spending money, versus me or you spending the same amount of money. We’ll be a little bit more afraid to take a risk about dropping $100,000 for a particular effort than Donald Trump might be. So again, the idea of taking a risk to achieve a level of performance is incorporated into the model explicitly. And the decision has to address that explicitly also.
So an example of the kind of things we put into the model, we talked –– Katherine introduced the idea that we came up with three generic, if you will, outcomes. Our idea of what three outcomes might be used to measure successful public diplomacy efforts and they’re not lacking concrete. How did we come up with these? Well, through our research we found out that the evaluation measurement unit at the Department of State has an effort ongoing called the Public Diplomacy Impact Project in which they go out to several countries around the world and measure the impact of a particular public diplomacy effort. But again, they’re very broad measures of the impact and they measure it in six areas, based on surveys and follow-up surveys. These three represent a subset of those six areas that the evaluation measurement unit considers outcomes of successful public diplomacy. So again, this is a subset, based on work that’s already being done in there. It can be expanded, it can be reduced, it can be changed, however you want to work it.
At the next level in the hierarchy then is, who are these efforts targeting in there? And this is on that spreadsheet that you have in front of you, the hierarchy spreadsheet you have in front there. Who are we targeting these efforts to? The one that you have on the top page is the understanding outcome. There, that’s the first page that you have, broken down into the target audience. In this case the page that you have is the general population. We’re targeting these efforts at how well the general population understands something. And what is it that we’re trying to get them to understand? Something in these areas. We’re trying to influence their understanding of U.S. culture, of our understanding of our foreign policy, our street policy or something. Again, these are notional. They are measures and areas to be developed as a result of the interviews we conducted, the surveys we conducted, etc. Again, they may change depending on the decision maker, depending on the department goals, etc.
So we’re trying to influence, on the example you have, the general public’s understanding of one of these. And I forget which it is in the spreadsheet, if it’s foreign policy, I think, or culture, that’s broken out. Well then, how do we know they understand U.S. culture better? We break it down to the next level and we actually have the performance measures. And in our case, in contrast to some of the efforts that are already ongoing in the Department of State, Katherine referred to them as output measures. There is the map, the mission activity tracker efforts going on currently at the embassy level, that generates data backup into the Department of State, which reports, here’s what we’re doing on our public diplomacy work at –– in public affairs work at the embassy level. We had these kind of events, we had this many of those events. At these particular events we had this many people attend of this type. You might be able to conjecture those are output measures. Those are bean counts. We had this many people. They don’t tell you, so what? What happened as a result of 100 people coming to this particular event? So what we were looking for and what we’re proposing as a result of some of our metrics are changes in attitudes, whether they’ve done surveys or interviews or percent participation or if you were looking at security policy, etc. How many, was there a change in the number of demonstrations or in the participation in anti-U.S. demonstrations in the street? So we’re looking for deltas, changes in percents and changes in participation, rather than bean counts, of actually how many people participated. Because it’s awful hard to track millions of dollars per head at a particular embassy party or an art exhibit in some foreign country, etc. But what we’re trying to do is establish a link and say, if I spend this much money, then I can track that down through the hierarchy and back up through the hierarchy and say, I had this event at this embassy and I measured how successful it was and I can track that all the way back up to its overall impact on Department of State’s public diplomacy effort, back up the hierarchical chain that you have there. So these are notional, again, suggested ways of measuring the performance there.
Once we have these performance measures defined, we go back and we have to establish what’s most important to us. Yes, it’s going to be iterative, we might establish priority and say this is the most important as we go through it, but then again, an iterative process, go back and say, all right, what is our most important outcome? Is it most important that they understand us, or that we are able to influence that general public as a result of our public diplomacy efforts? Again, it has to be set by the decision makers as part of the strategic planning process, the strategic goals at the Department level and every level in between, all the way down to the embassy and the mission level.
We’ll establish priorities –– they pick up a priority on this 0 to 10 scale. Ten is absolutely must have, got to have this, extremely important, zero is I don’t care, absolutely no importance. I mentioned earlier that we could eliminate some of the target audiences simply by establishing a zero priority. So we can target our efforts and say this is absolutely of no meaning to us and give it a zero priority and that eliminates it from the model directly. Again, so they can establish how important a particular measure is. A member, as everybody knows, if everything is top priority, priority ten, nothing is top priority. Again, so that’s part of the decision making process. The discussion process that’s going on around the table and up and down the chain of command.
Once they’ve got all this hierarchy built and all these measures put in place, saying this is how we determine whether or not we’re successful at understanding in influencing the public’s understanding, breaking it all the way down to the measures. Then they go down to each bottom level measure, the metric Excel and say, all right, what is –– what do I expect out of this effort? I’m going to throw a concert in Zimbabwe, we’re going to put on a concert in Zimbabwe and we’re going to keep track of who comes, etc. What do I consider a successful way of influencing their understanding of U.S. culture? Is it a survey, before and after survey that I do, etc. How do I measure that? What would be the ideal level of performance? If I change 100% of the concert-goers attitudes, is that ideal? Is that realistic? So we go through and we define realistic expectations with respect to how the results of that effort should be measured, and then zero –– and it’s scored on a 0 to 100 scale, because we’re all familiar with grades, we all get our grades in school on 0 to 100, people can relate to 0 to 100 scales, 100 being the highest score you can possibly achieve, zero being absolutely nothing.
So we’ll ask the decisionmakers, again, to go through, establish their measures, prioritize the measures and define what their expectations are with respect to those measures. And then go back again and say, how much risk am I willing to take to achieve that level of performance? The ideal level of performance. And there are ways in the model to characterize their risk behavior. Those of you familiar again with multi-attribute utility decision making theory know that we can break the risk behavior down into this, where we are willing to take a risk to achieve high levels of performance, a risk taker. Or we’re not willing to get high levels of performance, but we’re more worried at the low end of the performance spectrum, changing what we can do at the least cost, the least influence on resources, they may be more risk adverse. Risk neutral, we don’t necessarily care either way.
So the decisionmakers, the people sitting around the table have to come up with these elements and they can put them into the structure that we provided. Amanda, again, will show you how we go about doing that. What do you get out? When you put all these things in and you say –– push the button, well, not really push the button, it’s being calculated as you put the things in, so it’s dynamically changing as you put all these elements into the spreadsheet model. What do we get out of it? We get out at every level in the model a performance score on a 0 to 100 scale. So we know how we did on this 0 to 100 scale in effecting the understanding of the general population with respect to U.S. culture. Do they understand U.S. culture. We can compare that into how successful to where we wanted to be, vis-a-vis where we wanted to be, in our ability to influence them. So at the very top level, understanding income in influence and favorability rating of the U.S., we can find out how successful were each of these outcomes, relative to where we wanted to be, and at each level subsequent that, all the way down to the lowest performance metric. This is what we wanted to have our survey say the change in attitude was as a result of that concert. Did we achieve that on a 0 to 100 scale? And at each level down, we can attribute how successful we were to the higher level by looking at the subordinate measures. So if we did not do real well at the very top, we can go down to the next level and say, where did we fall apart? Was it in our ability to influence their understanding of U.S. culture, or their understanding of the U.S. foreign policy? Where were we really weak? And we can focus then, if it was foreign policy, focus on foreign policy. And break foreign policy down.
Was it the method or means we used to disseminate the message? Was it the broadcast, was it the media, the journal print media? Or where was the weakness there? And then break it down further into what actually failed in this particular area. And we can start focusing and making strategic decisions based on that breakdown of where we were weak versus where we were strong in the model.
The advantage of a hierarchical structure of a model, that’s what we just talked to you right there. What that provides you then is, in the model, the physical model, audit trail of results. What did we do well, what did we not do so well? And it’s all there for everybody to see, at all levels that you’re using the model at, the decision makers on down. And it gives the opportunity for decision makers to address those particular areas. So we have an audit trail of where we expended the resources, what the results of expending those resources in that particular area were. And we put that on a graph because people are visual to begin with. So we use a graph, and Amanda will show you some graphs, it’s in the handout that you have, show you how we can take those graphs and break them down into subsequent levels and apportion blame, or credit, if you will, at each level in the hierarchy. So we take all these numbers, we give you the numerical results, we can do that in the table if you want, but visually it makes more sense when you’re sitting around a conference table and having a discussion about goals, objectives, etc.
So how can we use this? If you accept this idea of a hierarchical structure broken down into subordinate elements, each of which is measured relative to some expectation on level of performance, what can we do with that? Well, the obvious thing is, put it up there and say, where do we want to go with the strategic planning process? What do we consider successful? If we spend this much money on public diplomacy, how do we know it’s money well spent? And how are we going to determine whether or not we are successful in that? So it gets the decision makers talking at all levels to define, what’s our goals? What are our objectives? How do we measure whether or not we’ve accomplished those goals? Objectives from your planning process were smart, specific, measurable, obtainable, realistic and time-driven. So how do we know when we’ve achieved those goals? And these –– on these objectives, how do we measure to what extent we have accomplished these objectives? Again, you can see the hierarchical structure evolving or developing there.
We can compare them and so this model or this structure, this framework can be used at all levels. You can sit down in a mission or an embassy and compare programs. Where are we strong, where are we weak? So within a single mission you can compare across several periods. We change the amount of resources we allocated to this effort. We put more into this. Did that result in a change in our performance, or the outcomes that we achieved as a result of those efforts? So we can take a single program and track it over several years, or you can take a single program and compare it over multiple missions or multiple countries. Say we’re spending $10 million in Zimbabwe, $15 million in Namibia and $40 million in South Africa. How are we doing, relative to where we expected to be as a result of those efforts? And we can compare those three or four countries on there. So you can use it all different types of levels.
Again, we’ve already mentioned –– identified strong and weak results, we can do a "what if". What if this wasn’t the number one priority? What if I made this number one priority, how does that affect my interpretation of the results and our efforts? It changes your graphs, but you have those changes in front of you now, the old one to the new one and you can see the resulting input. What did it affect in the chain or in the hierarchy, how did it influence my overall measure of performance and my ability to accomplish the goals? So we can do "what if" analyses.
But what’s really nice on this is, one of the things that we did with the state agencies and the other government entities that we did involves this –– we can take your budget, how much money did we spend on that? What was the change in the overall score, if you will, as a result of adding these resources to it? Was this money well spent? We spent an additional a million here, we’ve got this much change in the accomplishment of our goals. But we spent a million dollars over here and we only got this much in the overall attainment of our goals. So it helps the decision makers again, identify where our money is being well spent. Now that’s not explicit in the model yet, but the ability to do that is there in the model.
By the way, this is also drawing from some work done by Excenture and they have a book on it called “Public Sector Value” or “Public Service Value Models.” This is where we first got introduced to this idea and this concept. They had us test their model out with these public entities and there were some weaknesses and fallacies that we found in them and we’ve since adapted it to this approach now, where we use relative scoring scales, on a 0 to 100 scale. Their method of doing cost effectiveness would blow most accountants away, though. Try and bring in the cost –– the cost of money over time and just an unbelievable amount of factors, number of factors in the cost effectiveness. But you can do that in this model if you want, or you can keep it simple in the gross expenditures.
So how do we go about using the model? That’s a basic layout of what the model structure is, the elements of the model and what we’re trying to accomplish with the model and how it might be used. What I’d like to do is turn it over to Amanda now and have her show you what we can with the model, how the user can actually input things into the model and change them in the model.
MS. DILLON: Thank you. So in order to use this, as Dr. Mat has already pointed out, you have to identify each of these components within the model in order to decide, I guess, how to most effectively use it.
So first you have to come up with your outcomes. Now we’ve selected influence, understanding and favorability, but these can be changed if the State Department decides that this isn’t necessarily the direction they want to move in. Second is the audience. We selected the audiences of members of foreign governments, elites and the general population. Now, suboutcomes is a breakdown of how the outcomes are achieved. For example, under understanding, the three suboutcomes are dissemination or the spreading of information, reception or before audience actually receiving the information, and finally comprehension, whether or not they understand what they received.