| Abstract |
| Negotiation training evaluation tends to be aspectual and piecemeal; often focusing on one or two salient outcomes of training. A model for negotiation training evaluation research is presented which offers a comprehensive conceptualization of the hypothesized individual and systemic effects of training in collaborative negotiation. This model assesses change at the individual level in conflict-related cognition, attitude, affect and behavior; and at the systems level in outcomes and work climate. The Negotiation Evaluation Survey (NES), a time-delayed, multi-source feedback approach to assessment and development, is presented as a means of addressing some of the conceptual and methodological problems inherent in more common methods of training evaluation. An example assessment of one model of collaborative negotiation training for adults, the Coleman/Raider Model (1992), is presented. The results, implications, and future research objectives are discussed.
|
| A Comprehensive Approach to Assessing the Effects of
Collaborative Negotiation Training on Individuals and Systems |
| Negotiation training is a multi-million dollar industry that has become prominent in many fields including education, business, law, government, and international affairs. However, despite the growth of this industry, there has been little systematic theorizing or research specifically directed to training in this area. As Deutsch (1995) noted, "There is an appalling lack of research on the various aspects of training in this field. We haven't begun to collect the data that answers such questions as who benefits and how, through what type of training, for how long, by what trainers and in what circumstances"Ö(P. 128). The most basic question that remains unanswered is simply "Does negotiation training really work?" In other words, does participation in negotiation training help people to negotiate more effectively in their personal and work lives (Lewicki, 1997)? Of course, the answer to this question can depend on how one defines "effectiveness" in negotiation and how this definition translates to measurable objectives in training. |
| Lewicki (1997) describes negotiation as a complex skill set composed of a variety of component parts. He argues that too often we teach negotiation as though it was a single skill and fail to identify and develop critical skill subsets. Similarly, most attempts at negotiation training evaluation focus on one or two salient outcomes of the process, such as behavioral changes in negotiating styles (Rahim, 1985), analytical shifts in problem-solving strategies (Roderick, 1998); cognitive changes in fantasies and attribution biases (Roderick, 1998), or more generally in levels of participant satisfaction. This overly general or piecemeal approach to training evaluation lacks in the kind of specificity and clarity needed to further develop the various conceptual and practical models in use in the field. |
| Methodological problems with training evaluation are common as well. Evaluation data is often collected through self-report methods that ask participants to reflect on changes that have occurred in their own behaviors, cognition, etc. as a result of the training. This method often results in a "self-enhancement effect", with respondents evaluating themselves more positively than others would (John & Robbins, 1994). The data is also commonly collected at the conclusion of training and therefore fails to assess the transfer of the effects of the training into the actual lives of the participants. |
| If the practice of negotiation training is to develop as deeply as it has broadly, we need to begin to conceptualize and systematically measure the desired effects of training on the various sub-components of the skills that contribute to effective negotiation. This article presents a model and a method for evaluating the effects of collaborative negotiation training at the individual and systemic level. This approach assesses change at the individual level in conflict-related cognition, attitude, affect and behavior; and at the dyad and systems levels in outcomes and work climate. The Negotiation Evaluation Survey (NES), a time-delayed, multi-source feedback approach to assessment and development, is introduced as a means of addressing some of the conceptual and methodological problems inherent in more common methods of training evaluation. An example assessment of one model of collaborative negotiation training for adults, The Coleman/Raider Model (1992), is presented. The implications for basic, applied and action research on negotiation training will be discussed. |
| The Conceptual Training Model |
| There is a broad array of approaches to negotiation training available in the conflict resolution field today. This includes training in the competitive, win-lose strategies and tactics of distributive bargaining (based on work by Schelling, 1960; Alinsky, 1971; Karass, 1974); the cognitive, rational, game theoretical orientation (see Bazerman and Neale, 1992; Raiffa, 1982); the integrative, issue-based methods of "principled negotiation" (see Fisher, Ury & Patton, 1991); the social-psychological, collaborative needs-based approach (such as Deutsch, 1973; Coleman/Raider, 2000); and in the social justice, critical theoretical systems orientation (see Lederach, 1997; Mindell, ?; Reynolds, 1998). Each of these training approaches operate off of a distinct set of assumptions, values and objectives which is important to make explicit at the onset of any evaluation process. For example, the objectives of a training can range from learning to maximize one's own outcomes by dominating, intimidating, and "winning" (at all costs) in a negotiation, to creating a sustainable and fair relationship that offers the potential for mutual transformation, long-term stability and peace. |
| Proposition #1: A comprehensive approach to evaluation of negotiation training begins with the identification of the underlying assumptions, values and objectives of the training model. |
| For example, the basic training in collaborative negotiation that we offer at the International Center for Cooperation and Conflict Resolution at Teachers College, Columbia University is essentially a social-psychological model of negotiation that incorporates both integrative and social justice components (Coleman/Raider Model; see Raider, Coleman & Gerson, 2000). The basic elements of this approach are as follows: |
|
|
|
|
|
|
| These elements translate into the following knowledge, skill and attitude objectives for training in the Coleman/Raider Model of Collaborative Negotiation: |
| Knowledge Objectives |
|
|
|
|
|
| Skills Objectives |
|
| Attitude Objectives |
|
| As is evident from the above outline, the basic elements and training objectives of the Coleman/Raider model are somewhat eclectic, but specific to that model. Adequate training evaluation must begin at this level of analysis in order to identify basic differences that exist between the various approaches to training. |
| A Systems Approach to Training Evaluation |
| Our general approach to training evaluation views training as an educational intervention aimed at fostering change in people and systems. Whatever the objectives of a particular approach to training, we assume that training will attempt to foster change in how participants see, feel, think, or act when negotiating conflict, and that those changes can effect the systems they interact with. Therefore, a high degree of clarity and specificity regarding the targets for change is recommended. Proposition #2: In order to evaluate sufficiently the impact of negotiation training on individuals and systems, one needs to carefully consider the different targets for change; distinguishing between the various psychological and social elements, while keeping in mind the ways the elements interrelate. |
| We have used the MACBE model as a broad organizing framework for targeting change associated with negotiation training. The MACBE model (Pruitt & Olczak, 1995) is an eclectic, multimodal systems approach to addressing social conflict which traces the resolution of a conflict to changes in five distinct yet interdependent "sub-systems" of an individual: their motivation, affect, cognition, behavior, and the surrounding environment. The model was adapted from one of psychological disorder (Carson and Butcher, 1992) and views these five modes of human experience as interactive and working with "circular causality"; affecting and being affected by changes in the other modes. We have modified these labels and added the category of negotiation outcomes in order to assess change in this critical aspect of the process as well (see Figure 1). This conceptual framework allows us to identify and assess the effects of training on each of the various sub-components of the negotiation process (thinking, feeling, etc.) in order to evaluate the relative strengths and weaknesses of a given model. |
| A Time-Delayed, Multi-Source Approach to Negotiation Training Evaluation |
| As previously mentioned, methodological problems with training evaluation are common. Evaluation data are primarily collected through self-report methods that ask training participants to reflect on changes that have occurred in their own behaviors, cognition, etc. as a result of the training. This method often results in a "self-enhancement effect", with respondents evaluating themselves more positively than others would (John & Robbins, 1994; See Schwartz, 1999 for a summary of other problems presented by the use of self-report methods). One method for correcting for self-report bias is the use of multi-source feedback data. |
| Multi-source feedback (MSF) is the process of eliciting perceptions of a target person's behavior from a variety of sources (such as supervisors, peers, subordinates, self, and relevant others) for the purposes of evaluation and professional development (Church & Bracken, 1997). This process of performance assessment is particularly popular in organizations today (London & Smither, 1995). Although research on multi-source feedback as a formal practice is relatively recent, hundreds of articles on various aspects of MSF have been written since its introduction in 1979. Typically, anonymous feedback regarding specific behaviors or outcomes (such as negotiation styles) is collected from the target person and from people familiar with the target person. The data is then summarized and reported back to the target. This information can be used to appraise past performance, to motivate change, or to set new goals for the future. Research has found that when information on an individual's performance is obtained from multiple perspectives, it provides richer and more varied data (Funderburg & Levy, 1997), and enhances the credibility of the feedback, thereby increasing the motivation to change (Hellervik, Hazucha, & Schnieder, 1992). |
| Proposition # 3: Negotiation training evaluation data that are collected from a variety of sources are likely to offer a more accurate and credible assessment of behavioral change in the participant than when self-report methods are used in isolation. |
| The timing of measurement is another important consideration. For purposes of convenience, negotiation training evaluation data are commonly collected at the immediate conclusion of a training session. This method is questionable in its valid assessment of the transfer of the effects of the training into the actual lives of the participants. Data collection that is conducted so soon after an intervention is susceptible to the "Hawthorne effect", where short-term effects of an intervention appear to be present as a result of the attention participants received from trainers and other participants, and not due to any direct effect of the content of the training. These "attention" effects tend to dissipate shortly after the training. |
| Proposition # 4: Negotiation training evaluation data that are collected at a time-delay (of several weeks) are likely to offer a more accurate assessment of transfer of training to the participant than data that are collected immediately after an intervention. |
| The Negotiation Evaluation Survey (NES) |
| The general rationale for the development of the Negotiation Evaluation Survey (NES) was to create an instrument based on a comprehensive conceptualization of the hypothesized effects of training in collaborative negotiation that addressed the conceptual and methodological concerns previously outlined. The two main objectives for this instrumentation were 1) to create a multi-source developmental tool for feedback and coaching of training participants and 2) to evaluate the effects of training on individuals and on the social systems in which the participants live and work (e.g. classroom, workplace, community). |
| The universe of constructs for the instrument were assembled from current literature on negotiation training (Lewicki, 1997; Deutsch, 1995, Raider 1995) as well as from the specific parameters of the collaborative negotiation training model under assessment (Coleman/Raider Model,?). These various cognitive, affective, attitudinal, behavioral, environmental, and outcome-related constructs were organized by our conceptual model and are presented in Table 2. |
| The process of developing the final version of the NES presented in this article involved repeated iterations over two years. The scales are based on theoretical reflection, piloting, factor analyses, feedback from experiment participants, and specific attempts to increase factor loadings and internal consistencies by item analyses and revisions. |
| The NES is currently used as an integral part of our basic training in collaborative negotiation at the ICCCR. As such, we have framed the use of the instrument in a manner that we believe facilitates openness and yet provides valuable developmental opportunities for the participants. We request that participants fill out Part A of the instrument, and that they give Part B to 3 people that know them well (one person from their personal life and a superior and subordinate [if possible] from their professional life). The participants are informed that these same individuals will be asked to respond to the questionnaire again in several months and that the information from this questionnaire will be kept confidential. |
| We ask the participants to give the survey to others for several reasons. First, we have found that doing so can give participants a better sense of not only how they feel they respond when in conflict with others but also of how others feel they respond. This information can be invaluable in giving them a clearer sense of their strengths and areas for improvement in their conflict resolving skills. Second, we have found it useful for people to think about and get feedback from others in both their personal and professional lives, as well as from both bosses and subordinates. This kind of information can give people a better sense of how they might respond differently in different situations. Finally, we believe it is important that the participants approach the feedback they get as the beginning of a discussion that they may wish to have about their negotiation skills, both with the trainers and other participants in the workshop as well as with those individuals who were kind enough to provide them with honest feedback. |
| Evaluation of the Coleman/Raider Model with the NES |
| The following studies are presented to demonstrate the use of the NES for negotiation training evaluation. Two studies were conducted to evaluate the effects of the Coleman/Raider Model of collaborative negotiation on participant's cognition, affect, attitudes, and behaviors, and on the resulting negotiation outcomes and workplace climate. The first study was a simple pre-post measure of a single training group (with a 3-month delay between measures). The second study used a quasi-experimental design to evaluate the independent and interaction effects of the training and the pre-test instrument. |
| Study 1 |
| Methods |
| Sample. A total of 31 students and 93 feedback-respondents participated in this study. Training participants were 31 students who attended the Basic Practicum in Conflict Resolution course at Teachers College, Columbia University. 88 students enrolled in the course but only 31 students returned both the pre- and post-tests (35% return rate). Of the 31 students: 8 (26%) were males, 23 (74%) were females; 2 (6.5%) were below the age of 25, 23 (74%) were between 25 and 34 years, 2 (6.5%) were between 35 and 44 years, 1 (3%) was 45 and over, and 3 (10%) did not indicate their age. 16 students (51%) indicated that they had some prior conflict resolution experience while 15 (49%) indicated that they had no prior experience. Participants received approximately 20 hours of collaborative negotiation training taught over 2.5 days. |
| Design. Study 1 was a single group, pre-test / post-test design. |
| Dependent Measures. These were paper-and-pencil measures that employed multi-source (or 360-degree) feedback in which participants were requested to fill out a self-report survey and elicit feedback from three people who knew them well about their perceptions of the participants' behavior when engaged in conflict with them. Pre- and post-tests consisted of 4 parts: (a) Part A was filled out by the participant; (b) Part B was given to a personal friend or family member; (c) Part C was given to a supervisor at work; and (d) Part D was given to a subordinate or colleague. Items in the pre- and post-tests were based on 2 scales: a frequency scale which measured frequency of observable behaviors (from 1 [Never] to 7 [Always]); and an agreement scale which measured level of agreement with statements regarding conflict-related cognition, attitudes, affect, outcomes and climate (from 1 [Disagree] to 7 [Agree]). |
| Procedure. Participants were instructed to facilitate the completion of all 4 pre-test instruments (self and 3 others) prior to the first day of the training. Individualized negotiation base-line profiles were generated for each participant from this data, distributed, and reviewed during the course of the 20-hour training. One month after the training, participants were instructed to facilitate the completion of all 4 post-test instruments (self and 3 others) and to return to the course instructors. |
| Analysis |
For purposes of clarity, composite variables were computed for each of the categories from our conceptual model (Figure 1). The dependent variables were as follows:
|
| Responses from self and other were collapsed for this analysis (any discrepancies between self and other were summarized and presented to the participants in pre-test personal profiles). T-tests were then conducted to assess the differences between pre- and post-tests among the dependent measures. |
| Results |
| The findings from Study 1 are presented in Table 2. Overall, the training appeared to have positive effects on the participants' collaborative negotiation skill-sets. The results indicated a significant shift in the use of constructive conflict behaviors, with more use of opening (p<.01) and uniting (p<.01) behaviors after training. There was also a significant reduction in attacking (p<.001) and negative evading (p<.001) behaviors after training. Responding to how they felt when engaged in conflicts with others, participants reported a significant increase in positive emotions (p<.01) and a significant decrease in negative emotions (p<.01). While there were no significant effects for conflict-related cognition, outcomes, or work climate, trained participants indicated that their attitudes toward conflict became more cooperative after the training (p<.01). |
| Discussion |
| The results from Study 1 were promising, although lacking with regard to environmental effects. That is, even when correcting for self-report bias and short-term attention effects, we still saw positive changes in attitudes, affect, and behavior a month after training. The absence of change in conflict-related cognition was puzzling, perhaps accounted for by the small sample size and the fact that 50% of the sample had received some prior training in conflict resolution. However, despite the effects we did find, we became concerned about the potential confounding effects of the instrument itself. In other words, was the act of reflecting on one's own skills, eliciting one's friends and colleagues responses, and reviewing one's base-line data accounting for the effects we witnessed in Study 1? Our next study attempted to answer that question. |
| STUDY 2 |
| Method |
| Participants. A total of 106 students and 318 feedback-respondents participated in this study. Participants in the treatment groups were 64 students who attended the Basic Practicum in Conflict Resolution course at Teachers College, Columbia University. Approximately 124 students enrolled in the course but only 64 students returned both the pre- and post-tests (52% return rate). Of the 64 students: 19 (30%) were males, 45 (70%) were females; 5 (8%) were below the age of 25, 48 (75%) were aged between 25 and 34 years, 4 (6%) were aged between 35 and 44 years, 2 (3%) were aged 45 and over, and 5 (8%) did not indicate their age. 30 students (47%) indicated that they had some prior conflict resolution experience, 28 (44%) indicated that they had no prior experience, and 6 (9%) did not indicate their prior experience. Participants in the control groups were 42 students attending other Teachers College courses. These students were given extra credit for participating in this study. Of the 42 students: 7 (17%) were males, 35 (83%) were females; 3 (7%) were below the age of 25, 32 (76%) were aged between 25 and 34 years, 3 (7%) were aged between 35 and 44 years, 2 (5%) were aged 45 and over, and 2 (5%) did not indicate their age. 3 students (7%) indicated that they had some prior conflict resolution experience, 37 (88%) indicated that they had no prior experience, and 2 (5%) did not indicate their prior experience. |
| Design. A quasi-experimental Solomon 4-group design was used for Study 2 (see Table 3). Using previously existing groups (sections of the same course), participants in the treatment group were assigned to 1 of 2 treatment conditions. In the first condition (Group 1, n=32), participants received the pre-test, conflict resolution training, and the post-test. In the second treatment condition (Group 2, n=32), participants received conflict resolution training and the post-test only. Participants in the control groups were also divided into 2 conditions. In the first condition (Group 3, n=20), participants completed both the pre-test and the post-test but received no training. In the second control condition (Group 4, n=22), participants completed the post-test only without training. |
| Procedure. Participants in Groups 1 and 3, were instructed to facilitate the completion of all 4 pre-test instruments (self and 3 others) prior to the first day of the training. Individualized negotiation base-line profiles were generated for participants in Group 1 only as part of the training (no base-line data from pre-test was available for Group 2). One month after the training, participants in all 4 groups were instructed to facilitate the completion of all 4 post-test instruments (self and 3 others) and to return to the course instructors. |
| Analysis |
| The composite variables computed for Study 1 were used in Study 2. A 2 x 2 analysis of variance was conducted to determine the main effects of training, pre-test, and the interaction effects. When there was a significant effect, t-tests were conducted to interpret the results. |
| Results |
| The results of the 2 x 2 ANOVA indicated that there was no significant effect of the pre-test on any of the dependent variables. This analysis also indicated no interpretable interactions between pre-testing and training. Therefore, only the main effects of the training are reported. |
| The results from the t-tests comparing the treatment and control conditions for Study 2 are presented in Table 4. In general, when compared to the control condition, training had significant effects on most of the participants' collaborative negotiation skills, outcomes, and work-climate. The was a significant use of constructive conflict behaviors after training, with more use of opening (p<.001), uniting (p<.001), and informing (p<.001) behaviors by trained participants than non-trained. There was also significantly less attacking (p<.001) and negative evading (p<.01) behaviors by trained than non-trained participants. Responding to how they felt when engaged in conflicts with others, trained participants reported significantly less negative emotions (p<.01) than non-trained. Trained participants also indicated that their attitudes toward conflict were more cooperative (p<.05) and less competitive (p<.001) and that their thinking regarding conflict was more constructive (p<.001) than participants in the control conditions. Finally, there was a significant difference in constructive conflict outcomes (p<.05), constructive work climate (p<.001), and destructive work climate (p,>001) between the groups, with the collaborative negotiation training having substantially more positive effects on outcomes and work climate after a month when compared with no training. |
| Discussion |
| In general, the findings from Study 2 are compelling. The data overwhelmingly support the assertion that a 20-hour training in collaborative negotiation can have a substantial effect on how people think, feel, and act in negotiation, and that these differences can effect the outcomes they achieve and the more general climate of their work environment. We found no direct effects of the instrument alone on the dependent variables, offering more support for the main effects of the training. These results were found even though the raters included people other than the training participants themselves, and despite the one-month delay in post-test measurement. The results were more robust than Study 1, perhaps due to the significantly larger sample size. These data offer strong support for the effective transfer of the training objectives into the actual lives of the training participants. |
| Conclusion |
| This research was motivated by a need in the field of negotiation for more specific and comprehensive conceptual models of the effects of negotiation training and for more rigorous and systematic methods for training evaluation. A model for conceptualizing the change effects of training in six distinct yet interdependent sub-systems of individual negotiators was presented. The Negotiation Evaluation Survey (NES), a time-delayed, multi-source feedback approach to assessment and development, was introduced by way of an example assessment of one model of collaborative negotiation training for adults, The Coleman/Raider Model. This approach to evaluation holds much promise for future basic, applied, and action research on negotiation. |
| For example, the NES could be used to explore the relationship in training between skill acquisition, knowledge enhancement, and changing attitudes and feelings regarding conflict and negotiation. Most approaches to training bundle these all together and approach them with the same adult-learning pedagogy. However, perhaps there are different exercises that are more and less useful for addressing different modes of the participant's experience. And perhaps there is a particular sequence that can best move participants through their anxiety about change into a more receptive place for learning a new orientation and new skills for negotiation. And how might differences in learning styles interact with this? Of course, these are questions individual trainers address every day, but we as a field are not currently compiling and testing this implicit knowledge. |
| The multi-source feedback approach of the NES offers much potential for examining contextual factors as well. Questions such as "Does negotiation training that is effective in one's work life transfer to how one handles conflict in his or her personal life as well?" could be examined. The NES could also allow us to explore how differences in relative power (when negotiating with superior, equal or subordinate others) can effect the acquisition and use of new skills. |
| The NES could be a particularly valuable tool for action research. The instrument could allow for the ongoing assessment and refinement of specific aspects of a training model, such as its capacity to motivate participants sufficiently to try new strategies. This could assist in the development of standing training programs. The NES could also be modified to examine the effects of training on specific aspects of a given work environment (such as out-of-role encounters) that are seen as obstacles to the use of skilled negotiation strategies. |
| References |
| John, O. P. & Robins, R. W. (1994). Accuracy and bias in self-perception: Individual differences in self-enhancement and the role of narcissism. Journal of Personality and Social Psychology, 66 (1), 206-219. |

1. Cognitive & Analytical Skills:
|
2. Affect:
|
3. Attitudes:
|
4. Behavioral and communication skills:
|
5. Outcomes:
|
6. Climate:
|
| Dependent Variables | Mean Difference1 | Standard Deviation | t |
1. Constructive Conflict Behaviors (n=124)
|
.043 .14 .25 .31 |
1.46 .98 .96 1.06 |
.33 1.62 2.90** 3.21** |
2. Destructive Conflict Behaviors (n=124)
|
-.41 -.50 |
.90 1.26 |
-5.12*** -4.37*** |
3. Emotions in Conflict (n=31)
|
.47 -.44 |
.80 .88 |
3.30** -2.80** |
4. Attitudes toward Conflict (n=31)
|
.52 -.24 |
1.09 1.03 |
2.64** -1.28 |
5. Conflict Outcomes (n=124)
|
.11 |
1.17 | 1.08 |
6. Cognition toward Conflict (n=31)
|
.18 |
.69 | 1.46 |
7. Work Climate (n=93)
| .01 -.11 |
1.51 1.07 |
.062 -1.00 |
| Mean Difference1 is the subtraction of pre-test mean from post-test mean, i.e., post-test mean minus pre-test mean * p < .05 ** p < .01 *** p <.001 | |||
| Group | Condition | N | Pre-test | Training | Post-test |
| 1 | Treatment | 32 | Yes | Yes | Yes |
| 2 | Treatment | 32 | No | Yes | Yes |
| 3 | Control | 20 | Yes | No | Yes |
| 4 | Control | 22 | No | No | Yes |
| Dependent Variables | Mean Difference1 | Standard Deviation | t |
1. Constructive Conflict Behaviors (n=424)
|
.06 .10 .14 .18 |
.68 .44 .44 .48 |
.71 2.22* 3.11** 3.69*** |
2. Destructive Conflict Behaviors (n=424)
|
-.19 -.18 |
.51 .56 |
-3.63*** -3.30** |
3. Emotions in Conflict (n=106)
|
.20 -.28 |
.54 .46 |
1.81 -3.04** |
4. Attitudes toward Conflict (n=106)
| .22 -.38 |
.47 .47 |
2.37* -4.16*** |
5. Conflict Outcomes (n=424)
|
.15 |
.62 | 2.32* |
6. Cognition Regarding Conflict (n=106)
|
.58 |
.29 | 10.23*** |
7. Work Climate (n=318)
| .26 -.28 |
.44 .60 |
5.18*** -3.93*** |
| Mean Difference2 is the subtraction of control group mean from treatment group mean, i.e., treatment group mean minus control group mean * p < .05 ** p < .01 *** p <.001 | |||