Synchrony facilitates altruistic decision making for non-human avatars
Jun
04,
2025
Negotiation is a ubiquitous social activity in which individuals work together to reach a consensus on issues that are in dispute. The complicated interplay of competition and cooperation is a remarkable feature of negotiation, and negotiators go into it with a highly mixed motivation. Most negotiators seek to reach an agreement with the other, but they also strive for an agreement that serves their own goals. Thus, effective negotiators must find a tenuous balance between facilitating positive and cooperative interactions within a competitive and often distrustful environment.
Prior research on negotiation emphasizes the importance of rapport, a feeling of connection, mutual attentive ness and positivity, in establishing cooperation and obtaining mutually beneficial outcomes. Rapport is known to be reflected in a particular pattern of nonverbal behavioral dynamics between interlocutors: interactional synchrony, hereafter, synchrony (Bernieri & Rosenthal, 1991; Fujiwara et al., 2020). Synchrony refers to ‘‘similarity in rhythmic qualities and enmeshing or coordination of the behavioral patterns of both parties’’ in an interaction (Burgoon et al., 1995, p. 128), which is believed a natural part of human interactions.
The research field of human-computer interaction, known as HCI, has emerged to bring together ideas from diverse fields such as psychology, engineering, computer science, design, sociology, and communication. Advances in automation now allow humans to have realistic interactions with virtual human interviewers and have interactive conversations that resemble human-human interactions. The degree to which the human and non-human agent behaviors are synchronized with one another goes a long way toward building rapport and improving outcomes. Thus, the purpose of this research was to examine whether human negotiators would synchronize their movements when working with non-human avatars and if so, whether that would affect their perceptions of their negotiations.
In formulating our hypotheses, we relied on two assumptions. Namely, one is that people unconsciously treat computers as social actors (the “media equation” effect), and the other is that synchrony breaks down when the competitive aspect of negotiation is emphasized. The three hypotheses formulated on the basis of these are as follows:
H1. Participants will synchronize their movements with the movements of their non-human negotiation partner.
H2. Greater synchrony will occur when participants engage in an integrative (win-win) negotiation compared to distributive (win-lose or zero-sum) negotiations.
H3. Participants that engage in synchrony will report greater impression of affiliation, a proxy of rapport, with their non-human partner.
People are not purely selfish in negotiations, but try to accommodate the other party’s goals somehow. The extent of this “other regard” is shaped by the relationship to the other party. Indeed, prior research suggests that affiliation towards one’s negotiation partner will influence concession-making, so in addition to these hypotheses, we also examine how synchrony impacts the outcomes participants obtain.
Methods
The 172 participants were instructed to negotiate with an avatar in order to come to an agreement, the maximum interaction time for this task was 10 min. The avatar was named “Sam” for both the male and female version of the avatar (Fig. 1). There were several items on the negotiation table, and participants were informed of the relative value of each item before starting the interaction. The specific values of these items were determined by the setting of the study; in the distributive (zero-sum) setting, items were worth the same amount for both participant and avatar. On the contrary, in the integrative condition, the item value differed between the participant and the avatar, allowing both the participant and the avatar to receive their highest value items. Participants were not told the avatar’s value of items in either condition, however they could discover this during the negotiation by asking questions or observing the avatar’s pattern of offers (the avatar always responded truthfully to questions about what it values). If no agreement was reached within the 10 min time limit, each party would only receive a single copy of their highest value item.
Fig. 1. Appearance of female and male avatar (“Sam”). The skeleton rendered by OpenPose is overlaid on each joint point of the avatar.
Participants could talk to the agent without any restrictions, and the avatar was controlled by two experimenters through a graphical user interface; i.e., the Wizard of Oz method. One experimenter controlled the agent’s speech and verbal behavior, while the other controlled the avatar’s nonverbal behavior such as facial expressions and gestures. The interactions were video recorded, which was used in the subsequent synchrony analysis.
Using video films of the negotiation, time-series bodily movement data was obtained using OpenPose (Cao et al., 2021; Fujiwara & Yokomitsu, 2021). Since a video is a series of still images, the movement time-series was created by calculating the distance the joint points traveled between each frame. For avatars, 10 coordinate points (i.e., eyes, nose, neck, shoulders, elbows, and hands/wrists) was used whereas 3 coordinate points (i.e., eyes and nose) were targeted for humans because many participants were videotaped with only their faces (i.e., above the chin). The sampling frequency was set to 30 Hz, which was equal to the video frame rate.
As for synchrony analysis, we performed the dynamic time warping for each dyadic time series (Figure 2). The distance score obtained was the main variable (as an inverse) indicating the degree of synchrony. To test the validity of a synchrony measure, we randomly shuffled data within each time series to generate artificial “interactions.” This technique is a time-series equivalent of a permutation test that offers a baseline to assess the level of synchrony in the genuine dyadic interactions.
Fig. 2. Example of dynamic time warping on movement time series: (A) A genuine human-avatar interaction, (B) A pseudo interaction of a randomly shuffled series. The heatmap represents the cost matrix created based on the difference between the two series; blue denotes smaller costs, red denotes larger costs. The black least-cost path minimizes the total cost (the cumulative addition of the costs on the path) from the start to the end of the matrix. In (A), the path can travel through the blue area, while the path must take the red area in (B), resulting in (A) having a smaller difference or a greater similarity. Deviations from the white diagonal show traces of adjustment using “warping” series, which align time series by their shape instead of time.
Results
We used the distance score to test H1 and H2. H1 predicted that the participants would synchronize their movement with the non-human avatar, which was supported, as the distance was significantly smaller in the genuine human-avatar dyad compared to the dyad with shuffled artificial data. H2 was also supported; the distance score was significantly smaller in the integrative (cooperative) condition than that in the distributive condition.
H3 predicted that synchrony with non-human avatars will lead to a favorable impression of the avatar. A mediation analysis that incorporated the impact of the experimental manipulation verified that the impact of the negotiation setting on impression of avatar was fully mediated by synchrony (Figure 3). Thus, H3 was supported.
Fig. 3. The effect of experimental manipulation on the impression of affiliation to the avatar is mediated by synchrony. Negotiation setting is binarily coded: integrative (1), distributive (0). All the estimates were standardized (**p < .01).
In addition to the hypothesis tests, we explored whether synchrony leads to better negotiation outcomes for the human player. The results showed that participants’ willingness to engage in synchrony boosted the avatar-friendly decision in the integrative negotiation where they could accomplish a win-win deal (Figure 4). Instead, synchrony had no impact on their outcomes in the zero-sum negotiation.
Fig. 4. The interaction effect of synchrony and the negotiation setting on the negotiation outcome. On the left is the human player's score: the more synchrony they show with the avatar (left side of the figure), the lower their score. On the right is the avatar's score: synchrony was associated with increased score.
Overall, these findings reinforce the importance of synchrony in human-machine interactions. Here, we demonstrated that human-machine synchrony predicted positive subjective feelings as well as the outcomes that were negotiated. Our findings further suggest that people seem strategic in their willingness to engage in synchrony (Dunbar et al., 2020), because they only synchronized their movements to the avatar partner when there appeared to be material benefits to this behavior in an integrative negotiation.
Also, our study may raise important practical and ethical questions for agent design. We complement a growing body of research that anthropomorphic interfaces can use subtle nonverbal signals to shape both perceptions and behavior. Some of these findings have obvious societal benefits such as enhancing physical and mental health. But other uses may be more problematic. For example, we have found that agents can extract greater concessions from a human negotiation partner by strategically using synthetic emotional expressions (de Melo et al., 2011) and potential consumers would want to own such an agent if it helped them manipulate others for their own personal gain (Mell et al., 2020). The field of artificial intelligence is becoming more aware of the need to work through ethical principles to guide the development of such technology. Our findings lend greater urgency to such efforts.
Related papers
Bernieri, F. J., & Rosenthal, R. (1991). Interpersonal coordination: Behavior matching and interactional synchrony. In R. Feldman, & B. Rime (Eds.), Fundamentals of nonverbal behavior: Studies in emotion & social interaction (pp. 401–432). Cambridge University Press.
Burgoon, J. K., Stern, L. A., & Dillman, L. (1995). Interpersonal adaptation: Dyadic interaction patterns. Cambridge University Press.
Cao, Z., Hidalgo, G., Simon, T., Wei, S., & Sheikh, Y. A. (2021). OpenPose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43, 172–186. https://doi.org/10.1109/TPAMI.2019.2929257
Dunbar, N. E., Giles, H., Bernhold, Q., Adams, A., Giles, M., Zamanzadeh, N., et al. (2020). Strategic synchrony and rhythmic similarity in lies about ingroup affiliation. Journal of Nonverbal Behavior, 44(1), 153–172. https://doi.org/10.1007/s10919-019- 00321-2
Fujiwara, K., Kimura, M., & Daibo, I. (2020). Rhythmic features of movement synchrony for bonding individuals in dyadic interaction. Journal of Nonverbal Behavior, 44(1), 173–193. https://doi.org/10.1007/s10919-019-00315-0
Fujiwara, K., & Yokomitsu, K. (2021). Video-based tracking approach for nonverbal synchrony: A comparison of Motion Energy Analysis and OpenPose. Behavior Research Methods, 53(6), 2700–2711. https://doi.org/10.3758/s13428-021-01612-7
Mell, J., Lucas, G., Mozgai, S., & Gratch, J. (2020). The effects of experience on deception in human-agent negotiation. Journal of Artificial Intelligence Research, 68, 633–660. https://doi.org/10.1613/jair.1.11924
de Melo, C., Carnevale, P. J., & Gratch, J. (2011). The effect of expression of anger and happiness in computer agents on negotiations with humans. Taipei, Taiwan: Paper presented at the Tenth International Conference on Autonomous Agents and Multiagent Systems.
