Stanford Seminar - Accounting for Human Engagement Behavior to Enhance AI-Assisted Decision Making

Introduction

• Recent research has focused on enhancing AI-assisted decision-making, as AI technology has made significant progress in the past decade and is now widely applied in various areas, including finance, healthcare, and criminal justice 12s.
• AI models have been trained to uncover hidden insights from big data, and in decision-making tasks, they can analyze the task, come up with decision recommendations, and present them to human decision-makers to assist in their decision-making 47s.
• The collaboration between human decision-makers and AI models has transformed decision-making from a human-only task to a joint task, with the hope that they can complement each other's strengths and weaknesses and achieve better decision-making performance 1m14s.
• However, this human-AI collaboration is rarely observed in reality, and it is essential to understand why and how to improve AI-assisted decision-making performance 1m38s.

Improving AI-Assisted Decision-Making

• To improve AI-assisted decision-making, two essential steps are necessary: obtaining more empirical understandings of how human decision-makers engage with AI-based decisions and leveraging this understanding to improve the designs of AI-based decisions 1m59s.
• Empirical research has identified that human trust and reliance on AI are shaped by various performance indicators of the AI model, especially its performance observed in the field 3m30s.
• Humans' confidence in their own independent judgment also influences how much they are willing to listen to AI's suggestions, and they are more likely to be receptive to AI's decision recommendations when their own decision confidence is relatively low 3m52s.
• In cases where humans do not observe AI models' performance, it is essential to understand how they decide how much to rely on AI's decision recommendations 4m16s.
• A human subject experiment was conducted to study how humans gauge the trustworthiness of AI models when performance information is absent, and it was found that humans tend to use the level of agreement between AI model decisions and their own independent judgment to assess trustworthiness 4m20s.
• In the experiment, human subjects were asked to complete speed dating prediction tasks, where they were given a speed dating profile containing demographic background information and other relevant details, and then asked to make their own independent judgment on whether the participant would be willing to see their date again 4m49s.
• After making their independent judgment, the subjects were shown an AI model's prediction as a decision recommendation, and then asked to submit their final decision, with the option to follow the AI's suggestion or not 5m44s.

Experimental Design and Methodology

• A pilot study was conducted to categorize different decision-making tasks into several categories or groups based on the decisions and confidence levels of the human subjects 6m2s.
• The pilot study found that decision-making tasks can be categorized into four groups: tasks where the majority of subjects make correct decisions with high confidence, tasks where subjects make wrong decisions with high confidence, and tasks where subjects make correct or wrong decisions with low confidence 6m40s.
• The formal experimental design involved asking subjects to complete a sequence of 40 decision-making tasks with the help of an AI model, without providing information on the AI model's performance, and the tasks were divided into two phases of 20 tasks each 7m42s.
• The first 20 tasks in Phase One were selected from the first two categories of tasks identified in the pilot study, which included tasks where the majority of subjects make correct decisions with high confidence and tasks where subjects make wrong decisions with high confidence 8m11s.
• The experiment involves three experimental treatments that vary in the frequency of AI model decision recommendations agreeing with human majority judgment on 20 tasks in phase one, with agreement levels ranging from 40% to 100% 8m30s.
• In phase one, human subjects make independent decisions on the same 20 tasks but receive different AI model recommendations based on their assigned treatment, resulting in varying levels of agreement between AI and human judgment 9m10s.
• The AI model's actual accuracy is kept the same across all treatments to avoid confounding variables, achieved by setting AI recommendations to match human majority judgment in the high agreement treatment and flipping AI recommendations on tasks where humans are correct or incorrect in the low agreement treatment 9m45s.
• The low agreement AI model is designed to complement human expertise by being correct in cases where humans are wrong, making it a model that complements human expertise the most 10m38s.
• In phase two, the goal is to determine if subjects will trust or rely on AI differently after experiencing varying levels of agreement between AI and human judgment in phase one 11m1s.
• The 20 tasks in phase two are selected from the third and fourth categories of instances in the pilot study, where humans tend to make independent judgments with low confidence 11m24s.
• In phase two, subjects receive the same AI model recommendations across all treatments, simulating how the AI will perform, and the AI model's predictions are assigned to simulate its performance 11m49s.
• To quantify subjects' trust and reliance on an AI model, researchers asked subjects to report their perception of the AI model at the middle and end of the experiment, and also looked at how frequently human subjects' final decisions matched the AI model's decision recommendations, especially when their initial judgments differed from the AI's recommendations 12m23s.
• The metrics used to quantify people's reliance on AI were agreement and switch fractions, with higher values indicating more reliance on AI 12m57s.

Results and Findings

• The experiment found that as the level of agreement between the AI model and humans' independent judgments increased, humans generally reported seeing the AI model as more reliable, trustworthy, and understandable, and relied more on its decision recommendations 13m12s.
• The findings suggest that humans tend to use the level of agreement between AI and their own independent judgments as a heuristic to gauge the trustworthiness of the AI model, especially when they lack performance information about the AI 13m51s.
• This tendency is related to the human cognitive bias of confirmation bias, where individuals tend to follow recommendations from those who agree with them and ignore those who disagree 14m17s.
• The study found that humans may exhibit confirmation bias towards AI models that provide decision recommendations, which can lead to risks such as over-trusting AI models' wrong recommendations and under-trusting correct recommendations 14m34s.
• The results also showed that humans may miss the opportunity to fully realize the potential of human-AI complementary decision-making due to their tendency to under-trust AI models that complement their own expertise 16m7s.

Implications for AI System Design

• The study's findings have implications for the design of AI-assisted decision-making systems, highlighting the need to consider human cognitive biases and develop strategies to mitigate their effects 16m29s.
• Human-AI collaboration may be vulnerable to attacks by adversary parties who can exploit human confirmation bias towards AI, leading to a loss of trust in AI, and this vulnerability can be strategically targeted using computational models like hidden Markov models to minimize the cost of attack and maximize the reduction of human-AI joint team performance in decision-making 16m55s.
• Adversary parties can use computational models to categorize their attacks on AI models and influence people's trust and reliance on the AI model, and develop strategies to deploy these attacks 17m12s.
• It is urgent to leverage our understanding of human engagement behavior with AI to combat adversary parties and secure human-AI collaboration 17m48s.

Estimating and Communicating Correctness Likelihood

• To improve human-AI team performance in decision-making, humans need to collaborate with AI models by combining their independent judgment with AI decision recommendations, but humans often leverage their independent judgment in a way that does not utilize their strengths and avoid their weaknesses 18m29s.
• To help humans utilize their strengths and avoid their weaknesses, it is necessary to let them know where their strengths and weaknesses are, especially in comparison to AI, and estimate the likelihood of human and AI decision correctness on individual decision-making tasks 19m4s.
• Estimating the correctness likelihood of human and AI decisions is a practical challenge, and using AI calibrated confidence scores as an indication of correctness likelihood may be easier for AI, but estimating human correctness likelihood on different decision-making tasks in real-time is a challenge 19m50s.
• Communicating the correctness likelihood estimation to human decision-makers in a way that encourages them to take more rational actions by either following AI or themselves is another challenge 20m25s.
• There are two main challenges in accounting for human engagement behavior to enhance AI-assisted decision making: estimating humans' correctness likelihood on different decision-making tasks and communicating this information back to the human decision maker 20m39s.
• Estimating humans' correctness likelihood is complicated, especially when the ground truth answer is unknown, so an important assumption is made that if a person's decision on a similar task is correct, their decision on the current task is more likely to be correct 21m9s.
• To estimate humans' correctness likelihood, a four-step workflow is developed, starting with having human decision makers complete a small set of decision-making tasks, then learning a decision tree to categorize their decision-making process, extracting rules from the decision tree, and presenting the rules back to the human decision maker for modification 22m6s.
• Using the user-modified set of decision rules, the predicted decisions of the human decision maker on each of the training task instances of the AI model's training data set are determined, and whether those predicted decisions are correct or not is assessed 23m1s.
• To estimate the decision maker's correctness likelihood on a new task instance, a number of training task instances similar to the new task instance are fetched, and the frequency of correct predicted decisions on those similar training task instances is used as the correctness likelihood estimation 23m21s.
• The second challenge is communicating this information back to the human decision maker, and one approach is to directly display the estimated correctness likelihood of the human decision maker and the AI model's calibrated confidence score on the interface 24m8s.
• The direct display approach presents both types of information to the human decision maker and asks them to process the information and react accordingly 24m36s.
• The initial intervention involves directly displaying information to users in the way they prefer, serving as a straightforward approach to communication 24m42s.
• However, this approach may be too blunt, leading to the consideration of more subtle and indirect methods of communication, such as using interfaces that decrease human reliance on AI 25m1s.
• Previous research has shown that certain interfaces in AI-assisted decision-making processes can decrease human reliance on AI, but this decrease occurs regardless of whether the AI is correct or incorrect 25m30s.
• Despite this limitation, these interfaces provide opportunities for indirect communication of correctness estimation between humans and AI, allowing for the design of adaptive interfaces that decrease human reliance on AI when necessary 26m1s.

Interventions for Enhanced Decision-Making

• The second intervention, called adaptive workflow, involves adaptively deciding when to present AI model decision recommendations to human decision-makers based on the likelihood of human or AI correctness 26m48s.
• If AI is more likely to be correct, the AI's decision recommendation is shown upfront; otherwise, humans are asked to make their own independent judgment before seeing the AI's recommendation 27m12s.
• The third intervention, called adaptive recommendation, involves adaptively changing whether to provide AI model recommended decisions to humans based on the AI's correctness likelihood 27m46s.
• If the AI's correctness likelihood is higher, both the AI's explanation and recommended decision are shown; otherwise, only the AI's explanation is provided 28m7s.
• A human subject experiment was conducted to test whether interventions can improve AI-assisted decision-making performance, and the results showed that providing three types of decision recommendation interventions to human decision makers led to a marginal or significant increase in their final decision-making accuracy 28m53s.
• The increase in decision-making accuracy was mostly caused by the decrease of humans' overreliance on AI when AI makes a wrong decision recommendation, and the interventions did not significantly affect people's trust and reliance on AI when AI makes correct decision recommendations 29m45s.

Behavior-Aware AI Models

• The traditional approach to improving AI-based decision making is to adaptively change how AI's recommendation is presented to human decision makers, but an alternative approach is to retrain the AI model to anticipate humans' engagement behavior and accommodate to humans' engagement behavior 30m12s.
• This requires moving away from training behavior-oblivious AI models that optimize for AI's own decision-making performance, and instead training behavior-aware AI models that explicitly account for how humans will react to AI models' decision recommendations 30m53s.
• A recent study used a human behavior model inspired by empirical understandings of humans' engagement with AI, which assumes that human decision makers will go with their own independent judgment when their confidence in their own judgment is higher than a threshold, and otherwise will go with AI models' decision recommendations 31m31s.
• The study found that optimizing for the human-AI team performance in decision making is effectively the same as trying to solve a weighted loss optimization problem, where the weight assigned to each training task instance is one minus the CDF function value for humans' own decision confidence on the training instance 32m10s.
• The results of the human subject study and simulation showed that when humans collaborate with behavior-aware AI models, they are able to achieve a significantly higher level of final decision-making accuracy than when they collaborate with behavior-oblivious AI models 32m42s.

AI-Assisted Group Decision-Making

• The goal is to enhance AI-assisted decision making, but in reality, many decisions are made by a group of people, so it's essential to think about how to enhance AI-assisted group decision making 33m3s.
• To achieve this, it's necessary to start by obtaining a better understanding of how a group of decision makers will engage with AI-based decisions and then design AI-based decisions that account for the group's engagement behavior 33m35s.
• Research was conducted through human subject studies, where decision makers made recidivism predictions by looking at the profiles of criminal defendants and deciding whether they may reoffend in the future 34m31s.
• Two experimental treatments were created: one where decision makers made independent judgments with the help of an AI model, and another where decision makers were organized into groups of three to discuss and come up with a consensus decision with the help of an AI model 34m50s.
• The study looked into various aspects of how individual and groups engage with AI, including final decision-making accuracy, reliance on AI, fail confidence, accountability, and understanding of AI 35m26s.
• One of the most striking results was that groups of decision makers relied on AI's decision recommendations even more than individual decision makers, both when AI was correct and when AI was wrong 35m51s.
• This led to a lower degree of under-reliance but a higher degree of over-reliance for groups, and further analysis of chat logs revealed that group members often used AI's agreement with their own judgment to persuade others to vote together with them 36m22s.
• Another frequently observed pattern in group discussions was the use of AI's decision recommendations to support their own judgments and persuade others 36m56s.
• In some cases, a group cannot reach a consensus, and someone may suggest using an AI model's recommendation as a tiebreaker, which can significantly increase the chance of people using AI models' decision recommendations as their final decision 37m2s.
• To improve AI-assisted group decision-making performance, it is necessary to help groups critically reflect on the trustworthiness of AI models' decision recommendations and engage in more critical deliberation 37m26s.

The Devil's Advocate Approach

• Introducing a devil's advocate in the group discussion process is a frequently used approach to encourage critical deliberation, but the traditional type of devil's advocate may suffer from limitations, such as the assigned person not actually agreeing with the opposing views they are asked to present 37m54s.
• Another limitation of the traditional devil's advocate approach is that the person assigned to this role may suffer from psychological safety issues, feeling isolated in the process 38m40s.
• To address these limitations, using a large language model to power the devil's advocate is a potential solution, as it can generate more genuine opposing views without suffering from psychological harms 39m0s.
• The language model-powered devil's advocate can be designed to vary in two dimensions: who the devil's advocate should object to, with options including opposing the majority view or specifically advocating against AI models' decision recommendations, and the interactivity of the devil's advocate, with options including a static or more interactive design 39m54s.
• The simplest design is a static devil's advocate that generates critical initial questions to encourage groups to engage in critical thinking, but more interactive designs are also possible 40m44s.
• A Dynamic Devil's Advocate, a language model that actively participates in group discussions and responds to other members' arguments, can enhance AI-assisted decision-making 41m8s.
• This approach was tested in a human subject experiment, which found that the Dynamic Devil's Advocate can help decision-makers achieve higher decision-making accuracy and decrease their overreliance on AI recommendations 41m34s.
• The language model-powered Devil's Advocate exhibits emerging behaviors, such as encouraging group members to express their opinions and ensuring everyone's voice is heard, even when not explicitly prompted 42m27s.
• The Devil's Advocate also recognizes when the group's decision is based on wrong information and corrects them, and encourages holistic evaluation of the case 43m2s.
• Additionally, the Devil's Advocate identifies and challenges assumptions made by the group, promoting critical thinking and more informed decision-making 44m5s.
• The benefit of the Dynamic Devil's Advocate comes from the interaction between the people seeing the adversarial interaction, which helps the group come to a better decision, rather than just the AI system itself being better in this environment 44m51s.
• The role of AI in decision-making can be enhanced by having a "Devil's Advocate" that challenges AI suggestions and encourages critical reflection, especially when the AI model's performance is on par with human performance 45m56s.
• The effectiveness of the Devil's Advocate approach may not generalize to settings where the AI model's performance is significantly better than human performance, and it may be necessary to redesign the approach to account for such cases 46m14s.

Experiment with a Language Model-Powered Devil's Advocate

• In an experiment, a Devil's Advocate powered by a language model was introduced to a group of decision-makers, and the group was informed about the presence of the Devil's Advocate 46m57s.
• The experiment used a recidivism prediction task, where participants were asked to predict whether a defendant would reoffend within two years, and the data set provided the ground truth answer 47m24s.
• The language model was prompted to generate thought-provoking questions and encourage deliberation, but it was observed to play a facilitator role, which may be due to its design or its ability to adapt to the situation 49m1s.
• The language model's ability to adopt a facilitator role may be a result of its strategy to engage in arguments and encourage discussion, even when playing the role of a Devil's Advocate 48m40s.
• The experiment highlights the importance of considering the role of human engagement behavior in AI-assisted decision-making and the need to design AI systems that can effectively interact with humans 45m39s.
• Language models often try to ensure everyone is contributing equally and considering all possible features of a task, but the reason behind this behavior is not well understood 49m33s.
• Future work could involve engineering language models to separate roles, such as acting as a facilitator or coordinator, and throwing out opposing views to see how different roles interact and which is most helpful in encouraging group decision-making accuracy 49m51s.

Future Directions and Conclusion

• Understanding human engagement with AI is crucial as it can help design adaptive AI that promotes more appropriate engagement from humans and potentially retrain AI models to tailor to human engagement behavior 50m29s.
• The goal is to promote a more appropriate engagement from the human side and to design AI that can adapt to human behavior, leading to more effective human-AI collaboration 50m37s.