“Is your machine better than you? You may never know.”

Information Systems and Operations Management

Speaker: Francis de Vericourt (ESMT)

Abstract: 

AI systems are increasingly demonstrating their capacity to make better predictions than human experts. Yet, recent empirical studies suggest that professionals sometimes doubt the quality of these systems, and as a result overrule machine-based prescriptions. This paper explores the extent to which a decision maker (DM) can properly assess whether a machine produces better recommendations. To that end, we analyze an elementary dynamic Bayesian framework, in which a machine performs repeated decision tasks under a DM’s supervision. The task consists in deciding whether to take an action or not. Crucially, the DM observes the accuracy of the machine’s prediction on the task only if she ultimately takes the action. As she observes the machine’s accuracy, the DM updates her belief about whether the machine’s predictions outperform her own. Depending on this belief, however, the DM sometimes overrides the machine, which affect her ability to assess it.

In this set-up, we characterize the evolution of the DM's belief and overruling decisions over time. We identify situations under which the DM’s belief oscillates forever, i.e., the DM always hesitates whether the machine is better. In this case, the DM never fully ignores the machine but regularly overrules it. We further find that the DM’s belief sometimes converges to a Bernoulli random variable, i.e., the DM ends up wrongly believing that the machine is better (or worse) with positive probability. We fully characterize the conditions under which these failures to learn occur. These results highlight some fundamental limitations in our ability to determine whether machines make better decision than experts. They further provide a novel explanation for why humans may collaborate with machines – even when one may actually outperform the other.

Joint work with Huseyin Gurkan (ESMT)