(Submitted on 2 Apr 2020)

Abstract: Autonomous agents (AA) will increasingly be interacting with us in our daily
lives. While we want the benefits attached to AAs, it is essential that their
behavior is aligned with our values and norms. Hence, an AA will need to
estimate the values and norms of the humans it interacts with, which is not a
straightforward task when solely observing an agent’s behavior. This paper
analyses to what extent an AA is able to estimate the values and norms of a
simulated human agent (SHA) based on its actions in the ultimatum game. We
present two methods to reduce ambiguity in profiling the SHAs: one based on
search space exploration and another based on counterfactual analysis. We found
that both methods are able to increase the confidence in estimating human
values and norms, but differ in their applicability, the latter being more
efficient when the number of interactions with the agent is to be minimized.
These insights are useful to improve the alignment of AAs with human values and
norms.