Bank for International Settlements tests ChatGPT for synthetic payment app surveys and flags bias and low response variability

The Bank for International Settlements published a working paper assessing whether generative AI can simulate consumer survey responses on mobile payment apps, focusing on privacy attitudes and perceived benefits. Using ChatGPT to generate synthetic survey answers and benchmarking them against a Dutch survey, the authors conclude that generative AI can complement survey work but cannot replace human surveys. Simulations with ChatGPT-4o broadly reproduced several patterns seen in the Dutch data, including that more privacy-concerned respondents view payment apps less favourably and perceive higher risks, and that users tend to rate apps as more beneficial and less risky than non-users even when user status is not explicitly linked to those attitudes in prompts. However, the synthetic responses showed materially lower dispersion than human responses and tended to overclassify respondents as “privacy fundamentalists,” with very few “privacy unconcerned” outcomes. The paper also finds that results are sensitive to prompt design, that increasing the model’s temperature did not meaningfully restore human-like variance, and that changing persona details or model version did not eliminate the privacy-skewed bias.