This paper examines the possibilities and limits of social accountability, characterized by face-to-face interactions between organizational actors and beneficiary others. We explore how social accountability is produced in and through visibility relations which recognize or may expose the other. We examine the consequences of such changing visibility relations on social accountability in two case studies of acute and chronic refugee crisis in Rwanda. Our study contributes to ongoing debates on accountability and the implications for responsible action towards the other. We argue that social accountability initially presents a way to recognize the other as valid and entitled to assistance, which spurs an important sense of responsibility. Prolonged exposure to social accountability, however, continuously reinforces the other as ‘in need’, which becomes a burden for the exposed as well as for the exposing actor in their response to a situation. We argue that exposure risks organizational actors developing a rather myopic perception of others’ needs, to the extent of becoming irresponsible in their action towards the other. We conclude the paper with reflections on the importance of proper distance for social accountability and outline the potential of our findings to inspire future work on social accountability and visibility relations in other organizational settings.