Rankers, Rankees, & Rankings: Peeking into the Pandora's Box from a Socio-Technical Perspective

This paper has been published in Workshop on Interrogating Human-Centered Data Science at CHI 2022. The authors of this paper are:

Jun Yuan PhD student, Department of Data Science, New Jersey Institute of Technology, USA
Julia Stoyanovich Associate Professor, Tandon School of Engineering and Center for Data Science, New York University, USA
Aritra Dasgupta Assistant Professor, Department of Data Science, New Jersey Institute of Technology, USA

Abstract

Algorithmic rankers have a profound impact on our increasingly data-driven society. From leisurely activities like the movies that we watch, the restaurants that we patronize; to highly consequential decisions, like making educational and occupational choices or getting hired by companies-these are all driven by sophisticated yet mostly inaccessible rankers. A small change to how these algorithms process the rankees (i.e., the data items that are ranked) can have profound consequences. For example, a change in rankings can lead to deterioration of the prestige of a university or have drastic consequences on a job candidate who missed out being in the list of the preferred top-k for an organization. This paper is a call to action to the human-centered data science research community to develop principled methods, measures, and metrics for studying the interactions among the socio-technical context of use, technological innovations, and the resulting consequences of algorithmic rankings on multiple stakeholders. Given the spate of new legislations on algorithmic accountability, it is imperative that researchers from social science, human-computer interaction, and data science work in unison for demystifying how rankings are produced, who has agency to change them, and what metrics of socio-technical impact one must use for informing the context of use.

Fig: • The ecosystem of ranking producers and consumers.

Online Access

Paper: ResearchGate(PDF Version)

BibTeX Reference

        @article{yuan2022rankers,
          title={Rankers, Rankees, \& Rankings: Peeking into the Pandora's Box from a Socio-Technical Perspective},
          author={Yuan, Jun and Stoyanovich, Julia and Dasgupta, Aritra},
          journal={arXiv preprint arXiv:2211.02932},
          year={2022}
        }