The importance of repeatability, replicability, and reproducibility is broadly recognized in the computational sciences, both in supporting desirable scientific methodology as well as sustaining empirical progress. The Open-Source IR Replicability Challenge (OSIRRC2019), organized as a workshop at SIGIR 2019, aims to improve the replicability of ad hoc retrieval experiments in information retrieval by building community consensus around a common technical specification, with reference implementations.

In order to precisely articulate the goals of this workshop, it is first necessary to establish common terminology. We use the below terms in the same manner as recent ACM guidelines pertaining to artifact review and badging:

These guidelines are also being adopted by the SIGIR community, which has recently established a task force to determine how to implement them for IR.

This workshop tackles the replicability challenge for ad hoc document retrieval, with three explicit goals:

  1. Develop a common Docker specification to support images that capture systems performing ad hoc retrieval experiments on standard test collections. The solution that we have developed is known as the jig.

  2. Build a curated library of Docker images that work with the jig to capture a diversity of systems and retrieval models.

  3. Explore the possibility of broadening our efforts to include additional tasks, diverse evaluation methodologies, and other benchmarking initiatives.

Participants in OSIRRC 2019 help in building consensus on the Docker specifications and contribute Docker images that conform to the agreed-upon specifications.

Interested individuals and teams are requested to email for more details or directly join the Google groups


We solicited two kinds of contributions from the community:

Docker images that encapsulate strong baselines as well as state-of-the-art techniques: examples include runs based on query expansion, entity annotations, and neural networks.

These Docker images have to be accompanied by a paper describing the experience of a team in participating to the challenge.

Position papers

Position papers around the issues of replicability in the IR field, with a special focus on infrastructures to enable them.

The goal of these papers is to further stimulate the discussion at the workshop and help shaping the path forward for the community.

Here, you can find full details about Submission Instructions and Important Dates.

OSIRRC 2019 Proceedings at CEUR-WS

The OSIRRC 2019 Proceedings are available online at CEUR-WS.

You can cite them as:

Clancy, R., Ferro, N., Hauff, C., Sakai, T., and Wu, Z. Z., editors (2019). Proc. of the Open-Source IR Replicability Challenge (OSIRRC 2019). CEUR Workshop Proceedings (, ISSN 1613-0073, http: //

OSIRRC 2019 Image Library

The OSIRRC 2019 Image Library repository catalogs the images that have been submitted to the OSIRRC 2019. The actual raw runs are held in a separate repository.


08:55-09:00 Welcome and Opening

09:00-09:30 OSIRRC Overview

09:30-10:30 Lightning Talks [4 minutes each]

Position Papers
Docker Papers

10:30-11:00 Coffee Break

11:00-11:45 Breakout Group Discussion

11:45-12:30 General Discussion and Closing

12:30-13:30 Lunch


Program Committee

OSIRRC 2019 Chairs