Reproducible Research in Pattern Recognition (RRPR) Badge

Introduction

Reproducibility is a cornerstone of scientific research. The ability of independent researchers to verify experimental results, understand research methods, and build upon previous work is essential for the continued advancement of pattern recognition.

ICPRv 2027 encourages authors to adopt open and reproducible research practices by making the research artifacts associated with their work available whenever possible. Such artifacts may include source code, datasets, trained models, software environments, configuration files, documentation, and other materials needed to reproduce the reported results.

To recognize outstanding efforts towards transparent and reproducible research, ICPRv 2027 will award the Reproducible Research in Pattern Recognition (RRPR) Badge to accepted papers that successfully complete a separate reproducibility evaluation coordinated by the Reproducibility Chairs.

The RRPR Badge is intended to recognize and promote research that can be independently understood, verified, and built upon by the scientific community.

Purpose of the RRPR Badge

The RRPR Badge recognizes authors who make a significant effort to ensure that the experimental results presented in their paper can be independently reproduced.

Receiving the badge demonstrates that the authors have provided sufficient information and research artifacts to enable other researchers, whenever reasonably possible, to reproduce the principal experimental findings described in the paper.

The RRPR Badge is intended to promote transparency, facilitate scientific progress, and encourage the adoption of reproducible research practices throughout the pattern recognition community.

Eligibility

All accepted Long Papers and Short Papers are eligible to be considered for the RRPR Badge.

Authors wishing to participate will be able to indicate their interest during paper submission through the conference submission system.

Participation in the RRPR evaluation is entirely voluntary.

Research Artifacts

Authors seeking the RRPR Badge are encouraged to provide, where applicable:

  • source code;
  • trained models or model checkpoints;
  • datasets or clear instructions for obtaining them;
  • data preprocessing scripts;
  • training and evaluation scripts;
  • configuration files and hyperparameter settings;
  • software dependency information;
  • execution instructions;
  • documentation describing the experimental workflow; and
  • any additional materials required to reproduce the principal experimental results.

Not every item is applicable to every paper. The objective is to provide sufficient information to enable an independent researcher with appropriate expertise to reproduce the main experiments and conclusions presented in the paper.

Authors are encouraged to use publicly accessible repositories whenever possible. If the paper is accepted, research artifacts should be publicly available no later than the camera-ready deadline, unless legal, ethical, licensing, privacy, or commercial restrictions prevent public release.

Restricted Resources

Some research artifacts cannot be shared publicly because of licensing, privacy, ethical, legal, security, or commercial constraints.

Such restrictions do not automatically preclude a paper from receiving the RRPR Badge.

In these cases, authors should provide the maximum level of transparency reasonably possible. This may include detailed documentation of the data acquisition process, preprocessing pipeline, model architecture, evaluation protocol, software environment, and any procedures through which qualified researchers may obtain access to the restricted resources where permitted.

Where exact datasets cannot be redistributed, authors are encouraged to provide representative examples, synthetic substitutes, or clear instructions that enable researchers to perform equivalent validation experiments whenever appropriate.

Evaluation Criteria

The RRPR evaluation focuses on the reproducibility of the reported research rather than the scientific novelty or significance of the paper.

RRPR reviewers will evaluate aspects including:

  • completeness of the provided research artifacts;
  • clarity and completeness of the accompanying documentation;
  • consistency between the paper and the released materials;
  • ease of reproducing the reported experiments; and
  • the extent to which the provided materials support the principal claims of the paper.

Perfect numerical replication is not always possible because of differences in hardware, software versions, random initialization, or stochastic optimization. The objective of the evaluation is to determine whether an independent researcher can reasonably reproduce the reported methodology and obtain results that support the same scientific conclusions.

Review Process

The RRPR evaluation is conducted independently of the main scientific review.

Only papers that have been accepted by the Program Committee will be considered for the RRPR Badge. The reproducibility evaluation therefore has no influence on the paper acceptance decision.

The evaluation will be carried out by specially designated RRPR reviewers under the supervision of the Reproducibility Chairs. Based on the reviewers' reports, the Reproducibility Chairs will make the final recommendation regarding the award of the RRPR Badge.

Authors may be contacted during the evaluation process to clarify technical issues or provide minor corrections to the submitted reproducibility materials.

RRPR Badge

Papers that successfully complete the reproducibility evaluation will receive the ICPRv 2027 Reproducible Research in Pattern Recognition (RRPR) Badge.

The badge will appear in the published version of the paper and will indicate that the associated research artifacts have undergone a positive reproducibility evaluation by the conference.

The RRPR Badge recognizes the authors' contribution to transparent, trustworthy, and reusable scientific research. It is intended to increase the visibility of reproducible research and encourage broader adoption of open scientific practices within the pattern recognition community.

Preserving Anonymity

ICPRv 2027 uses a double-blind review process. To preserve author anonymity, the evaluation for the Reproducible Research in Pattern Recognition (RRPR) Badge is conducted only after the paper has been accepted.

Authors who wish to be considered for the RRPR Badge simply indicate their interest during paper submission. No research artifacts are submitted or evaluated as part of the scientific review process.

Following the acceptance notifications, authors of accepted papers who opted to participate will be invited to submit the research artifacts required for the RRPR evaluation. These artifacts may include source code, datasets, trained models, software environments, documentation, and other materials needed to reproduce the reported results.

This process allows authors to provide complete reproducibility materials without the need to anonymize repositories or other research artifacts, while ensuring that the double-blind review process remains uncompromised. The RRPR evaluation is conducted independently of the scientific review and has no influence on the paper acceptance decision.

Questions

Questions concerning the RRPR Badge or the reproducibility evaluation process should be directed to the Reproducibility Chairs at icpr27pc@iapr.org