Pyserini Reproductions (original) (raw)

The main results table provides commands for reproducing runs using the following models:

The symbol shown in parentheses (e.g., bm25-flat) refers the parameter used in the --condition option for programmatic execution (see below). In addition, we provide the following models whose runs can be reproduced, but are not shown in the results table:

CQADupStack Results

The "cqadupstack (average)" row above represents the average across the individual CQADupStack datasets, shown below.

References

• [1] Ehsan Kamalloo, Nandan Thakur, Carlos Lassance, Xueguang Ma, Jheng-Hong Yang, and Jimmy Lin.Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard. arXiv:2306.07471, June 2023.
• [2] Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant.From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective. Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2353–2359.
• [3] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave.Towards Unsupervised Dense Information Retrieval with Contrastive Learning. arXiv:2112.09118, December 2021.
• [4] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff.C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597, December 2023.

Programmatic Execution

All experimental runs shown in the above table can be programmatically executed based on the instructions below. To list all the experimental conditions:

python -m pyserini.2cr.beir --list-conditions

These conditions correspond to the table rows above.

For all conditions, just show the commands in a "dry run":

python -m pyserini.2cr.beir --all --display-commands --dry-run

To actually run all the experimental conditions:

python -m pyserini.2cr.beir --all --display-commands

With the above command, run files will be placed in the current directory. Use the option --directory runs/ to place the runs in a sub-directory.

To show the commands for a specific condition:

python -m pyserini.2cr.beir --condition bm25-flat --display-commands --dry-run

This will generate exactly the commands for a specific condition above (corresponding to a row in the table).

To actually run a specific condition:

python -m pyserini.2cr.beir --condition bm25-flat --display-commands

Again, with the above command, run files will be placed in the current directory. Use the option --directory runs/ to place the runs in a sub-directory.

Finally, to generate this page:

python -m pyserini.2cr.beir --generate-report --output beir.html

The output file beir.html should be identical to this page.