GitHub - pltrdy/files2rouge: Calculating ROUGE score between two files (line-by-line) (original) (raw)

Files2ROUGE

Motivations

Given two files with the same number of lines, files2rouge calculates the average ROUGE scores of each sequence (=line). Each sequence may contain multiple sentences. In this case, the end of sentence string must be passed using the --eos flag (default: "."). Running files2rouge with a wrong eos delimiter may lead to incorrect ROUGE-L score.

You may also be interested in a Python implementation (instead of a wrapper): https://github.com/pltrdy/rouge.

$ files2rouge --help usage: files2rouge [-h] [-v] [-a ARGS] [-s SAVETO] [-e EOS] [-m] [-i] reference summary

Calculating ROUGE score between two files (line-by-line)

positional arguments: reference Path of references file summary Path of summary file

optional arguments: -h, --help show this help message and exit -v, --verbose Prints ROUGE logs -a ARGS, --args ARGS ROUGE Arguments -s SAVETO, --saveto SAVETO File to save scores -e EOS, --eos EOS End of sentence separator (for multisentence). Default: "." -m, --stemming DEPRECATED: stemming is now default behavior -nm, --no_stemming Switch off stemming -i, --ignore_empty

Getting Started

0) Install prerequisites

pip install -U git+https://github.com/pltrdy/pyrouge

(NOTE: running pip install pyrouge would not work as the package is out of date on PyPI)

1) Clone the repo, setup the module and ROUGE

git clone https://github.com/pltrdy/files2rouge.git
cd files2rouge python setup_rouge.py python setup.py install

Do not forget to run setup_rouge

2) Run files2rouge.py

files2rouge references.txt summaries.txt

**Outputs:**When --verbose is set, the script prints progress and remaining time on stderr. This can be changed using --verbose in order to outputs ROUGE execution logs.

Default output example:

Preparing documents...
Running ROUGE...
---------------------------------------------
1 ROUGE-1 Average_R: 0.28242 (95%-conf.int. 0.25721 - 0.30877)
1 ROUGE-1 Average_P: 0.30157 (95%-conf.int. 0.27114 - 0.33506)
1 ROUGE-1 Average_F: 0.28196 (95%-conf.int. 0.25704 - 0.30722)
---------------------------------------------
1 ROUGE-2 Average_R: 0.10395 (95%-conf.int. 0.08298 - 0.12600)
1 ROUGE-2 Average_P: 0.11458 (95%-conf.int. 0.08873 - 0.14023)
1 ROUGE-2 Average_F: 0.10489 (95%-conf.int. 0.08303 - 0.12741)
---------------------------------------------
1 ROUGE-L Average_R: 0.25231 (95%-conf.int. 0.22709 - 0.27771)
1 ROUGE-L Average_P: 0.26830 (95%-conf.int. 0.23834 - 0.29818)
1 ROUGE-L Average_F: 0.25142 (95%-conf.int. 0.22741 - 0.27533)

Elapsed time: 0.458 secondes

Call files2rouge from Python

import files2rouge
files2rouge.run(hyp_path, ref_path)

ROUGE Args

One can specify which ROUGE args to use using the flag --args (or -a).
The default behavior is equivalent to:

files2rouge reference.txt summary.txt -a "-c 95 -r 1000 -n 2 -a" # be sure to write args betwen double-quotes

You can find more informations about these arguments here

Known issues

More informations