doi:10.48550/arXiv.2410.08643> can be used to answer these questions, by fixing a test subset, training models on Same/Other/All subsets, and then comparing test error rates (Same versus Other and Same versus All). Also provides code for estimating how many train samples are required to get accurate predictions on a test set.">

mlr3resampling: Resampling Algorithms for 'mlr3' Framework (original) (raw)

A supervised learning algorithm inputs a train set, and outputs a prediction function, which can be used on a test set. If each data point belongs to a subset (such as geographic region, year, etc), then how do we know if subsets are similar enough so that we can get accurate predictions on one subset, after training on Other subsets? And how do we know if training on All subsets would improve prediction accuracy, relative to training on the Same subset? SOAK, Same/Other/All K-fold cross-validation, <doi:10.48550/arXiv.2410.08643> can be used to answer these questions, by fixing a test subset, training models on Same/Other/All subsets, and then comparing test error rates (Same versus Other and Same versus All). Also provides code for estimating how many train samples are required to get accurate predictions on a test set.

Version: 2025.6.23
Imports: data.table, R6, checkmate, paradox, mlr3 (≥ 0.21.1), mlr3misc, batchtools, filelock
Suggests: ggplot2, animint2, mlr3tuning, lgr, future, testthat, knitr, markdown, nc, rpart, directlabels, mlr3torch, torch
Published: 2025-06-23
DOI: 10.32614/CRAN.package.mlr3resampling
Author: Toby Hocking ORCID iD [aut, cre], Michel Lang ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Bernd Bischl ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Jakob Richter ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Patrick Schratz ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Giuseppe CasalicchioORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Stefan Coors ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Quay Au ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Martin Binder [ctb], Florian Pfisterer ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Raphael Sonabend ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Lennart Schneider ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Marc Becker ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified), Sebastian Fischer ORCID iD [ctb] (Author of mlr3 when Resampling/ResamplingCV was copied/modified)
Maintainer: Toby Hocking <toby.hocking at r-project.org>
BugReports: https://github.com/tdhock/mlr3resampling/issues
License: LGPL-3
URL: https://github.com/tdhock/mlr3resampling
NeedsCompilation: no
Materials:
CRAN checks: mlr3resampling results

Documentation:

Downloads:

Linking:

Please use the canonical formhttps://CRAN.R-project.org/package=mlr3resamplingto link to this page.