Using RedisParam (original) (raw)

Contents

• 1 Getting started
• 2 Use
  • 2.1 Manager and workers from a single R session
  • 2.2 Independently-managed workers
• Session info

Getting started

RedisParam implements a BiocParallel backend using redis, rather than sockets, for communication. It requires a redis server; see?RedisParam for host and port specification. redis is a good solution for cloud-based environments using a standard docker image. A particular feature is that the number of workers can be scaled during computation, for instance in response to kubernetes auto-scaling.

Use

Ensure that a redis server is running, e.g., from the command line

$ redis-server

Manager and workers from a single R session

On a single computer, in R, load and use the RedisParam package in the same way as other BiocParallel backends, e.g.,

library(RedisParam)
p <- RedisParam(workers = 5)
result <- bplapply(1:7, function(i) Sys.getpid(), BPPARAM = p)
table(unlist(result))

Independently-managed workers

For independently managed workers, start workers in separate processes, e.g.,

Sys.getpid()
library(RedisParam)
p <- RedisParam(jobname = "demo", is.worker = TRUE)
bpstart(p)

Start and use the manager in a separate process. Be sure to use the same jobname =.

Sys.getpid()            # e.g., 8563
library(RedisParam)
p <- RedisParam(jobname = 'demo', is.worker = FALSE)
result <- bplapply(1:7, function(i) Sys.getpid(), BPPARAM = p)
unique(unlist(result)) # e.g., 9677

Independently started workers can be terminated from the manager

rpstopall(p)

Session info

This version of the vignette was built on 2025-04-15 with the following software package versions: