14  Choose where the analysis runs

A worker is an R process that runs an analysis task. A controller starts and manages those workers, either on your machine or through a cluster scheduler. multiomeR uses crew for this. Choose the setup below before the first demo run.

Use local workers on a suitable workstation. On a shared cluster, ask your support team which scheduler, account, and resource limits to use. The targets distributed-computing guide explains the general setup; this page covers multiomeR’s configuration file.

14.1 Local execution

A fresh clone includes a local crew_controllers.R sized for a 16-CPU, 256-GB workstation, with four light workers and two heavy workers. A machine near the 60-GB minimum should reduce concurrency to one heavy worker and should not run several memory-intensive targets simultaneously. Edit the worker counts and resource tiers directly in crew_controllers.R, keeping controller names identical between controller_list and controller_resources_tibble.

After changing the file, restart R or reload the project runtime explicitly:

R
load_project_runtime(force = TRUE)

Rebuild a narrow manifest selection before starting the data run to validate the controller contract.

14.2 Scheduler execution

For SLURM, PBS, SGE, or LSF, replace the local controllers with the corresponding crew.cluster controllers. The commented SLURM section in crew_controllers.R shows the expected shape.

For every scheduler tier:

  1. Match the controller name in both the controller object and resource table.
  2. Align scheduler CPU and memory requests with the capacity declared in the table.
  3. Set queue, account, wall-time, module, and worker-startup options required by the cluster.
  4. Keep GPU tiers separate; GPU controllers are considered only for targets requesting GPUs.
  5. Test a small target selection before increasing worker counts.

Scheduler startup failures, resource-routing errors, and target failures are handled separately in Troubleshooting. Developer-facing details about runtime bootstrap and get_tar_resources() are in Implementation conventions.

14.3 Controller contract

crew_controllers.R is sourced during load_project_runtime() and must return a named list containing these components:

  • controller_list: a non-empty list of crew controllers with unique controller names.
  • controller_resources_tibble: a data frame with exactly controller_name, cores, RAM_GB, and gpus, in that order.

The resource columns must be numeric, non-missing, and contain one unique row per controller name represented in controller_list. The first resource-table row is the default controller. For explicit requests, get_tar_resources() selects the first compatible row after applying the requested CPU, RAM, and GPU constraints.

controller_resources_tibble <- tibble::tribble(
  ~controller_name, ~cores, ~RAM_GB, ~gpus,
  "local-light",        1,      16,     0,
  "local-heavy",        6,      60,     0
)

The table describes controller capacity for routing. A local controller does not create physical memory: its workers value must be low enough that concurrent jobs cannot exhaust the machine.