Troubleshooting

Find the target name and first error message in the run output. Fix that cause, then rerun the same selection.

What happened?	Start here
Setup fails before any data processing	The manifest does not build
A run ended and you need to know whether it built everything	Check that a run finished
A target reports an error	A run reports errored targets
You need to check a result that a failed target used	Inspect one upstream value
An error names an input file, GEM well, donor, or metadata column	Input and metadata failures
Workers do not start or are killed	Controller and scheduler failures
A completed result needs rebuilding	A target is unexpectedly outdated

Run the commands below in the repository-root R session. For general techniques, see the targets debugging guide.

The manifest does not build

Reload the runtime and build the manifest:

R

load_project_runtime()
targets::tar_manifest(callr_function = NULL)

Failures at this stage usually mean:

• configuration.local names a missing directory, or a cfg_* file is missing from the selected configuration directory;
• a YAML parameter is unknown, misspelled, of the wrong type, or missing;
• an aggregation selects a GEM well that is absent from cfg_GEM_wells.tsv, or an active aggregation selects an inactive GEM well;
• an aggregation enables an unknown module, or the module’s configuration has no entry for it;
• crew_controllers.R breaks the controller rules; or
• an R package fails to load.

Correct the field or file named in the error, then repeat the manifest check.

Check that a run finished

The pipeline keeps building after an error: a failed target skips only the targets that depend on it. tar_make() then returns without an R error, so a script that runs it exits with status 0. Check the targets you selected right after the run:

R

selected <- targets::tar_manifest(
  names = tidyselect::ends_with("8_multimodal_QC.my_aggregation"),
  fields = name
)$name
list_unfinished_targets(selected)

It lists each selected target that has a stored error or that the run did not complete. A target without progress was never reached, for example because the run stopped early. The next run resets the progress this check reads.

A run reports errored targets

List the errors of the current pipeline, one target per distinct error message:

R

list_distinct_errored_targets()

To add commands and stored tracebacks, filter errored targets by a regular expression that matches a target, aggregation, or module:

R

list_distinct_errored_targets_w_tracebacks(
  target_name_pattern = "my_aggregation"
)

Some targets run separately for multiple groups; these runs are called branches, and their names end in a hash suffix. Copy the full target or branch name from the listing to inspect its saved workspace:

R

inspect_target_workspace("categorical.UMAPs.8_multimodal_QC.my_aggregation")

The inspection reports the target command, stored error and traceback, and short summaries of its dependencies. Empty tables, zero-dimensional matrices, missing model columns, or unexpected labels usually point to an upstream data or configuration problem.

Inspect one upstream value

Read a dependency by its full name and print only a small part of it:

R

value <- targets::tar_read(metadata_w_cell_types_tibble.WNN.my_aggregation)
dim(value)
head(value)

Input and metadata failures

The configuration files are checked when the pipeline starts, the donor metadata table and the Cell Ranger outputs when the targets that read them run. Compare the error with the rules checked for the GEM well table, the donor metadata table and the aggregation configuration, and with the Cell Ranger inputs each GEM well needs. Nuclei with NA donor variables have a donor ID missing from the donor metadata table; see Matching donors to nuclei.

Controller and scheduler failures

If workers do not start, are killed, or no controller can run a target:

1. Check crew_controllers.R against the controller rules, then reload it with load_project_runtime().
2. An error starting with No controller found means that no tier offers the requested cores, RAM, or GPUs; add or enlarge a tier.
3. For scheduler controllers, read the scheduler’s output and error logs, and check the queue, account, wall time, memory, CPU, module, and file-system settings.
4. If local workers are killed for lack of memory, lower their workers values as described in Local execution.
5. A target whose worker is killed is retried up to its controller’s crashes_max times. One more crash stops the whole run with the crew worker of task ... crashed. That target stores no error, and targets that had not started are not built. Give the target more memory, then rerun the same selection.

A target is unexpectedly outdated

Ask targets which part of a narrow selection needs rebuilding:

R

targets::tar_outdated(
  names = tidyselect::ends_with("8_multimodal_QC.my_aggregation"),
  callr_function = NULL
)

Changes to code, configuration, input files, or any upstream target make downstream targets outdated. Look for the most upstream name in the result before assuming that the final target itself is the cause.

Results change slightly after a rebuild

Neighbor searches build their index on one thread, so rebuilding a target from unchanged inputs gives the same neighbors, clusters and UMAP layouts. A small upstream change can still move results: removing a few nuclei, for example, redraws the random subsets in which WNN modality weights are estimated, and clusters, UMAP coordinates and downstream statistics shift with them. This is expected; compare cluster composition and marker results rather than exact labels, and keep the store when results must stay unchanged.

Old plots remain in the store

Plots of removed targets, of aggregations no longer configured and of branches that an upstream change renamed stay in <store>/plots/. List the images that no current target owns, check the list, and only then delete them:

R

orphaned <- prune_orphaned_plots()
prune_orphaned_plots(delete = TRUE)

Pass every target script that writes plots to the store as scripts, since images owned by any of their graphs are kept. An aggregation removed from the configuration counts as gone, so its plots are listed too.

Rerun safely

After fixing the cause, rerun the same selection; up-to-date results are reused:

R

targets::tar_make(
  names = tidyselect::ends_with("8_multimodal_QC.my_aggregation")
)

Run targets::tar_make() without names only when you want every active aggregation and enabled module built. Do not delete the store to debug: it holds the error records, and deleting it forces unrelated recomputation.