7  Adapt and run the demo workflow

Use the working two-GEM-well demo as the starting point, then replace its inputs and settings one decision at a time. Each step follows the same loop:

  1. edit only the configuration needed for the next decision;
  2. preview and run the corresponding targets checkpoint;
  3. inspect the named outputs; and
  4. either revise the settings and rerun, or accept the result and continue.

The examples below use my_dataset and my_aggregation. Replace these with the identifiers in your configuration. Commands are run from a repository-root R session after completing Install and prepare the demo. Keep Configuration and inputs open for the complete TSV and YAML structures.

The numbered checkpoints are review boundaries, not universal acceptance criteria. Use thresholds justified for your tissue and sampling design. Each preview must return targets before you run it; an empty selection usually means the dataset or aggregation suffix is wrong. Descriptive tags select outputs; targets still builds their upstream dependencies.

Before starting, make sure crew_controllers.R describes the computer or scheduler you intend to use. See Distributed computing for that configuration.

7.1 1. Review pre-aggregation QC

7.1.1 Add GEM wells and inspect distributions

Start by adding one row per cellranger-arc count output to cfg_GEM_wells.tsv. For each new row:

  1. assign a unique GEM_well_ID and a shared GEM_well_dataset for GEM wells that should be compared during pre-aggregation QC;
  2. set the count-output directory, Cell Ranger reference, and donor fields;
  3. set GEM_well_QC_exclude_list to NA so that custom thresholds are not applied during the first inspection;
  4. add any required GEM_well_metadata_ columns; and
  5. set GEM_well_is_active to TRUE.

The committed demo rows already contain reviewed example thresholds and remain ready to run. The NA starting point applies when adding or adapting rows for new data.

NoteRequired background knowledge

The pipeline first joins GEX, ATAC, Cell Ranger, AMULET, and optional Vireo metrics for each barcode. The pre-aggregation QC checkpoint exposes the Cell Ranger-called metadata and cross-GEM-well QC distributions before GEM_well_QC_exclude_list is applied. The new GEM wells do not need to be added to an aggregation yet.

Preview the existing targets selected for this dataset, then run them:

R
targets::tar_manifest(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:1_pre-aggregation-QC]")
  ) & tidyselect::ends_with(".my_dataset"),
  callr_function = NULL
)[, c("name", "description")]

targets::tar_make(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:1_pre-aggregation-QC]")
  ) & tidyselect::ends_with(".my_dataset")
)

Use the two-GEM-well demo’s RNA count distributions and TSS enrichment distributions as examples of the generated per_dataset_QC_violins.my_dataset plot family. When your distributions need closer investigation, read their source metadata:

R
targets::tar_read(per_dataset_cellranger_kept_metadata_tibble.my_dataset)

Compare QC distributions between GEM wells and look for sample-specific tails, missing metrics, or plausible biological populations that a threshold would remove. Values such as RNA counts, mitochondrial fraction, TSS enrichment, and nucleosome signal do not have universally appropriate cutoffs.

When the distributions are understood, add complete R exclusion expressions to each row’s GEM_well_QC_exclude_list, separated by ;;:

cfg_GEM_wells.tsv
TSS.enrichment < 4 ;; nucleosome_signal > 4 ;; nCount_RNA < 250

Continue only when every active GEM well has either a justified filter or an intentional NA value.

7.1.2 Define and approve the aggregation input

Now add the metadata and aggregation that connect the accepted GEM wells:

  1. add one row per donor to a donor metadata TSV, keyed by donor_id;
  2. add an aggregation entry to cfg_aggregations.yaml;
  3. set aggregation_donor_id_metadata_tsv and list the intended GEM_well_ID values under aggregation_GEM_well_IDs; and
  4. set the aggregation’s is_active field to true.

Use the demo aggregation as the template for schema-required settings, but defer tuning marker genes and analysis parameters until their review steps.

NoteRequired background knowledge

This checkpoint applies the per-GEM-well exclusion expressions, combines the selected GEM wells, and stops before GEX dimensionality reduction. Its UpSet plots show overlapping exclusion reasons; the retained metadata shows the cells that would enter GEX.

R
targets::tar_manifest(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:1_pre-aggregation-QC]")
  ) & tidyselect::ends_with(".my_aggregation"),
  callr_function = NULL
)[, c("name", "description")]

targets::tar_make(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:1_pre-aggregation-QC]")
  ) & tidyselect::ends_with(".my_aggregation")
)

Review the two-GEM-well demo’s Cell Ranger-called exclusion-overlap plot. The corresponding all-barcode plot is an advanced diagnostic for investigating disagreement between Cell Ranger and other barcode calls.

R
cells_before_filtering <-
  targets::tar_read(per_dataset_cellranger_kept_metadata_tibble.my_dataset) |>
  dplyr::count(GEM_well_ID, name = "cells_before_filtering")

cells_after_filtering <-
  targets::tar_read(GEX_cellranger_kept_metadata_tibble.my_aggregation) |>
  dplyr::count(GEM_well_ID, name = "retained_cells")

cells_after_filtering |>
  dplyr::left_join(cells_before_filtering, by = "GEM_well_ID") |>
  dplyr::mutate(retained_fraction = retained_cells / cells_before_filtering)

Check whether one GEM well or donor loses an unexpected fraction of its cells and whether exclusion reasons overlap as intended. Revise GEM_well_QC_exclude_list and rerun this checkpoint until the retained input is credible.

7.2 2. Review GEX PCA before clustering

Configure the normalization method, variable-gene selection, PCA dimensions, and any GEX Harmony covariates before constructing the neighbour graph.

R
targets::tar_manifest(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:2_GEX-PCA-QC]")
  ) & tidyselect::ends_with(".my_aggregation"),
  callr_function = NULL
)[, c("name", "description")]

targets::tar_make(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:2_GEX-PCA-QC]")
  ) & tidyselect::ends_with(".my_aggregation")
)

Review variable-gene variance, gene loadings, the PCA singular-value elbow, embedding spread, and associations with biological and technical metadata. Check for residual batch or QC effects before choosing PCs. Harmony coordinate spread is not explained variance. Revise the settings and rerun this checkpoint before committing to the GEX neighbour graph and clustering.

7.3 3. Review GEX clusters and cell types

Review the GEX settings for the biological system before this run. In particular, configure:

  • aggregation_GEX_marker_genes;
  • the requested PCA dimensions and neighbour settings;
  • GEX Harmony variables and clustering resolution; and
  • GEX categorical and continuous variables used in review plots.

For an initial inspection of scDblFinder evidence, configure:

cfg_aggregations.yaml
aggregation_scDblFinder_GEX_remove_called_doublets: false
aggregation_scDblFinder_GEX_max_doublet_fraction_per_cluster: null
NoteRequired background knowledge

This checkpoint uses the reviewed PCA/Harmony representation to construct the neighbour graph, clusters, marker scores, cell-type annotations, and scDblFinder analysis. The accepted GEX cell set and cell types are subsequently used for ATAC peak calling.

R
targets::tar_manifest(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:3_GEX-QC]")
  ) & tidyselect::ends_with(".my_aggregation"),
  callr_function = NULL
)[, c("name", "description")]

targets::tar_make(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:3_GEX-QC]")
  ) & tidyselect::ends_with(".my_aggregation")
)

Review:

  • PCA, Harmony, metadata-association diagnostics, and the resulting GEX UMAP;
  • cluster markers, marker-module scores, and the marker dot plot;
  • categorical composition across clusters and GEM wells;
  • the GEX scDblFinder score distributions; and
  • the cell annotations in metadata_w_cell_types_tibble.GEX.my_aggregation.

Set the desired cell- and cluster-level GEX scDblFinder policy only after reviewing its scores, then rerun the same checkpoint. Continue when this is the GEX cell set and annotation that should guide peak calling.

7.4 4. Review peak-based ATAC QC

For the first peak-QC inspection, omit aggregation_QC_exclude_list_combined_object or set it to null.

NoteRequired background knowledge

Peak-based QC is evaluated after GEX because peak calling uses the accepted GEX cell set and configured grouping. This checkpoint calculates ATAC QC metrics from the resulting consensus peak matrix and fragments, but stops before the aggregation-level ATAC exclusion expressions are applied.

R
targets::tar_manifest(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:4_peak-QC]")
  ) & tidyselect::ends_with(".my_aggregation"),
  callr_function = NULL
)[, c("name", "description")]

targets::tar_make(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:4_peak-QC]")
  ) & tidyselect::ends_with(".my_aggregation")
)

Use the two-GEM-well demo’s peak-based QC distributions as the visual reference. Read your source metadata when individual distributions need closer investigation:

R
targets::tar_read(metadata_w_QC_tibble.ATAC.my_aggregation)

Compare peak counts, fraction of fragments in peaks, blacklist fraction, and the other configured peak-based metrics across GEM wells. Then add justified expressions to aggregation_QC_exclude_list_combined_object, for example:

cfg_aggregations.yaml
aggregation_QC_exclude_list_combined_object:
  - nCount_ATAC < 1000
  - atac_peak_counts_frac < 0.1
  - atac_peak_counts_blacklist_frac > 0.01

7.5 5. Review the filtered ATAC input before LSI

NoteRequired background knowledge

This checkpoint applies the configured peak-based exclusion expressions and exposes both their overlap and the retained metadata before LSI and ATAC clustering.

R
targets::tar_manifest(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:5_pre-LSI-QC]")
  ) & tidyselect::ends_with(".my_aggregation"),
  callr_function = NULL
)[, c("name", "description")]

targets::tar_make(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:5_pre-LSI-QC]")
  ) & tidyselect::ends_with(".my_aggregation")
)
R
cells_before_peak_filtering <-
  targets::tar_read(metadata_w_QC_tibble.ATAC.my_aggregation) |>
  dplyr::count(GEM_well_ID, name = "cells_before_filtering")

cells_after_peak_filtering <-
  targets::tar_read(metadata_filtered_tibble.ATAC.my_aggregation) |>
  dplyr::count(GEM_well_ID, name = "retained_cells")

cells_after_peak_filtering |>
  dplyr::left_join(cells_before_peak_filtering, by = "GEM_well_ID") |>
  dplyr::mutate(retained_fraction = retained_cells / cells_before_filtering)

Review the two-GEM-well demo’s peak-QC exclusion-overlap plot alongside the retained fractions from your run.

Confirm that the overall loss, loss per GEM well, and overlapping exclusion reasons are reasonable. Revise the aggregation-level filters and rerun this checkpoint if they are not.

7.6 6. Review ATAC

Now configure the ATAC analysis settings, including:

  • LSI dimensions and neighbours;
  • ATAC Harmony variables and clustering resolution;
  • marker transcription factors; and
  • ATAC scDblFinder removal settings.

As for GEX, an initial run with cell- and cluster-level ATAC doublet removal disabled lets the score distributions inform the final policy.

cfg_aggregations.yaml
aggregation_scDblFinder_ATAC_remove_called_doublets: false
aggregation_scDblFinder_ATAC_max_doublet_fraction_per_cluster: null
NoteRequired background knowledge

The ATAC checkpoint starts from the accepted peak-QC cell set, performs LSI, optional Harmony correction, clustering, cell typing, motif-family analysis, and configured regulatory summaries.

R
targets::tar_manifest(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:6_ATAC-QC]")
  ) & tidyselect::ends_with(".my_aggregation"),
  callr_function = NULL
)[, c("name", "description")]

targets::tar_make(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:6_ATAC-QC]")
  ) & tidyselect::ends_with(".my_aggregation")
)

Review LSI diagnostics, metadata associations, the two-GEM-well demo’s ATAC UMAP, cluster stability, and ATAC scDblFinder score distributions. Also review motifs, gene activity, and coverage or differential-accessibility outputs where configured. Continue when the ATAC result is credible independently of the RNA result.

7.7 7. Review the multimodal result

Finally, review the WNN neighbour, resolution, and UMAP settings.

NoteRequired background knowledge

The multimodal checkpoint combines the accepted GEX and ATAC representations using WNN, then produces integrated clusters, metadata, review plots, and the optional Seurat/Signac compatibility object.

R
targets::tar_manifest(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:7_multimodal-QC]")
  ) & tidyselect::ends_with(".my_aggregation"),
  callr_function = NULL
)[, c("name", "description")]

targets::tar_make(
  names = targets::tar_described_as(
    tidyselect::contains("checkpoint:7_multimodal-QC]")
  ) & tidyselect::ends_with(".my_aggregation")
)

Review the two-GEM-well demo’s integrated WNN UMAP and RNA/ATAC modality weights, then compare your WNN cell types and clusters against the accepted single-modality results. The final compatibility object is multimodal_Seurat_object.my_aggregation.

Use Verify and inspect the outputs for tar_read() and output-path examples. Continue to differential analyses or genetic enrichment only after accepting the main aggregation.

7.8 Request an additional result

Each gallery card names its target. To request only that output and its dependencies, use its exact name with your aggregation suffix:

R
targets::tar_make(
  names = tidyselect::all_of("categorical.UMAPs.WNN.my_aggregation")
)

Use multimodal_Seurat_object.my_aggregation for the final compatibility object. That endpoint does not include every review plot. Preview an exact selection with tar_manifest() and callr_function = NULL first; all_of() reports an error when the name is absent.

7.9 Rerun after a change

Reuse the same selection after changing inputs or settings. Replace tar_manifest() with tar_outdated() to inspect which selected targets and their dependencies need rebuilding, keeping callr_function = NULL. Then rerun the checkpoint and review its outputs again.

7.10 Build the complete active scope

An unqualified targets::tar_make() constructs every active GEM well, active aggregation, derived review output, and enabled optional module. Use it only after the checkpoint-sized runs are accepted and only when that complete scope is intended:

R
targets::tar_make()

Keep unavailable GEM wells, aggregations, and modules inactive before this broad execution. If a target fails, use Troubleshooting and rerun the narrowest affected checkpoint.