Manifests and safe downloads with nemoR

Introduction

The central reproducibility object in nemoR is a manifest. A manifest is a table that records which NeMO files were selected, where they can be downloaded, where they should live locally, and what happened during download. This vignette demonstrates the manifest and safety layers without downloading large external data.

nemoR complements Bioconductor workflows by making the data-access step explicit before data are loaded into objects such as SingleCellExperiment. The loading helpers are deliberately optional because different NeMO file formats require different Bioconductor or CRAN readers.

Installation

install.packages("nemoR")

library(nemoR)

Create a small manifest

The manifest below uses example URLs, so it is suitable for documentation and tests. A real NeMO manifest can be created with nemo_search_manifest() or nemo_fetch(dry_run = TRUE).

manifest <- nemo_manifest_from_urls(
  urls = c(
    "https://example.com?file=processed_counts.h5ad",
    "https://example.com?file=parameters.json"
  ),
  collection_id = "example-collection",
  file_id = c("example-counts", "example-parameters")
)

manifest$data_type <- c("counts", "parameters")
manifest$size <- c(5 * 1024^2, 2048)
manifest
#> # A tibble: 2 × 15
#>   collection_id      file_id  file_name file_format   size checksum download_url
#>   <chr>              <chr>    <chr>     <chr>        <dbl> <chr>    <chr>       
#> 1 example-collection example… example.… h5ad        5.24e6 <NA>     https://exa…
#> 2 example-collection example… example.… json        2.05e3 <NA>     https://exa…
#> # ℹ 8 more variables: local_path <chr>, download_status <chr>,
#> #   checksum_verified <lgl>, nemo_api_source <chr>,
#> #   manifest_schema_version <chr>, query_parameters <json>, retrieved_at <chr>,
#> #   data_type <chr>

Validate and summarize

Validation checks that the minimum columns required by the download and loading workflow are present.

nemo_validate_manifest(manifest)
nemo_download_plan(manifest, max_size_gb = 1)
#> # A tibble: 1 × 11
#>   n_files total_size_gb largest_file_gb unknown_size_files max_size_gb
#>     <int>         <dbl>           <dbl>              <int>       <dbl>
#> 1       2         0.005           0.005                  0           1
#> # ℹ 6 more variables: within_size_limit <lgl>, file_formats <chr>,
#> #   data_types <chr>, access <chr>, download_statuses <chr>,
#> #   files_with_checksum <int>

Understand download status columns

nemo_download() updates local_path, download_status, and checksum_verified. These fields make the manifest useful as a record of what happened, not only what was requested.

manifest[, c("file_id", "file_name", "download_status", "checksum_verified")]
#> # A tibble: 2 × 4
#>   file_id            file_name                 download_status checksum_verified
#>   <chr>              <chr>                     <chr>           <lgl>            
#> 1 example-counts     example.com?file=process… not_downloaded  NA               
#> 2 example-parameters example.com?file=paramet… not_downloaded  NA

Check a local checksum

When NeMO provides an MD5 checksum, nemo_download(verify_checksum = TRUE) can compare it with the downloaded file. The small local example below shows the MD5 value that would be stored in a manifest without downloading external data.

tmp <- tempfile()
writeLines("nemo", tmp)

checksum <- unname(tools::md5sum(tmp))
checksum
#> [1] "8a9f8013767d4fe955553bc97d18201d"
identical(checksum, unname(tools::md5sum(tmp)))
#> [1] TRUE

Download real files

Actual downloads are not run in this vignette because NeMO files can be large and depend on external archive availability. The code below is the intended interactive pattern after the manifest and size plan have been inspected.

downloaded <- nemo_download(
  manifest,
  destdir = "nemo_downloads",
  max_size_gb = 5,
  verify_checksum = TRUE
)

nemo_write_manifest(downloaded, file.path("nemo_downloads", "nemo_manifest.tsv"))

Load downloaded data

Loading requires local files and optional reader packages. The first-pass helpers support H5AD and 10x HDF5 files for SingleCellExperiment or Seurat workflows.

downloaded <- nemo_read_manifest("nemo_downloads/nemo_manifest.tsv")
sce <- nemo_load(downloaded, format = "SingleCellExperiment")
seurat_obj <- nemo_load(downloaded, format = "Seurat")

Session information

sessionInfo()
#> R version 4.6.0 (2026-04-24)
#> Platform: aarch64-apple-darwin23
#> Running under: macOS Tahoe 26.5.1
#> 
#> Matrix products: default
#> BLAS:   /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRblas.0.dylib 
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.1
#> 
#> locale:
#> [1] C/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
#> 
#> time zone: America/New_York
#> tzcode source: internal
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#> [1] nemoR_0.99.3
#> 
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.39   utf8_1.2.6      R6_2.6.1        fastmap_1.2.0  
#>  [5] xfun_0.58       magrittr_2.0.5  glue_1.8.1      cachem_1.1.0   
#>  [9] tibble_3.3.1    knitr_1.51      pkgconfig_2.0.3 htmltools_0.5.9
#> [13] rmarkdown_2.31  lifecycle_1.0.5 cli_3.6.6       vctrs_0.7.3    
#> [17] sass_0.4.10     jquerylib_0.1.4 compiler_4.6.0  tools_4.6.0    
#> [21] pillar_1.11.1   evaluate_1.0.5  bslib_0.11.0    yaml_2.3.12    
#> [25] otel_0.2.0      rlang_1.2.0     jsonlite_2.0.0