---
title: "4. Using blockCV with caret"
author: "Roozbeh Valavi"
date: "`r Sys.Date()`"
output: 
  rmarkdown::html_vignette:
    fig_caption: yes
vignette: >
  %\VignetteIndexEntry{4. Using blockCV with caret}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

## Introduction

This tutorial shows how to use spatial cross-validation folds generated by
`blockCV` in the `caret` modelling framework. The key step is to pass the
training and testing row indices stored in `folds_list` to
`caret::trainControl()` through its `index` and `indexOut` arguments.


```{r echo=FALSE}
options(scipen = 10)
```

## Prepare the data

Load the simulated presence-absence data and environmental raster covariates
included with `blockCV`.

```{r message=FALSE, warning=FALSE}
library(blockCV)
library(sf)
library(terra)

# import presence-absence species data
points <- read.csv(system.file("extdata/", "species.csv", package = "blockCV"))

# make an sf object from the data.frame
pa_data <- sf::st_as_sf(points, coords = c("x", "y"), crs = 7845)

# load raster covariates
covars <- terra::rast(
  list.files(system.file("extdata/au/", package = "blockCV"), full.names = TRUE)
)
```

Extract the raster covariate values at the species records. This is the
modelling table that will be supplied to `caret`.

```{r}
training <- terra::extract(covars, pa_data, ID = FALSE)
training$occ <- as.factor(pa_data$occ)

head(training)
```

## Create spatial folds

Create spatial blocks with `cv_spatial()`. Each item in `folds_list` contains
two integer vectors: the first is the training row indices and the second is the
held-out testing row indices.

```{r fig.height=5, fig.width=7, message=FALSE, warning=FALSE}
set.seed(123)

sb1 <- cv_spatial(
  x = pa_data,
  column = "occ",
  r = covars,
  size = 450000,
  k = 5,
  selection = "random",
  iteration = 50,
  progress = FALSE,
  report = TRUE,
  plot = TRUE
)
```

## Fit a caret model

The only conversion needed for `caret` is to separate the training and testing
indices from `folds_list`. The following example uses a random forest model
through `caret::train()`. The modelling chunk is not evaluated when the vignette
is built because `caret` and `randomForest` are optional modelling packages.

```{r eval=FALSE}
library(caret)

train_index <- lapply(sb1$folds_list, function(fold) fold[[1]])
test_index <- lapply(sb1$folds_list, function(fold) fold[[2]])

names(train_index) <- paste0("Fold", seq_along(train_index))
names(test_index) <- names(train_index)

control <- trainControl(
  method = "cv",
  number = length(train_index),
  index = train_index,
  indexOut = test_index,
  search = "random"
)

set.seed(123)

rf_model <- train(
  occ ~ .,
  data = training,
  method = "rf",
  trControl = control,
  tuneLength = 4,
  ntree = 500
)

rf_model
rf_model$resample
plot(rf_model)
```

The resampling results are now based on the spatial folds from `blockCV`,
rather than on random non-spatial folds generated internally by `caret`.

The [`CAST`](https://hannameyer.github.io/CAST/reference/CreateSpacetimeFolds.html)
package uses the same `caret` interface. Its `CreateSpacetimeFolds()` function
creates train and test lists from pre-defined spatial, temporal, or
spatio-temporal groups that can be passed directly to `trainControl(index = ...,
indexOut = ...)`.

Please cite `blockCV` by: *Valavi R, Elith J, Lahoz-Monfort JJ, Guillera-Arroita G. blockCV: An R package for generating spatially or environmentally separated folds for k-fold cross-validation of species distribution models. Methods Ecol Evol. 2019; 10:225-232.* [doi: 10.1111/2041-210X.13107](https://doi.org/10.1111/2041-210X.13107)

## References

- Meyer H, Reudenbach C, Hengl T, Katurji M, Nauss T. 2018. Improving performance of spatio-temporal machine learning models using forward feature selection and target-oriented validation. *Environmental Modelling & Software* 101: 1-9.

- Kuhn M. 2008. Building predictive models in R using the caret package. *Journal of Statistical Software* 28: 1-26.

- Valavi R, Elith J, Lahoz-Monfort JJ, Guillera-Arroita G. **blockCV: An R package for generating spatially or environmentally separated folds for k-fold cross-validation of species distribution models**. *Methods Ecol Evol.* 2019; 10:225-232. [doi: 10.1111/2041-210X.13107](https://doi.org/10.1111/2041-210X.13107)