What is nested parallelism in R using future package?

edit

Nested parallelism in R using the future package refers to the ability to run multiple parallel processes within another parallel process. In other words, it allows for "parallel within parallel" computations.

For example, if we have a function that takes a large dataset and performs some calculations on it, we might want to parallelize this function to speed up the computation. However, within the function, there may be some sub-tasks that could also be parallelized. This is where nested parallelism comes in; we can use the future package to parallelize both the main function and the sub-tasks within it.

To implement nested parallelism in R using the future package, we need to first set up a parallel backend using the future package, for example:

library(future)
plan(multiprocess)

Then, within our main function, we can use the %<-% operator to assign the parallel computations to futures:

# main function
my_fun <- function(data) {
  # parallelize the outer loop
  future_loop({
    for (i in 1:nrow(data)) {
      # parallelize the inner loop
      future({
        # do some computation on one row of data
      }) %<-% value
    }
  }) %<-% value
}

In this example, we use the future_loop function to parallelize the outer loop and the future function to parallelize the inner loop. The %<-% operator assigns the computation to a future, which allows it to run in parallel. The value function retrieves the result of the future.

By nesting parallel computations in this way, we can potentially achieve even faster computation times. However, it's important to note that there are some overhead costs associated with setting up and managing parallel processes, so the benefits of nested parallelism may depend on the specific use case.