🐥 A simple parallel mapping pattern using Elixir

If you've worked with Elixir for any amount of time, you've probably appreciated its strength in handling concurrent operations. One common use case is applying a function to a list of items in parallel. That’s where a custom pmap function comes in handy.

Let’s walk through a module that makes parallel mapping simple and readable.

defmodule Parallel do
  def pmap(collection, func) do
    collection
    |> Enum.map(&(Task.async(fn -> func.(&1) end)))
    |> Enum.map(&Task.await/1)
  end
end

This defines a pmap/2 function that takes a collection (like a list) and a function, and applies the function to each item in parallel using Task.async/1 and Task.await/1.

Each task runs concurrently, which is especially useful for I/O-bound or slow computations.

Let's say we want to fetch the titles of several websites:

defmodule WebScraper do
  import Parallel

  def fetch_titles(urls) do
    Parallel.pmap(urls, fn url ->
      {:ok, response} = Req.get(url)
      Regex.run(~r/<title>(.*?)<\/title>/i, response.body, capture: :all_but_first)
      |> List.first()
    end)
  end
end

urls = [
  "https://elixir-lang.org",
  "https://hexdocs.pm",
  "https://www.google.be"
]

IO.inspect WebScraper.fetch_titles(urls)

This will fetch the HTML content from each site in parallel and extract the <title> using a regular expression.

Gotchas:

• Each Task.await/1 has a default timeout (5 seconds). You can pass a custom timeout if needed.
• CPU-bound tasks won’t benefit much unless you also use multiple processes or leverage the BEAM’s schedulers properly.
• For massive lists, you may want to limit concurrency using Task.async_stream/3 instead.

Want to make pmap even more flexible? You can expose it with a keyword-list option for timeout or max concurrency:

def pmap(collection, func, timeout \\ 5000) do
  collection
  |> Enum.map(&(Task.async(fn -> func.(&1) end)))
  |> Enum.map(&Task.await(&1, timeout))
end

inspiration