Python typically runs all your code on a single core. Even when the program that you're running has components that can easily run in parallel. To make programs faster in parallel scenarios you might want to explore ray. It's not the only tool for this use-case but it's a tool we've come to like.
You can measure the overhead of ray with a small experiment.
import time def just_sleep(t): time.sleep(0.0001) return t @ray.remote def ray_sleep(t): time.sleep(0.0001) return t
This command took 26.1s on our machine. It's quite long.
%%time results = ray.get([ray_sleep.remote(t) for t in range(100_000)])
This code only took 13.4s on our machine. Much faster!
%%time results = [just_sleep(t) for t in range(100_000)]
Feedback? See an issue? Something unclear? Feel free to mention it here.
If you want to be kept up to date, consider signing up for the newsletter.