Let's pretend that we have a function that makes a request to an endpoint.
import random
from requests.exceptions import HTTPError
def pretend_request():
if random.random() < 0.01:
raise HTTPError("Something went wrong!")
return {"status": "alive"}
This function is fake, but it similates something that tends
to happen all the time. It simulates a HTTPError
one percent
of the time.
The error won't happen that often, but it will happen once in a while. If you're running a batch script that communicates with many services you may get unlucky and the job could break.
Just try running this for-loop.
# You'll likely hit an error
for i in range(500):
pretend_request()
Enter retry
This is where the retry library might help. It's a library with a single decorator that can tell the function to try again after failing. Here's how you can use it:
from retry import retry
@retry(tries=3, delay=0.1)
def pretend_request():
if random.random() < 0.01:
raise HTTPError("Something went wrong!")
return {"status": "alive"}
With these settings, the retry
function will retry three times
and between each retry it will wait for 0.1 seconds. If you were
now to run the for-loop, you shouldn't see any errors anymore.
# You'll likely hit an error
for i in range(500):
pretend_request()
Logging
The @retry
decorator will also appear in your logs.
import random
import logging
from requests.exceptions import HTTPError
from retry import retry
logging.basicConfig()
@retry(HTTPError, tries=10, delay=0.1, backoff=2)
def pretend_request():
if random.random() < 0.1:
raise rq.exceptions.HTTPError("Something went wrong!")
return {"status": "alive"}
for i in range(100):
pretend_request()
If you pay close attention to the logs, you can see that
the backoff
setting will double the delay time on every hit.
That's why you might see a few logs that look like:
WARNING:retry.api:Something went wrong!, retrying in 0.1 seconds...
WARNING:retry.api:Something went wrong!, retrying in 0.2 seconds...
WARNING:retry.api:Something went wrong!, retrying in 0.4 seconds...
Configuration
You could also configure the @retry
decorator to be more specific.
import random
from retry import retry
from requests.exceptions import HTTPError
@retry(HTTPError, tries=3, delay=0.1)
def pretend_request():
if random.random() < 0.01:
raise HTTPError("Something went wrong!")
return {"status": "alive"}
Now, the function will only retry if detects a HTTPError
.
If you were to configure it to only respond to ValueError
and
TypeError
then any other errors will be ignored.
@retry((ValueError, TypeError), tries=3, delay=0.1)
def pretend_request():
if random.random() < 0.01:
raise HTTPError("Something went wrong!")
return {"status": "alive"}
# You'll likely hit an error again
for i in range(500):
pretend_request()
Back to main.