All you need is closure

Decorators are well known examples of Python syntactic sugar, which basically let you change the behaviour of a function without modifying its definition, thanks to a handy syntax based on the @ symbol followed by the decorator name, everything put just above your function definition.

Sometimes, however, you can't exploit this syntax, for example because you don't have the function definition at your disposal - let's think at the case in which the function you want to decorate lives in another codebase (the latter might be one of the libraries your project depends on).

The syntactic sugar approach, generally speaking, gives a lot of advantages in terms of experimentation, reusability and reversibility. However, in cases like the ones mentioned above, this approach can do more harm than good, forcing you to wrap a function invocation in another function with the only aim of providing a function definition to decorate, with the risk of reduce the readability and maintainability of your codebase¹.

What you can do instead is take a step back and use the decorator function directly for what it is: a closure².

ScenarioClosure approachSyntactic sugar approach

from external_lib import some_func

def main():
    # ... complex logic that you can't refactor
    # which defines inputs for some_func  ...
    response = some_func(inputs)

from external_lib import some_func

def decorator(func):
    # ... custom logic ...

def main():
    # ... complex logic that you can't refactor
    # which defines inputs for some_func  ...
    response = decorator(some_func)(inputs)

from external_lib import some_func

def decorator(func):
    # ... custom logic ...

def main():
    # ... complex logic that you can't refactor
    # which defines inputs for some_func  ...
    @decorator
    def pointless_wrapper():
        return some_func(inputs)

    response = pointless_wrapper()

A real world example

I came across the above reasonings while I was working on error handling in a serverless architecture based on AWS.

Specifically, my need was to catch and retry a specific error raised by awswrangler (aws-sdk-pandas) when an Athena query execution fails i.e., awswrangler.exceptions.QueryFailed. This error is often raised by awswrangler.athena.read_sql_query or awswrangler.athena.start_query_execution, the latter being used with wait=True.

I recently read and enjoy Robust Python by Patrick Viafore - which I totally recommend to seasoned Python developers, for a lot of reasons far beyond the scope of this post - where the author recommends the backoff library for such tasks.

Backoff offers a handy decorator-based approach to handle retry logic in a wide range of different situations, with some customization options to be feed as decorator kwargs.

Thanks to the intuitive and well documented API, after few minutes of experiments I came up with this setup which seemed to do the job:

import awswrangler as wr
import backoff

@backoff.on_exception(backoff.expo, wr.exceptions.QueryFailed, max_tries=3)

I was ready to add the customized decorator to my codebase, but I then realized that I do not have access to awswrangler.athena functions definition. I suddenly found myself wondering whether or not I can decorate function invocation instead: a quick Google search gave me the simple (yet forgotten) answer.

All I had to do was aliasing my custom decorator and then use it wherever required by simply wrapping original function call with the defined alias³.

BeforeAfter

import awswrangler as wr

response = wr.athena.read_sql_query(sql, database)

import awswrangler as wr
import backoff

retry_when_query_fails = backoff.on_exception(
    backoff.expo,
    wr.exceptions.QueryFailed,
    max_tries=3
)

response = retry_when_query_fails(wr.athena.read_sql_query)(sql, database)

---

1. This can be referred to as syntactic diabetes. ↩

2. This blog post gives well explained details about both closures and decorators in Python. ↩

3. Since backoff library exposes functions which can be used as decorators, this ensures that they also can be used as plain closures. ↩