Def Over Lambda Part 1 - Factory Pattern and Dependency Injection

I recently read part of a pretty awesome book called Let Over Lambda by Doug Hoyte. Full of nice examples, wisdom and funnily opinionated remarks. I was also working on a personal project involving a couple of libraries where I suddenly saw how I could use one of the first patterns explained in the book.

Let Over Lambda

In lisp and lisp like languages you can rely on lexical binding to manage scope in a way that is for today's standards (at least to me) intuitive. When we define a variable in a let expression, we can still access it from within a lambda (which means any function), defined within the let scope.

Here is an example:

    (let ((myvar 0))
      (lambda () (incf myvar)))

In this lambda we can access myvar, which means we can build a lambda over let over lambda (lol) to have a very primitive factory of functions.

  (setq lexical-binding t)
  (require 'cl)
  (defun myfactory ()
     (let ((myvar 0))
       (lambda () (incf myvar))))

  (funcall (myfactory)))
1

In this example the variable defined outside the function can be referenced within the lambda we output from the factory. This code block will always return 1. Storing the output lambda function, and calling that "instance" multiple times, will yield a result which may not be immediately expected:

    (require 'cl)
    (defun myfactory ()
      (let ((myvar 0))
        (lambda () (incf myvar))))
    (let ((mylambda (myfactory)))
      (dotimes (j 5) (funcall mylambda))
      (funcall mylambda))
6

By calling multiple times the same function produced by the factory, we effectively create a "counter" feature by taking advantage of the lexical closure (let over lambda). This can also be used not just to implement counter, the same principle can be used to implement accumulators and factories that allow dependency injections and who knows what else.

Leveraging closures in Python

Accumulators and Factories

In my use case, I want to collect a list of authors and their songs from Spotify in a dictionary that has the song title as key and the author as value. However, there are multiple ways, or sources, to obtain songs and authors from the Spotify API. For example, we can obtain them from playlists, or from an artist or then again from an album.

I may want to obtain, in a single run of the program, the songs and authors from multiple calls to the function that retrieves them, so I will need a single variable to fill with the data from each call. Obviously, this can be easily achieved with a more procedural approach without any problems since it is a simple task. But for the sake of our exercises, let's say we want to keep this behavior behind the veil of our function. Our function will return the songs and authors dictionary, and if we call it again providing a different input, the function will return the previous dictionary updated with all the new songs and authors. We can then decide to disregard its output until the very last call. This accumulator we built is really no different than the previous counter implemented in lisp, and it is really no different than implementing memoization, aka a cache, in our function calls. Same principle, multiple uses.

Let's take a look at the actual implementation:

  def get_spotify_funcs(id_type: IdType) -> Callable[[dict], dict[str,str]]:
    song_auths: dict[str, str] = {} # Contains the song name and author
    match id_type:
      case IdType.PLAYLIST:
          def get_tracks(playlist:dict):
              for track in range(len(playlist["tracks"]["items"])):
                  song = playlist["tracks"]["items"][track]["track"]["name"]
                  auth = playlist["tracks"]["items"][track]["track"]["artists"][0]["name"]
                  song_auths[song] = auth
              return song_auths
      case IdType.ARTIST:      
          def get_tracks(playlist:dict):
              for track in range(len(playlist["tracks"])): 
                   song = playlist["tracks"][track]["name"]
                   auth = playlist["tracks"][track]["artists"][0]["name"]
                   song_auths[song] = auth
              return song_auths
      case _: raise NotImplementedError()
    return get_tracks

As mentioned, we can have different sources for our tracks from the Spotify API, this means we need different parsing for each, we can then make a function tailored for each object returned by the API depending on the API object we expect to parse. We have already implemented a factory pattern that returns functions rather than instantiated objects. Then, by closure, we reference in each function the dictionary that will be populated each time the function is called. We have implemented an accumulator/counter.

You may wonder why the match case is outside the returned function, why not have it inside. Doing it with a single nested function definition that then handles the match case is totally fine in my opinion, but the nice thing about this approach with different function definitions is that our outer factory/accumulator function get_spotify_funcs takes one argument, and also the nested functions require only one argument, simpler minimalist functions == you'll have a better time. They do not even have to know about external models or business data structure/logic to do their job, hence possible refactoring becomes easier as well. Second, smaller simpler functions are less bug prone and easier to test/refactor/remove. Third, we are practicing let over lambda and all its goodness, so we use it to its full potential.

Dependency injection

If you are not familiar with dependency injection, I have made an article some time ago on it covering approaches in python and go. There are different ways to achieve it, but the approach in this article will be different from the old one, it will be closer to the golang way of achieving dependency injection via decorator pattern.

Let's now say that we want to pass our functions produced by our factory function to a third party library that promises to implement them so that they will use them to get a list of songs and authors and update our apple music account with these songs and authors as our favorites. The third party library's function wants a function that will take a dictionary of strings, and return a generator of dictionary of strings. This is pretty easy to adjust, by just yielding instead of returning and correctly type hinting the inner functions, we can make mypy or whatever type hinter happy and we can expect the third party library to work as expected.

def third_party_function(
          generator_func: Callable[[dict[str,Any]], Generator[dict[str,str], None, None]]
  ):
      ...

Let's say, however, that I also want to include some calls to my Signal notification app. I want that every time my favorite song, played by my favorite artist (no covers please) is found, I get a notification message from my Signal bot so that I know my favorite song has finally been added. Well, something like this implies external dependencies, and possibly passing a lot of new arguments to our functions that would break the contract with the third party library function. But, thanks to our let over lambda approach, we can inject such dependencies taking advantage of the inner functions' scope. We can instanciate our notification client in the factory function, and then reference it/use it in our inner function, which will still satisfy the expectation of the third library function:

  def get_spotify_funcs(
          id_type: IdType
  ) -> Callable[[dict[str, Any]], Generator[dict[str, str], None, None]]:
    song_auths: dict[str, str] = {} # Contains the song name and author
    match id_type:
      case IdType.PLAYLIST:
          client = NotificationClient()
          def get_tracks(playlist: dict[str, Any]) -> Generator[dict[str, str], None, None]:
              for track in range(len(playlist["tracks"]["items"])):
                  song = playlist["tracks"]["items"][track]["track"]["name"]
                  auth = playlist["tracks"]["items"][track]["track"]["artists"][0]["name"]
                  if MY_FAVORITE_SONG in song and MY_FAVORITE_ARTIST in auth:
                      client.send(f"Song: {song} by {auth} found.")
                  song_auths[song] = auth
                  yield song_auths
              
      case IdType.ARTIST:      
          def get_tracks(playlist: dict[str, Any]) -> Generator[dict[str, str], None, None]:
              for track in range(len(playlist["tracks"])): 
                   song = playlist["tracks"][track]["name"]
                   auth = playlist["tracks"][track]["artists"][0]["name"]
                   song_auths[song] = auth
                   yield song_auths
      case _: raise NotImplementedError()
    return get_tracks

Finally, with this we have achieved 3 different things: An accumulator, dependency injection that will not violate the signature required by a third party library, and a factory pattern. Using these techniques can help maintain more composable and decoupled code, however I believe there should be a balance in choosing how many things and what things to reference from outside the scope of the inner functions. Complex objects that themselves have lots of dependencies and/or logic can become more difficult to debug, and if you need to reference too many things from outside the function it may indicate that you should refactor things so that they are more composable and require fewer shortcuts as these. So, closures can be powerful, but they can also hide dependencies and complexity. Always strive for a balance between clarity and cleverness.