Def Over Lambda Part 2 - Partials and Memoization

Using the let over lambda pattern with lexical binding allows us to return functions, because functions are first class objects in Python and they are values like any other in lisp languages like Common Lips or Emacs Lisp. This means that, as we have seen in the previous article, we can inject data in the functions by exploiting their scope with closures and we can also wrap extra behavior around them. However, sometimes we want to do less with a function rather than more, perhaps because we do not have all the arguments needed at hand in a certain moment, but we want the function to be already called with at least one, or some, of its arguments. Partially applying a function becomes really helpful with third party libraries: a function from the library requires 3 arguments, but you only have 2 available at the moment and will know the third further ahead. Or perhaps, you have an argument that will always be the same, and rather than passing it every single time, you would rather pass it just once.

For example, we may have a third party library function that adds a game name to different strings. We know the game name from the beginning, so technically we can be sure from the beginning about this argument, and we also know it will be the same every time the formatting function is called. So, instead of passing the game name string variable every time we call the formatting function, we can `partially apply` the function, by providing it only the game name. This will return a new function that only requires the string to format to be passed now, avoiding repetition in our code, and in some other cases allowing us to prepare some arguments to a function beforehand.

  (setq lexical-binding t)
  (let ((game "baldur's gate"))
    (defun format-game-name (string-to-format game-name)
      (format string-to-format game-name))

    ;; lambda let over lambda example
    (lambda (game-name)
      (lambda (string-to-format)
        (format-game-name string-to-format game-name)))

    (defun wrapper (game-name)
      (lambda (string-to-format)
        (format-game-name string-to-format game-name)))
    
    (let ((partial-format-game-name (wrapper game)))
      (funcall partial-format-game-name "Test string for game: %s")
      (funcall partial-format-game-name  "%s rocks")))

baldur's gate rocks

Notice how we can reuse our function without having to provide the game name anymore. This is easily achievable in Python either by using the functools partial function, otherwise it can be easily reimplemented with lambdas or by defining a wrapper function. The reason why you may want to handroll your partial utility is that you may want to preemptively pass some keyword arguments that are not in positional order to a function: e.g., fun (a,b,c,d,e) and you want to partially pass only the b and d arguments.

  from functools import partial

  def format_game_name (str_to_format: str, game_name: str):
      return str_to_format.format(game_name)

  apply_to_string = partial(format_game_name, game_name="baldur's gate")

  apply_to_string_lambda = lambda game_name: lambda str_to_format: format_game_name(str_to_format, game_name)
  
  apply_to_string_lambda("baldur's gate3")("test name chosen: {} ")

test name chosen: baldur's gate3 

Memoization, also called "what you give for granted until middle age starts catching up". This is a good way of improving performance, reduce costs, use more memory, cause memory leaks, and much more. With memoization we take advantage of the scope in the same way as it was done in the previous article with the accumulators, but this time the objective is to use a data structure which matches the usecase of caching.

By caching the results of a function, we can save computation time for heavy computations, or API/network costs, but we also have to remember that, just like an accumulator, this cache grows in size as the number of cached arguments increases, and that sometimes we DO want the latest response from an API, and not an old cached one. This being said, a good data structure to use is the hashmap, and in elisp it is built in.

We make this memoize wrapper function that stores as key in the hash table the argument passed to the function, and as value the output of the function. If the argument was already passed, we just retrieve the previously stored output.

    (setq lexical-binding t)
    (defun memoize (fn)
        (let ((cache (make-hash-table :test 'equal)))
          (lambda (arg)
            (if (gethash arg cache)
                (progn (message "FOUND from cache")
                       (gethash arg cache))
              (progn
                (message "NOT found in cache")
                (puthash arg (funcall fn arg) cache)
                (gethash arg cache))))))

      (defun get-request (endpoint)
        (request (format "https://www.youtube.com/%s" endpoint)
          :parser (lambda () (buffer-string))))
      (get-request "shorts")

      (let ((cached-get-request (memoize #'get-request)))
        (funcall cached-get-request "shorts")
        (funcall cached-get-request "shorts")
        (funcall cached-get-request "shorts"))

The implementation in Python is very similar, and the data structure we can use is the dictionary for the same task of caching.

  import requests
  def memoize (func):
       cache = {}
       def memoized_func(arg):
           if cache.get(arg):
               print("Found in cache")
               return cache.get(arg)
           print("Not found in cache!")
           cache[arg] = func(arg)
           return cache.get(arg)
       return memoized_func

  def get_request(endpoint):
      return requests.get(f"https://www.youtube.com/{endpoint}")._content[:50]

  cached_get_request = memoize(get_request)
  for _ in range(2):
      cached_get_request("shorts")
  cached_get_request("shorts")

b'<!DOCTYPE html><html style="font-size: 10px;font-f'

And so, by wrapping our function and providing a cache system to it, we reduce the number of calls to youtube in this case. And we achieve all of this with a functional approach by passing functions around as data. The functools module offers a lru_cache decorator as well, which basically does what this memoize function does, but better optimized.