Generating Multiple Dash Instances
The problem
While building a webapp for text analysis and NLP things, I noticed that the project was getting bigger than expected, and having everything run in a single monolithic dash instance was too messy. Therefore, I started looking into how I could combine several dash instances, while still working on a single project and single api server. Needless to say, it was a bit tricky at times, and Dash requires you to do some silly things if you want to decouple and organize your project into several modules.
Multiple dash apps at the same time
The solutions seems simple at first sight:
from flask import Flask
server = Flask(__name__)
dash_instance = dash.Dash(name, server=server, url_base_pathname="/dashboard/")
however, we still have the dash app in the same file, we may also want to have the callbacks in another
Expected Result
What I wanted, was to be able to generate a dash instance for every section of the platform, so, from the screenshot, each item in the menu would link to a different dash instance, all of them running in the same process however, so that this application could be easily deployable with a single Dockerfile.
From hardcoded mess to easily extensible product
As you can see from the code commented out, this module used to have all the dash instances basically hardcoded. By using a config file and importlib, however, we can drastically reduce the number of lines of code and easily extend our codebase to include more instances. I have also added the possibility to create template python files and needed directories when a new dash instance is added to the config, but its modules are not present.
"""
from home.home import app as home
from home.home_layout import layout as home_layout
from server.server import server
from vowel_clustering.vowel_clustering import app as vwl_clst
from vowel_clustering.vowel_clustering_layout import layout as vwl_clast_layout
from word_frequency.word_frequency import app as wf
from word_frequency.word_frequency_layout import base as wf_layout
from chi_squared.chi_squared import app as chi_squared_app
from chi_squared.chi_squared_layout import base as chi_squared_layout_var
from global_components.utils import layout
from text_classification.text_classification import app as text_classification_app
from text_classification.text_classification_layout import base as text_classification_layout
app = home
app.layout = layout # home_layout
app1 = vwl_clst
app1.layout = layout # vwl_clst_layout
app2 = wf
app2.layout = layout # layout # wf_layout
app3 = chi_squared_app
app3.layout = layout # chi_squared_layout_var
app4 = text_classification_app
app4.layout = layout
"""
This is instead an approach that allows to add as many dash instances as you want, without having to worry about updating anything aside from a json config file with the name of the instance and endpoint path.
import os
from importlib import import_module
from global_components.module_generator import ModuleGenerator
from server.server import server
from utils import load_module_configs
apps = []
modules = load_module_configs()
existing_modules = [f.path[2:] for f in os.scandir() if f.is_dir() and f.path[2:] in modules]
for module, endpoint in modules.items():
if module not in existing_modules:
module_generator = ModuleGenerator(".", module)
module_generator.generate_new_module()
dash_app = getattr(import_module(f"{module}.{module}"), "app")
dash_app.layout = getattr(import_module(f"{module}.{module}_layout"), "layout")
apps.append(dash_app)
if __name__ == "__main__":
server.run(debug=False, host="0.0.0.0", port=8080)
Main Flask server
The apps will set the server to this Flask instance in a dedicated module.
"""
This module creates the server instance.
"""
from flask import Flask
server = Flask(__name__)
So we can now instanciate each dash app like the following, and have for each endpoint (url_base_pathname) a dedicated dash instance.
from server.server import server
def generate_app(name, url_base_pathname):
return dash.Dash(name, server=server, external_stylesheets=[dbc.themes.LUX],
url_base_pathname=f"{url_base_pathname}", title="NLP Workspace")
app = generate_app(__name__, url_base_pathname='/')
With the following structure we can organize our dash apps, each one in its own directory, with an app module, a dedicated module for callbacks and layout.
|
|- server
| |
| |- flask_server
|
|- app1
| |
| |- app_module
| |- callback_module
| |- layout_module
Since the structure is quite simple and clear, it is also easy to extend and add new ones in a programmatic way using simple config files.
Dynamically generating dash instances
With the following we then load on startup, from the main module, all the dash apps, and also create the app_module, callback_module and layout_module templates of the ones included in the json config file but not yet created. To do this we will use a class whose job is to generate the directory and template files.
modules = load_module_configs()
existing_modules = [f.path[2:] for f in os.scandir() if f.is_dir() and f.path[2:] in modules]
for module, endpoint in modules.items():
if module not in existing_modules:
module_generator = ModuleGenerator(".", module)
module_generator.generate_new_module()
dash_app = getattr(import_module(f"{module}.{module}"), "app")
dash_app.layout = getattr(import_module(f"{module}.{module}_layout"), "layout")
apps.append(dash_app)
Module generator class
class ModuleGenerator:
def __init__(self, pwd, module_name):
self.pwd = pwd
self.module_name = module_name
# TODO: Check module name only contains alpha and underscores
self.files_to_generate = [f"{self.module_name}/__init__.py",
f"{self.module_name}/{self.module_name}.py",
f"{self.module_name}/{self.module_name}_layout.py"]
def _create_new_module_files(self):
logger.info("Generating new module directory and files...")
os.mkdir(self.module_name)
for file in self.files_to_generate:
with open(file, "w") as module_file:
module_file.write("")
def _fill_main_module(self):
with open(self.files_to_generate[1], "w") as file:
file.write(main_module_code % self.module_name)
def _fill_layout_module(self):
with open(self.files_to_generate[2], "w") as file:
file.write(layout_module_code % (self.module_name, self.module_name, self.module_name))
def generate_new_module(self):
self._create_new_module_files()
logger.info("Creating templates for new module files...")
self._fill_main_module()
self._fill_layout_module()
logger.info("New dash module created")
Lessons learned
Dash has some behaviors that do not play nicely with some of the tools from IDE’s such as PyCharm. For example, if you decide to use the same structure and have the app, callbacks and layout in different files, you need to import the callbacks module (even if you do not use anything from it) into the layout module. This, however will make the import line be highlighted as a useless import by PyCharm, and if you format the file automatically with Ctrl+Alt+L, you will lose the import and the callbacks will not work.
To check how all this was implemented in a project, check out this repository https://github.com/andcarnivorous/NLP-workspace-dashapp