nbconvert requires lxml_clean_html

[sjkim2322]

Describe the issue

If you run the notebook component on a runtime image that does not have lxml 5.2.0 or lower or lxml_clean_html installed in advance, the following error occurs.

        "lxml.html.clean module is now a separate project lxml_html_clean.\n"
        "Install lxml[html_clean] or lxml_html_clean directly." 

To Reproduce
Steps to reproduce the behavior:

1. Prepare a Python runtime image of a clean environment (in my case it is “docker.io/python:3.10.12”).
2. Run the elyra notebook component with the above image set as the runtime image.
3. The above error occurs while initializing the notebook environment.

Cause I'm guessing

• The lxml version is not specified in the requirements of nbconvert==6.5.1 used by elyra.
  Therefore, when lxml is newly installed, the latest version is installed, and this version requires lxml_clean_html.
• the changes in lxml 5.2.0 version.

Expected behavior

• I can pre-install it on the runtime image I will use, but since all images require work, it would be better to add it to elyra's requirements.

Deployment information
Describe what you've deployed and how:

• Elyra version: [3.16.0.dev0]
• Installation source: [PyPI]
• Deployment type: [Kubeflow [notebook server] ]
• Operating system: [linux]

[lresende]

Should this be in Elyra? Or in nbconvert ?

[sjkim2322]

@lresende
Thank you for answer.
I will also raise the issue on nbconvert.

However, I don't know if it is possible to modify the module dependency of a specific version that has already been deployed

In elyra, when the Jupyter notebook component is executed, it appears to download and install the requirements here. Since nbconvert is being installed here, I think it would be good if lxml_clean_html is also added here.

or, If there is a spec that can pre-initialize the notebook component's cell before execution, it may be possible to use it.

[lresende]

As you can see, we already removed a lot of transient dependencies from the requirements file to avoid us having to keep syncing to new versions, etc... if they don't enable that in nbconvert itself, then we can continue this conversation.

[sjkim2322]

Yes, I left an issue on nbconvert.
jupyter/nbconvert#2148

[lresende]

@sjkim2322 based on your nbcovert issue, can't we update the version of nbconvert?
does that still work with JupyterLab < 4?

[shalberd]

@lresende @sjkim2322

I can confirm that we can just update the version of nbconvert without any issues regarding runtime behavior with the runtime image.

I can specifically confirm that nbconvert in a higher version works fine with Jupyterlab less than 4. See this code here from Red Hat Open Data Hub folks. They still use Jupyterlab less than 4 as well:

https://github.com/opendatahub-io/notebooks/blob/main/runtimes/minimal/ubi9-python-3.9/utils/requirements-elyra.txt

https://github.com/opendatahub-io/notebooks/blob/main/jupyter/datascience/ubi9-python-3.9/Pipfile

We at our org have this running sucessfully. So yes, you can change to, among other version updates most likely

nbconvert==7.1.0

no compatibility issues with Jupyterlab 3.6.7

See my list of packages in my runtime image / jupyter image (I bake in the packages in the Jupyter image and runtime image combined to make these requirements for runtime elyra available airgapped, no problem.

I think if you sync or update requirements-elyra in this project in line with what is listed up in the link for requirments-elyra of opendatahub-io, you won't have any issues anymore, be it nbconvert or the other runtime requirements packages.

[1001050000@s-testjupyter-0 ~]$ pip list | grep nbconvert
nbconvert                                7.16.4
[1001050000@s-testjupyter-0 ~]$ pip list | grep jupyter
jupyter-bokeh                            3.0.7
jupyter_client                           7.4.9
jupyter_core                             5.7.2
jupyter-events                           0.10.0
jupyter-lsp                              2.2.5
jupyter_packaging                        0.12.3
jupyter-resource-usage                   0.7.2
jupyter_server                           2.14.0
jupyter_server_fileid                    0.9.2
jupyter-server-mathjax                   0.2.6
jupyter_server_proxy                     4.0.0
jupyter_server_terminals                 0.5.3
jupyter_server_ydoc                      0.8.0
jupyter-ydoc                             0.2.5
jupyterlab                               3.6.7
jupyterlab_git                           0.44.0
jupyterlab-lsp                           4.2.0
jupyterlab_pygments                      0.3.0
jupyterlab_server                        2.27.1
jupyterlab-streamlit-menu                0.1.0
jupyterlab_widgets                       3.0.10

# This is a comprehensive list of python dependencies that Elyra requires to execute Jupyter notebooks.
ipykernel = "==6.13.0"
ipython = "==8.10.0"
ipython-genutils = "==0.2.0"
jinja2 = "==3.0.3"
jupyter-client = "==7.3.1"
jupyter-core = "==4.11.2"
MarkupSafe = "==2.1.1"
minio = "==7.1.15"
nbclient = "==0.6.3"
nbconvert = "==7.1.0"
nbformat = "==5.4.0"
papermill = "==2.3.4"
pyzmq = "==24.0.1"
prompt-toolkit = "==3.0.30"
requests = "==2.31.0"
tornado = "==6.3.3"
traitlets = "==5.10.0"
urllib3 = "==1.26.18"

@lresende agreed the runtime and transitive dependencies issue is a little bit of a pain, but as mentioned, lifting the library version should be fine, even with Jupyterlab less than 4. I have tested this running in conjunction with airflow as a runtime engine, executing notebooks in tasks.