Logs flooded with 403s (original) (raw)
Thanks for having a look at this minrk. I do see the below error at start of singleuser server logs which I thought unrelated but perhaps not. I checked db for a couple users and thyey still have valid tokens:
cat jupyterhub_slurmspawner_1209786.log
[I 2025-03-18 13:42:20.240 ServerApp] Extension package dask_labextension took 1.6196s to import
[W 2025-03-18 13:42:20.240 ServerApp] A `_jupyter_server_extension_points` function was not found in dask_labextension. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
[I 2025-03-18 13:42:20.615 ServerApp] Extension package jupyterlab took 0.2812s to import
[I 2025-03-18 13:42:23.245 ServerApp] Extension package jupyterlab_h5web took 1.2598s to import
[I 2025-03-18 13:42:23.498 ServerApp] Extension package jupyterlab_jupytext took 0.2533s to import
[I 2025-03-18 13:42:23.792 ServerApp] Extension package pixi_kernel took 0.2905s to import
[I 2025-03-18 13:42:24.282 ServerApp] Extension package voila.server_extension took 0.4899s to import
[I 2025-03-18 13:42:24.282 ServerApp] dask_labextension | extension was successfully linked.
[I 2025-03-18 13:42:24.282 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2025-03-18 13:42:24.282 ServerApp] jupyter_server_proxy | extension was successfully linked.
[I 2025-03-18 13:42:24.288 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2025-03-18 13:42:24.288 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 5.2.1
[I 2025-03-18 13:42:24.288 JupyterHubSingleUser] Using default url from environment $JUPYTERHUB_DEFAULT_URL: /lab/tree/nsls2/users/jsmith
[I 2025-03-18 13:42:24.291 ServerApp] jupyterhub | extension was successfully linked.
[W 2025-03-18 13:42:24.294 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. This config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2025-03-18 13:42:24.296 ServerApp] jupyterlab | extension was successfully linked.
[I 2025-03-18 13:42:24.296 ServerApp] jupyterlab_h5web | extension was successfully linked.
[I 2025-03-18 13:42:24.296 ServerApp] jupyterlab_jupytext | extension was successfully linked.
/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/traitlets/traitlets.py:1897: DeprecationWarning: ServerApp.token config is deprecated in jupyter-server 2.0. Use IdentityProvider.token
return t.cast(Sentinel, self._get_trait_default_generator(names[0])(self))
[I 2025-03-18 13:42:24.311 ServerApp] notebook_shim | extension was successfully linked.
[I 2025-03-18 13:42:24.311 ServerApp] pixi_kernel | extension was successfully linked.
[I 2025-03-18 13:42:24.311 ServerApp] voila.server_extension | extension was successfully linked.
[I 2025-03-18 13:42:24.361 ServerApp] notebook_shim | extension was successfully loaded.
[I 2025-03-18 13:42:24.361 ServerApp] dask_labextension | extension was successfully loaded.
[I 2025-03-18 13:42:24.363 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2025-03-18 13:42:24.383 ServerApp] jupyter_server_proxy | extension was successfully loaded.
[I 2025-03-18 13:42:24.383 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2025-03-18 13:42:24.400 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2025-03-18 13:42:24.400 ServerApp] jupyterhub | extension was successfully loaded.
[I 2025-03-18 13:42:24.435 LabApp] JupyterLab extension loaded from /nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab
[I 2025-03-18 13:42:24.435 LabApp] JupyterLab application directory is /nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/share/jupyter/lab
[I 2025-03-18 13:42:24.436 LabApp] Extension Manager is 'pypi'.
[I 2025-03-18 13:42:24.459 LabApp] Extensions will be fetched using proxy, proxy host and port: ('proxy.sdcc.bnl.local', '3128')
[I 2025-03-18 13:42:24.464 ServerApp] jupyterlab | extension was successfully loaded.
[I 2025-03-18 13:42:24.670 ServerApp] Jupyterlab-h5web extension will serve HDF5 files from /
[I 2025-03-18 13:42:24.670 ServerApp] jupyterlab_h5web | extension was successfully loaded.
[W 2025-03-18 13:42:24.671 ServerApp] [Jupytext Server Extension] Async contents managers like {base_class.__name__} are not supported at the moment (https://github.com/mwouts/jupytext/issues/1020). We will derive a contents manager from LargeFileManager instead.
[I 2025-03-18 13:42:24.671 ServerApp] [Jupytext Server Extension] Deriving a JupytextContentsManager from LargeFileManager
[I 2025-03-18 13:42:24.673 ServerApp] jupyterlab_jupytext | extension was successfully loaded.
pixi-kernel INFO: Registered pixi_kernel server extension
[I 2025-03-18 13:42:24.673 ServerApp] pixi_kernel | extension was successfully loaded.
[I 2025-03-18 13:42:24.678 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2025-03-18 13:42:24.685 ServerApp] voila.server_extension | extension was successfully loaded.
[I 2025-03-18 13:42:24.686 ServerApp] Serving notebooks from local directory: /
[I 2025-03-18 13:42:24.687 ServerApp] Jupyter Server 2.15.0 is running at:
[I 2025-03-18 13:42:24.687 ServerApp] http://hpc005.nsls2.bnl.local:58481/user/jsmith/lab/tree/nsls2/users/jsmith?token=...
[I 2025-03-18 13:42:24.687 ServerApp] http://127.0.0.1:58481/user/jsmith/lab/tree/nsls2/users/jsmith?token=...
[I 2025-03-18 13:42:24.687 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2025-03-18 13:42:27.949 ServerApp] New terminal with automatic name: 1
[I 2025-03-18 13:42:28.584 LabApp] Build is up to date
pixi-kernel INFO: TermSocket.open: 1
pixi-kernel INFO: TermSocket.open: Opened 1
2025-03-18 13:42:31,746 - ServerApp - ERROR - Uncaught exception GET /user/jsmith/lab/api/extensions?query&page=1&per_page=30&refresh=0&1742319751723 (192.203.218.24)
HTTPServerRequest(protocol='https', host='jupyter.nsls2.bnl.gov', method='GET', uri='/user/jsmith/lab/api/extensions?query&page=1&per_page=30&refresh=0&1742319751723', version='HTTP/1.1', remote_ip='192.203.218.24')
Traceback (most recent call last):
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/tornado/web.py", line 1790, in _execute
result = await result
^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/handlers/extension_manager_handler.py", line 37, in get
extensions, last_page = await self.manager.list_extensions(query, page, per_page)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/manager.py", line 474, in list_extensions
await self.refresh(query, page, per_page)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/manager.py", line 508, in refresh
await self._update_extensions_list(query, page, per_page)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/manager.py", line 681, in _update_extensions_list
extensions, last_page = await self.list_packages(query, page, per_page)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/pypi.py", line 270, in list_packages
matches = await self.__get_all_extensions()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/pypi.py", line 324, in __get_all_extensions
self.__all_packages_cache = await self.__throttleRequest(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/pypi.py", line 223, in __throttleRequest
data = await current_loop.run_in_executor(None, fn, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1122, in __call__
return self.__send(self.__name, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1461, in __request
response = self.__transport.request(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1166, in request
return self.single_request(host, handler, request_body, verbose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1178, in single_request
http_conn = self.send_request(host, handler, request_body, verbose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1291, in send_request
self.send_content(connection, request_body)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1321, in send_content
connection.endheaders(request_body)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/http/client.py", line 1331, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/http/client.py", line 1091, in _send_output
self.send(msg)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/http/client.py", line 1035, in send
self.connect()
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/http/client.py", line 1011, in connect
self._tunnel()
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/http/client.py", line 979, in _tunnel
raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 403 Forbidden
[W 2025-03-18 13:42:31.750 ServerApp] wrote error: 'Unhandled error'
Traceback (most recent call last):
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/tornado/web.py", line 1790, in _execute
result = await result
^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/handlers/extension_manager_handler.py", line 37, in get
extensions, last_page = await self.manager.list_extensions(query, page, per_page)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/manager.py", line 474, in list_extensions
await self.refresh(query, page, per_page)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/manager.py", line 508, in refresh
await self._update_extensions_list(query, page, per_page)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/manager.py", line 681, in _update_extensions_list
extensions, last_page = await self.list_packages(query, page, per_page)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/pypi.py", line 270, in list_packages
matches = await self.__get_all_extensions()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/pypi.py", line 324, in __get_all_extensions
self.__all_packages_cache = await self.__throttleRequest(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/site-packages/jupyterlab/extensions/pypi.py", line 223, in __throttleRequest
data = await current_loop.run_in_executor(None, fn, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1122, in __call__
return self.__send(self.__name, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1461, in __request
response = self.__transport.request(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1166, in request
return self.single_request(host, handler, request_body, verbose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1178, in single_request
http_conn = self.send_request(host, handler, request_body, verbose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1291, in send_request
self.send_content(connection, request_body)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/xmlrpc/client.py", line 1321, in send_content
connection.endheaders(request_body)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/http/client.py", line 1331, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/http/client.py", line 1091, in _send_output
self.send(msg)
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/http/client.py", line 1035, in send
self.connect()
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/http/client.py", line 1011, in connect
self._tunnel()
File "/nsls2/software/common/jupyter/jupyter-server-env/.pixi/envs/prod/lib/python3.12/http/client.py", line 979, in _tunnel
raise OSError(f"Tunnel connection failed: {code} {message.strip()}")
OSError: Tunnel connection failed: 403 Forbidden