Jupyter-rsession-proxy with batchspawner (original) (raw)
a bit about the environment, it's an old school beowulf style cluster running rocky9 (RHEL variant) and I've not figured out how to get jupyterhub to spawn docker images on the compute nodes yet. I run the compute nodes diskless and added the rstudio rpm from posits to the compute image, I have a conda environmnet that contains batchspawner, r-base,jupyter-singleuser, nodejs, and jupyter-rsession-proxy packages.
I login to the hub, select the profile that calls the rstudio instance and it starts a juplyterlab instance that includes rstudio as a launcher. I launch it and it opens a new window that goes to https://servername.edu/auth-sign-in?appUri=%2f and not https://servername.edu/prefix/user/username/rstudio/auth-sign-in?appUri=%2f Now for the fun, if I change the URL to https://servername.edu/prefix/user/username/rstuido/auth-sign-in?appUri=%2f it lets me right into the rstudio instance and everything works fine.
The user who is requesting this wants to use it in an intro class though, so that isn't going to work.
Here are the relevant parts of my server config:
rstudioProlog = """
hostname
source /opt/ohpc/pub/Anaconda3/bin/activate /opt/ohpc/pub/conda-envs/rstudio
export LD_LIBRARY_PATH=/opt/ohpc/pub/conda-envs/rstudio/lib
export RS_LOG_LEVEL=DEBUG
export JUPYTER_RSESSION_PROXY_USE_SOCKET=yes
NB_USER=$USER
"""
c.SlurmSpawner.req_keepvars_extra = 'ALL'
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.ProfilesSpawner.profiles = [
('Compute node - 2 cores, 4GB, 2 Hours', 'ccn2c16g10min', 'batchspawner.SlurmSpawner',
dict(req_nprocs='2', req_queue='ALL', req_runtime='02:00:00', req_memory='4G', req_prologue = standardProlog)),
('Compute Node - 32 cores, 32 GB, 8 hours', 'ccn8c32gb8h', 'batchspawner.SlurmSpawner',
dict(req_nprocs='32', req_queue='ALL', req_runtime='8:00:00', req_memory='32G', req_prologue=standardProlog)),
('GPU Node - 2 cpu core, 1 gpu, 32 GB, 4 hours', 'ccn2c32gb4h1gpu', 'batchspawner.SlurmSpawner',
dict(req_nprocs='2', req_runtime='04:00:00', req_memory='32G', req_options='-p gpu --gres=shard:4', req_prologue = standa
rdProlog)),
('Rstudio-testing - adminonly', 'ccn2c15g10rserv' , 'batchspawner.SlurmSpawner',
dict(req_nprocs='2', req_partition='R740', req_runtime='02:00:00', req_memory='4G', req_prologue = rstudioProlog)),
]
/etc/rstudio/rserver.conf and /etc/rstudio/rsession.conf are empty
rserver.log:
025-04-23T14:01:02.357189Z [rserver] DEBUG (PID 366260): Initialized file locks (lock-type=linkbased, use-symlinks=false, timeout-interv
al=30s, refresh-rate=20s, log-file=)
2025-04-23T14:01:02.357661Z [rserver] INFO Encryption versions set to max: 0, min: 0
2025-04-23T14:01:02.358139Z [rserver] DEBUG Creating secure key file: "/tmp/tmpix02lny2" key hash: (C0D032B2)
2025-04-23T14:01:02.358682Z [rserver] INFO Connecting to sqlite3 database at /tmp/tmp4gkeuuzz/rstudio-os.sqlite
2025-04-23T14:01:02.359160Z [rserver] INFO Creating database connection pool of size 6 (source: default maximum with 6 CPUs)
2025-04-23T14:01:02.361413Z [rserver] INFO Database schema has not been created yet. Creating database schema...
2025-04-23T14:01:02.363277Z [rserver] INFO No environment variables found at /etc/rstudio/env-vars
2025-04-23T14:01:02.364170Z [rserver] INFO Running without privilege; using secure key at /home/user/.cache/rstudio/session-rpc-key
2025-04-23T14:01:02.364768Z [rserver] DEBUG Using secure key file: /home/user/.cache/rstudio/session-rpc-key
2025-04-23T14:01:02.365542Z [rserver] DEBUG Secure key hash: (944FD2CD)
2025-04-23T14:01:02.366368Z [rserver] INFO Running without privilege; using secure key at /home/user/.cache/rstudio/session-rpc-key
2025-04-23T14:01:02.367036Z [rserver] DEBUG Using secure key file: /home/user/.cache/rstudio/session-rpc-key
2025-04-23T14:01:02.367584Z [rserver] DEBUG Secure key hash: (944FD2CD)
2025-04-23T14:01:03.025406Z [rserver] DEBUG Request for main page with no user-identifier - redirecting to login page: /
2025-04-23T14:01:03.028829Z [rserver] DEBUG Request for main page with no user-identifier - redirecting to login page: /
2025-04-23T14:01:27.761340Z [rserver] DEBUG Created new UserSession for: user (total: 1)
2025-04-23T14:01:27.762657Z [rserver] DEBUG User db query start: user
2025-04-23T14:01:27.763587Z [rserver] DEBUG No record found for user in db: user
2025-04-23T14:01:27.764423Z [rserver] DEBUG Adding user to database: user
2025-04-23T14:01:27.765816Z [rserver] DEBUG Adding session cookie: user|Wed%2C%2023%20Apr%202025%2015%3A01%3A27%20GMT|SS9BZjU%2BmwLnSh
%2BoMIbbOJ2frHRxkN3IukkxUDfAJAs%3D for user: user
here is the output from the slurm job:
[I 2025-04-23 19:45:50.708 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2025-04-23 19:45:50.708 ServerApp] jupyter_server_proxy | extension was successfully linked.
[I 2025-04-23 19:45:50.712 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2025-04-23 19:45:50.712 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 5.3.0
[I 2025-04-23 19:45:50.712 JupyterHubSingleUser] Using default url from server extension lab: /lab
[I 2025-04-23 19:45:50.715 ServerApp] jupyterhub | extension was successfully linked.
[W 2025-04-23 19:45:50.716 LabApp] 'extra_template_paths' was found in both NotebookApp and ServerApp. This is likely a recent change. Th
is config will only be set in NotebookApp. Please check if you should also config these traits in ServerApp for your purpose.
[I 2025-04-23 19:45:50.718 ServerApp] jupyterlab | extension was successfully linked.
[I 2025-04-23 19:45:51.128 ServerApp] notebook_shim | extension was successfully linked.
[I 2025-04-23 19:45:51.150 ServerApp] notebook_shim | extension was successfully loaded.
[I 2025-04-23 19:45:51.151 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2025-04-23 19:45:51.161 ServerApp] jupyter_server_proxy | extension was successfully loaded.
[I 2025-04-23 19:45:51.162 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2025-04-23 19:45:51.166 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2025-04-23 19:45:51.166 ServerApp] jupyterhub | extension was successfully loaded.
[I 2025-04-23 19:45:51.172 LabApp] JupyterLab extension loaded from /opt/ohpc/pub/conda-envs/rstudio/lib/python3.13/site-packages/jupyter
lab
[I 2025-04-23 19:45:51.172 LabApp] JupyterLab application directory is /opt/ohpc/pub/conda-envs/rstudio/share/jupyter/lab
[I 2025-04-23 19:45:51.173 LabApp] Extension Manager is 'pypi'.
[I 2025-04-23 19:45:51.254 ServerApp] jupyterlab | extension was successfully loaded.
[I 2025-04-23 19:45:51.255 ServerApp] Serving notebooks from local directory: /home/user
[I 2025-04-23 19:45:51.255 ServerApp] Jupyter Server 2.15.0 is running at:
[I 2025-04-23 19:45:51.255 ServerApp] http://compute-16-10:52665/jhub/user/user/lab?token=...
[I 2025-04-23 19:45:51.255 ServerApp] http://127.0.0.1:52665/jhub/user/user/lab?token=...
[I 2025-04-23 19:45:51.255 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2025-04-23 19:45:51.696 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascrip
t-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserve
r, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languages
erver-bin, vscode-json-languageserver-bin, yaml-language-server
[I 2025-04-23 19:45:51.866 ServerApp] 200 GET /jhub/user/user/api (@172.16.0.1) 1.87ms
[I 2025-04-23 19:45:51.929 ServerApp] 302 GET /jhub/user/user/ -> /jhub/user/user/lab? (@172.16.0.1) 0.98ms
[W 2025-04-23 19:45:51.941 ServerApp] No Hub user identified for request
[W 2025-04-23 19:45:51.941 ServerApp] Token stored in cookie may have expired
[I 2025-04-23 19:45:51.944 ServerApp] 302 GET /jhub/user/user/lab? -> /jhub/hub/api/oauth2/authorize?client_id=jupyterhub-user-user
&redirect_uri=%2Fjhub%2Fuser%2Fuser%2Foauth_callback&response_type=code&state=[secret] (@172.16.0.1) 6.98ms
[I 2025-04-23 19:45:51.994 ServerApp] Logged-in user user
[I 2025-04-23 19:45:51.995 ServerApp] Setting new xsrf cookie for b'3fe07d317983400ab7e3e446577e968c:3776f4904d7a349743d0eb17ffd716845243
1e291d7f79c7850b9c6343e960cb' {'path': '/jhub/user/user/'}
[I 2025-04-23 19:45:51.997 ServerApp] 302 GET /jhub/user/user/oauth_callback?code=[secret]&state=[secret] -> /jhub/user/user/lab? (
user@172.16.0.1) 17.20ms
[I 2025-04-23 19:45:52.043 ServerApp] 200 GET /jhub/user/user/lab? (user@172.16.0.1) 15.09ms
[I 2025-04-23 19:45:52.542 ServerApp] 200 GET /jhub/user/user/api/kernelspecs?1745437552435 (user@172.16.0.1) 24.59ms
[I 2025-04-23 19:45:52.621 ServerApp] 200 GET /jhub/user/user/lab/api/settings?1745437552449 (user@172.16.0.1) 78.53ms
[I 2025-04-23 19:45:52.621 ServerApp] 200 GET /jhub/user/user/api/me?1745437552435 (user@172.16.0.1) 78.03ms
[I 2025-04-23 19:45:52.623 ServerApp] 200 GET /jhub/user/user/api/terminals?1745437552455 (user@172.16.0.1) 1.28ms
[I 2025-04-23 19:45:52.624 ServerApp] 200 GET /jhub/user/user/api/sessions?1745437552456 (user@172.16.0.1) 1.87ms
[I 2025-04-23 19:45:52.630 ServerApp] 200 GET /jhub/user/user/lab/api/workspaces?1745437552460 (user@172.16.0.1) 7.84ms
[I 2025-04-23 19:45:52.631 ServerApp] 200 GET /jhub/user/user/api/kernels?1745437552454 (user@172.16.0.1) 8.22ms
[I 2025-04-23 19:45:52.636 ServerApp] 200 GET /jhub/user/user/api/kernelspecs?1745437552469 (user@172.16.0.1) 1.23ms
[I 2025-04-23 19:45:52.641 ServerApp] 101 GET /jhub/user/user/api/events/subscribe (user@172.16.0.1) 0.82ms
[I 2025-04-23 19:45:52.651 ServerApp] 200 GET /jhub/user/user/api/me?1745437552556 (user@172.16.0.1) 0.53ms
[I 2025-04-23 19:45:52.864 ServerApp] 200 GET /jhub/user/user/lab/api/translations/default?1745437552748 (user@172.16.0.1) 32.37ms
[I 2025-04-23 19:45:52.902 ServerApp] 200 GET /jhub/user/user/api/config/jupyterlabapputilsextensionannouncements?1745437552809 (user@172.16.0.1) 5.71ms
[I 2025-04-23 19:45:52.958 ServerApp] 200 GET /jhub/user/user/lsp/status?1745437552870 (user@172.16.0.1) 5.82ms
[I 2025-04-23 19:45:53.064 ServerApp] 200 GET /jhub/user/user/lab/api/translations?1745437552928 (user@172.16.0.1) 53.85ms
[I 2025-04-23 19:45:53.122 ServerApp] 200 GET /jhub/user/user/server-proxy/servers-info?1745437553039 (user@172.16.0.1) 0.89ms
[I 2025-04-23 19:45:53.214 ServerApp] 200 GET /jhub/user/user/lab/api/settings/@jupyterlab/codemirror-extension:plugin?1745437553127 (
user@172.16.0.1) 4.13ms
[I 2025-04-23 19:45:53.220 ServerApp] 200 GET /jhub/user/user/lab/api/settings/@jupyterlab/docmanager-extension:plugin?1745437553127 (
user@172.16.0.1) 4.68ms
[I 2025-04-23 19:45:53.230 ServerApp] 200 GET /jhub/user/user/lab/api/settings/@jupyterlab/notebook-extension:panel?1745437553129 (user@172.16.0.1) 5.08ms
[I 2025-04-23 19:45:53.281 ServerApp] 200 GET /jhub/user/user/api/contents?content=1&hash=0&1745437553032 (user@172.16.0.1) 167.01ms
[I 2025-04-23 19:45:53.512 ServerApp] 200 GET /jhub/user/user/lab/api/workspaces/default?1745437553429 (user@172.16.0.1) 1.56ms
[I 2025-04-23 19:45:53.587 LabApp] `sys_prefix` level settings are read-only, using `user` level for migration to `lockedExtensions`
[I 2025-04-23 19:45:53.590 LabApp] Build is up to date
[I 2025-04-23 19:45:53.591 ServerApp] 200 GET /jhub/user/user/lab/api/build?1745437552951 (user@172.16.0.1) 558.11ms
[I 2025-04-23 19:45:53.763 ServerApp] 200 GET /jhub/user/user/lab/api/settings?ids_only=true&1745437553665 (user@172.16.0.1) 15.82ms
[I 2025-04-23 19:45:53.851 ServerApp] 200 GET /jhub/user/user/lab/api/update?1745437553667 (user@172.16.0.1) 86.77ms
[I 2025-04-23 19:45:54.700 ServerApp] 204 PUT /jhub/user/user/lab/api/workspaces/default?1745437554605 (user@172.16.0.1) 3.32ms
[I 2025-04-23 19:45:54.825 ServerApp] 200 GET /jhub/user/user/api/contents?content=1&hash=0&1745437554630 (user@172.16.0.1) 105.24ms
[I 2025-04-23 19:45:55.439 ServerApp] 204 PUT /jhub/user/user/lab/api/workspaces/default?1745437555336 (user@172.16.0.1) 3.45ms
[I 2025-04-23 19:45:58.009 ServerApp] 302 GET /jhub/user/user/rstudio/ -> //server.edu/auth-sign-in?appUri=%2F (user@172.16.0.1) 1390.14ms
[I 2025-04-23 19:46:03.489 ServerApp] 200 GET /jhub/user/user/api/terminals?1745437563395 (user@172.16.0.1) 2.02ms
[I 2025-04-23 19:46:03.499 ServerApp] 200 GET /jhub/user/user/api/sessions?1745437563396 (user@172.16.0.1) 1.94ms
[I 2025-04-23 19:46:04.503 ServerApp] 200 GET /jhub/user/user/api/kernels?1745437564407 (user@172.16.0.1) 1.76ms
[I 2025-04-23 19:46:04.518 ServerApp] 200 GET /jhub/user/user/lab/api/workspaces?1745437564409 (user@172.16.0.1) 12.16ms
[I 2025-04-23 19:46:05.640 ServerApp] 200 GET /jhub/user/user/api/contents?content=1&hash=0&1745437565454 (user@172.16.0.1) 100.64ms
[I 2025-04-23 19:46:54.697 ServerApp] 200 GET /jhub/user/user/api/kernelspecs?1745437614606 (user@172.16.0.1) 4.92ms
[I 2025-04-23 19:46:54.700 ServerApp] 200 GET /jhub/user/user/api/me?1745437614606 (user@172.16.0.1) 1.68ms
[I 2025-04-23 19:49:06.865 ServerApp] 101 GET /jhub/user/user/api/events/subscribe (user@172.16.0.1) 2.03ms
[I 2025-04-23 19:49:45.343 ServerApp] 302 GET /jhub/user/user/rstudio/auth-sign-in?appUri=%2F -> //server.edu/jhub/user/user/rstudio/ (user@172.16.0.1) 12.02ms
[I 2025-04-23 19:49:45.357 ServerApp] 200 GET /jhub/user/user/rstudio/ (user@172.16.0.1) 5.93ms
````Preformatted text`