Regular issues with vdi_jupyter.py

[rmholmes]

I regularly run into issues with the vdi_jupyter.py script. I usually have to run it quite a few times to actually connect properly. Below are two example errors that I sometimes run into. If anyone has any suggests on how to fix/get around these (or whether they are normal/expected), I'd love to hear them.

ryan@RYANSURF6:/mnt/c/Users/holme/Cloud/Research/Utilities/python/nci_scripts$ ./vdi_jupyter.py
[10:01:30 jupyter_vdi.py] Using config file: /home/ryan/cosima_cookbook.conf
[10:01:30 jupyter_vdi.py] Checking SSH keys to VDI are configured...
password:
[10:01:36 jupyter_vdi.py] SSH keys configured OK
[10:01:36 jupyter_vdi.py] Determine if VDI session is already running...
[10:01:37 jupyter_vdi.py] Time remaining: 167:55:31
[10:01:37 jupyter_vdi.py] Determine jobid for VDI session...47707
[10:01:37 jupyter_vdi.py] Get exechost for VDI session...
[10:01:38 jupyter_vdi.py] exechost: vdi-n1.nci.org.au
[10:01:38 jupyter_vdi.py] Running Jupyter on VDI...
[10:01:38 jupyter_vdi.py] Waiting for Jupyter to start...
Traceback (most recent call last):
  File "./vdi_jupyter.py", line 267, in <module>
    main_argv()
  File "./vdi_jupyter.py", line 263, in main_argv
    main(args)
  File "./vdi_jupyter.py", line 239, in main
    ret = s.expect('http://\S*:(?P<jupyterport>\d+)/\?token=(?P<token>[a-zA-Z0-9]+)')
  File "/home/ryan/.local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 344, in expect
    timeout, searchwindowsize, async_)
  File "/home/ryan/.local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 372, in expect_list
    return exp.expect_loop(timeout)
  File "/home/ryan/.local/lib/python2.7/site-packages/pexpect/expect.py", line 181, in expect_loop
    return self.timeout(e)
  File "/home/ryan/.local/lib/python2.7/site-packages/pexpect/expect.py", line 144, in timeout
    raise exc
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x7f08347380d0>
command: /usr/bin/ssh
args: ['/usr/bin/ssh', '-x', '-l', 'rmh561', 'vdi-n1.nci.org.au', '-t', 'bash -l -c "module use /g/data3/hh5/public/modules                  && module load conda/analysis3                && jupyter notebook --no-browser --port 8889"']
buffer (last 100 chars): "rmh561@vdi-n1.nci.org.au's password: "
before (last 100 chars): "rmh561@vdi-n1.nci.org.au's password: "
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 101
child_fd: 5
closed: False
timeout: 30
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: None
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: 0.05
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
    0: re.compile('http://\\S*:(?P<jupyterport>\\d+)/\\?token=(?P<token>[a-zA-Z0-9]+)')
ryan@RYANSURF6:/mnt/c/Users/holme/Cloud/Research/Utilities/python/nci_scripts$ ./vdi_jupyter.py
[10:02:12 jupyter_vdi.py] Using config file: /home/ryan/cosima_cookbook.conf
[10:02:12 jupyter_vdi.py] Checking SSH keys to VDI are configured...
password:
[10:02:16 jupyter_vdi.py] SSH keys configured OK
[10:02:16 jupyter_vdi.py] Determine if VDI session is already running...
[10:02:17 jupyter_vdi.py] Time remaining: 167:54:51
[10:02:17 jupyter_vdi.py] Determine jobid for VDI session...47707
[10:02:17 jupyter_vdi.py] Get exechost for VDI session...
Traceback (most recent call last):
  File "./vdi_jupyter.py", line 267, in <module>
    main_argv()
  File "./vdi_jupyter.py", line 263, in main_argv
    main(args)
  File "./vdi_jupyter.py", line 219, in main
    params.update(m.groupdict())
AttributeError: 'NoneType' object has no attribute 'groupdict'

[ScottWales]

Is this a recent thing or a long term problem? NCI are having problems with the /g/data3 filesystem, where conda lives, over the weekend.

[rmholmes]

It seems particularly bad now. But yes I've had the timeout issue quite a few times before last week.

[rmholmes]

It looks like I can't connect at all the morning, so you might be right it may be their issue with /g/data3

[vsilvafelipe]

Hi!
I think I am dealing with a different problem in /g/data3/

It's been almost a week that whenever I try to open or create a notebook in the browser via vdi_jupyter I get this error:
500 : Internal Server Error

Among many issues in my terminal, I found:
PermissionError: [Errno 13] Permission denied: '/g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.01/share/jupyter/nbconvert/templates/html/conf.json'

Right after the message above, it is written in the terminal:

[E 00:42:01.435 NotebookApp] {
      "Host": "localhost:8889",
      "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
      "Accept-Language": "en-US,en;q=0.5",
      "Accept-Encoding": "gzip, deflate",
      "Connection": "keep-alive",
      "Referer": "http://localhost:8889/tree",
      "Cookie": "username-localhost-8888=\"2|1:0|10:1610995459|23:username-localhost-8888|44:MzRiNGM1MTRiNGMxNDg0ZTgyMTU3MmUxZWZlMGIwZWI=|e53ad3b9561fe5d6535d3afae67ba712357b518cb0fdf42354f1b155f753357a\"; username-localhost-8889=\"2|1:0|10:1611063016|23:username-localhost-8889|44:OWY1NTUwODQwMDU2NDc0MjhhMzNkMzU3MTAzMWQyYTg=|30b2781904fe52d4a480b1c112c4d38773a1131141e3d114315d3f6fbc4766e2\"; _xsrf=2|a38b2e64|5435a5b6df14cb11fee498a0f67d8326|1610114075; username-localhost-8890=\"2|1:0|10:1610989147|23:username-localhost-8890|44:MjAxMTNkMGRiMzA2NDYyZmE1ZjQ2MzkxNGU2MWI4YTc=|7f5fe81be1b6287082f5e1b5dc13c160aeb0c295c42e33bd0ec018395886e355\"",
      "Upgrade-Insecure-Requests": "1"
    }
[E 00:42:01.435 NotebookApp] 500 GET /notebooks/Untitled.ipynb (::1) 1066.800000ms referer=http://localhost:8889/tree

I opened my jupyter notebook in the virtual desktop via strudel and it's working well there.
Has anyone ever seen anything similar? I would be glad with any support.

[ccarouge]

To use vdi_jupyter, you need to be a member of the hh5 project at NCI. Please ask for membership via my.nci.org.au

[vsilvafelipe]

To use vdi_jupyter, you need to be a member of the hh5 project at NCI. Please ask for membership via my.nci.org.au

Thank you, @ccarouge . Since 10/20, it worked without being a hh5 member. I have just asked for membership.