devel and doc builds are deadlocking on the flock

[tfoote]

udevd appears to be locking the file too. After a few retries repeatedly failing it has stopped failing.

root@host03:~# strace -p 9201
Process 9201 attached - interrupt to quit
flock(3, LOCK_EX^C <unfinished ...>
Process 9201 detached
root@host03:~# readlink /proc/9201/fd/3
/var/cache/pbuilder/jenkins_tools.trusty.amd64-0.4.1.updatelock
root@host03:~# lsof /var/cache/pbuilder/jenkins_tools.trusty.amd64-0.4.1.updatelock
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
flock    9201 root    3u   REG  253,3        0 1408567 /var/cache/pbuilder/jenkins_tools.trusty.amd64-0.4.1.updatelock
udevd   25361 root    3r   REG  253,3        0 1408567 /var/cache/pbuilder/jenkins_tools.trusty.amd64-0.4.1.updatelock

[tfoote]

Seen again today @dirk-thomas also on doc jobs.

[tfoote]

Killing udevd lets the job continue

[dirk-thomas]

It hangs every time it has to recreate the pbuilder image.

[dirk-thomas]

Either:

• ssh into a single slave and killall udevd to unblock the jobs
• use a bash function to make it less painful: https://gist.github.com/dirk-thomas/105b14e9fa92ca5efaba
• use https://github.com/wg-buildfarm/storm_automation/blob/master/storm/slave_cssh to invoke commands on all slaves at the same time

[dirk-thomas]

May be instead of using flock directly we might want to give it shot to using something like this: libfile-fcntllock-perl

[tfoote]

I've also been looking at withlock https://code.google.com/p/withlock/