thingspeak hangs

[rpj]

The thingspeak tool seems to hang very occasionally after running fine for long periods (~days), and seems to do so somewhere in the PSubscriber code for listening to redis. That's about all I know now.

[rpj]

It has to have something to do with how thingspeak is using the class, because other uses of it (oled-display, sql-sink) can run for many days without trouble (sql-sink is just about to write its 4 millionth row on charlie without a single error, for example...)

[rpj]

Might need a way (TS has examples) to detect a lack of data ingestion and send an alert, so I can at least examine the failure mode while live...

[rpj]

Actually, I now believe this was likely caused by the lack of a timeout argument to urllib2.urlopen(), causing that call to hang indefinitely. Have a few processes with instrumentation to prove this running right now, should find out for sure soon enough...

[rpj]

Ah ha, I do believe I've caught it! Managed to get one of the processes to hit 10 timeouts in a row over the course of about 5 minutes, but catching those instead of letting the process die allowed it to eventually recover just fine. Looping around those timeout and trying a couple more times immediately (rather than just bailing on that send and waiting until the next send point rolls around) might make things even better. Sweet! Finally.