|Posted on Thursday, June 13, 2013 - 2:18 am: |
Working through installation of 12.1.004e - server registration works using a test application server. However when attempting to register the real server (where passwordless ssh connection already works fine), registration fails. Subsequent manual ssh to the server now fails, where it had previously worked. I believe this is a firewall issue; the registration seems to involve many rapid ssh connections (I could see 27 in about 1 second) which I guess the firewall regards as a DOS attack.
Firstly, is this behaviour (many rapid ssh connections during server registration) as expected?
Secondly, if it is as expected then can it be changed? I'd rather have a longer registration process that is more likely to succeed (say 1 connection per second) rather than 27 connections per second which is bound to fail.
Thanks for any insight ..
Post Number: 355
|Posted on Thursday, June 13, 2013 - 10:07 am: |
The registration process DOES involve many SSH connections, as there are numerous file copies / command that need to be run (via SSH) to configure the remote server. However I count only ~15 not 27.
This should be less of an issue once the server is added. In the mean time, you might add a "sleep(1);" statement to the "remote_run_command" and "remote_copy_to" subroutines in "servercontrol.cgi" to effectively throttle the rate of SSH commands being issued.
REMOVE THESE STATEMENTS after registering the server(s).
Note that you are likely to have a MUCH more stable connection if you are not running a connection through a hardware firewall (e.g. to a server in the same building, etc.) because WebMO does require a stable connection. In my experience, using truly remote servers yields intermittent connection problems which can cause jobs to fail. (This is not a frequent issue, but does happen occasionally).
|Posted on Thursday, June 13, 2013 - 9:53 pm: |
It turns out (from other testing) that I needed a timeout of 20 seconds per ssh connection. I added sleep(20) in those subroutines you mentioned, however I'm not positively sure it has worked. On the one hand, the browser (after clicking the Add button) eventually came up with a "Gateway Timeout" error message i.e. looked like failure. However the server now appears in the "Edit servers" tab of the remote server manager i.e. looks like success.
I also see that a webmo directory has appeared on the server (with quite a bit in there); also 2 perl scripts (findfiles.pl and pscheck.pl), which had previously been there, presumably after earlier registration attempts failed to complete, are now gone - suggesting some cleanup has happened.
Is there any way to confirm that server registration has really completed successfully?
Post Number: 357
|Posted on Thursday, June 13, 2013 - 10:09 pm: |
The fact that the findfiles.pl and pscheck.pl are still there suggest that the addition of the server did NOT complete. The "Gateway Timeout" means that the web server terminated the script prior to completion since it took too long to complete (due to sleep 20!)
I don't have a good workaround for this. In principle this could be re-written to use fewer SSH calls, but I have never encountered this situation before!
If you add the server from a local network connection, it will likely work fine once it is added
|Posted on Friday, June 14, 2013 - 12:31 am: |
I meant that since adding sleep(20), despite the Gateway Timeout, those perl scripts have now been cleaned up (they were not cleaned up previously), so I thought things were looking good - I was just looking for some other confirmation.
Since then, looking through servercontrol.cgi I see that removing pscheck.pl is the last thing done in the add_server function so I'm regarding the absence of pscheck.pl as even more of a positive sign.
Better still, I've now changed the webserver configuration, adding "Timeout 600" to http.conf (I believe the built in default was 60), so now there is no Gateway Timeout error any more. I can monitor the application server and see the various files & directories as they're created (and eventually cleaned up).
Its not ideal but seems to work so far ..
Thanks for your help,
Post Number: 358
|Posted on Friday, June 14, 2013 - 10:06 am: |
This sounds good. Depending on the threshold for number of connections before the firewall freaks out, this may (or may not) cause problems when submitting jobs to remote servers. That ALSO requires multiple SSH connections, although far fewer then adding the server.