Pragma Systems Technical Forum

sftp connect to fortress ssh server failed with "connection refused" when concurrent connecting

https://forums.pragmasys.com/Topic534.aspx

By alex qiu - 11/19/2020 5:19:56 AM

Hi team,

We now meet an issue when remote client sftp connect to our windows ssh server concurrently, occasionally the client get "Connection Refused" error.
We can for sure the inetd service is up-and-running, port is correct the same. sftp client is connecting to same FQDN/user/port. There shall not be firewall block in between. The current max connection in ssh server is set to 2000 and in testing env it's for sure much less than that number.

Would like to seek your advise to see how to further investigate the root cause for this? (As the connection refused error is before the connection established.)  Appreciated in advance.

Attached couple of screens for failure and normal case client debug log (-v), and one screenshot for the TCP monitoring, it's windows ssh server send RST ack to reset the conn causing connection refused.

By Technical Support Group (TSG) - 11/19/2020 11:27:37 AM

alex qiu - 11/19/2020 11:19:56 AM
Hi team,

We now meet an issue when remote client sftp connect to our windows ssh server concurrently, occasionally the client get "Connection Refused" error.
We can for sure the inetd service is up-and-running, port is correct the same. sftp client is connecting to same FQDN/user/port. There shall not be firewall block in between. The current max connection in ssh server is set to 2000 and in testing env it's for sure much less than that number.

Would like to seek your advise to see how to further investigate the root cause for this? (As the connection refused error is before the connection established.)  Appreciated in advance.

Attached couple of screens for failure and normal case client debug log (-v), and one screenshot for the TCP monitoring, it's windows ssh server send RST ack to reset the conn causing connection refused.


Hello,

Please check our event log, accessible from the Logging page of the Local Server Configuration program, for an error event at the time of the failed connection. The details of the event should tell you why the connection is failing. If you do not have any events, go to the InetD page and turn on "Record attempted connections..." this will create a warning event when the connection is made to any port monitored by InetD. If you get the expected events, then turn on Server Operation Logging on the Logging page. Set a level of 6 and configure a directory. Near the end of the log file there should be a message telling you why the server is closing.
By alex qiu - 12/14/2020 2:45:41 AM

Technical Support Group (TSG) - 11/19/2020 5:27:37 PM
alex qiu - 11/19/2020 11:19:56 AM
Hi team,

We now meet an issue when remote client sftp connect to our windows ssh server concurrently, occasionally the client get "Connection Refused" error.
We can for sure the inetd service is up-and-running, port is correct the same. sftp client is connecting to same FQDN/user/port. There shall not be firewall block in between. The current max connection in ssh server is set to 2000 and in testing env it's for sure much less than that number.

Would like to seek your advise to see how to further investigate the root cause for this? (As the connection refused error is before the connection established.)  Appreciated in advance.

Attached couple of screens for failure and normal case client debug log (-v), and one screenshot for the TCP monitoring, it's windows ssh server send RST ack to reset the conn causing connection refused.


Hello,

Please check our event log, accessible from the Logging page of the Local Server Configuration program, for an error event at the time of the failed connection. The details of the event should tell you why the connection is failing. If you do not have any events, go to the InetD page and turn on "Record attempted connections..." this will create a warning event when the connection is made to any port monitored by InetD. If you get the expected events, then turn on Server Operation Logging on the Logging page. Set a level of 6 and configure a directory. Near the end of the log file there should be a message telling you why the server is closing.

Hi team,

Thanks for the information, we have enabled all these and reproduced the case, while for those "Connection Refused" connections, there is no any logs showing in either log file or windows event logs.

We have also followed the FAQ advise https://www.pragmasys.com/ssh-server/faq#SmallSession to update inetD desktop count and the windows registry value, this doesn't resolve the "Connection Refused" issue.

All of the "Connection Refused" error is having the same network packet pattern,
SYN -> RST, ACK, it seems before having any handshake it's refused.

And we have retried the same test cases with another open source sftp server Mina, no more "Connection Refused" issue. So wondering any config on Fortress can resolve this. Hence if  you can shed some lights on this one?


By Technical Support Group (TSG) - 12/17/2020 10:45:01 AM

alex qiu - 12/14/2020 8:45:41 AM
Technical Support Group (TSG) - 11/19/2020 5:27:37 PM
alex qiu - 11/19/2020 11:19:56 AM
Hi team,

We now meet an issue when remote client sftp connect to our windows ssh server concurrently, occasionally the client get "Connection Refused" error.
We can for sure the inetd service is up-and-running, port is correct the same. sftp client is connecting to same FQDN/user/port. There shall not be firewall block in between. The current max connection in ssh server is set to 2000 and in testing env it's for sure much less than that number.

Would like to seek your advise to see how to further investigate the root cause for this? (As the connection refused error is before the connection established.)  Appreciated in advance.

Attached couple of screens for failure and normal case client debug log (-v), and one screenshot for the TCP monitoring, it's windows ssh server send RST ack to reset the conn causing connection refused.


Hello,

Please check our event log, accessible from the Logging page of the Local Server Configuration program, for an error event at the time of the failed connection. The details of the event should tell you why the connection is failing. If you do not have any events, go to the InetD page and turn on "Record attempted connections..." this will create a warning event when the connection is made to any port monitored by InetD. If you get the expected events, then turn on Server Operation Logging on the Logging page. Set a level of 6 and configure a directory. Near the end of the log file there should be a message telling you why the server is closing.

Hi team,

Thanks for the information, we have enabled all these and reproduced the case, while for those "Connection Refused" connections, there is no any logs showing in either log file or windows event logs.

We have also followed the FAQ advise https://www.pragmasys.com/ssh-server/faq#SmallSession to update inetD desktop count and the windows registry value, this doesn't resolve the "Connection Refused" issue.

All of the "Connection Refused" error is having the same network packet pattern,
SYN -> RST, ACK, it seems before having any handshake it's refused.

And we have retried the same test cases with another open source sftp server Mina, no more "Connection Refused" issue. So wondering any config on Fortress can resolve this. Hence if  you can shed some lights on this one?



Hi Alex,

Sorry for the delay in getting back to you.

It appears you might be hitting an issue with the key exchange algorithms. I've seen this when the client is upgraded to require the rsa-sha2 key exchange algorithms. Please update to the latest build at www.pragmasys.com, which includes support for those algorithms.
By alex qiu - 12/28/2020 2:54:51 AM

Technical Support Group (TSG) - 12/17/2020 4:45:01 PM
alex qiu - 12/14/2020 8:45:41 AM
Technical Support Group (TSG) - 11/19/2020 5:27:37 PM
alex qiu - 11/19/2020 11:19:56 AM
Hi team,

We now meet an issue when remote client sftp connect to our windows ssh server concurrently, occasionally the client get "Connection Refused" error.
We can for sure the inetd service is up-and-running, port is correct the same. sftp client is connecting to same FQDN/user/port. There shall not be firewall block in between. The current max connection in ssh server is set to 2000 and in testing env it's for sure much less than that number.

Would like to seek your advise to see how to further investigate the root cause for this? (As the connection refused error is before the connection established.)  Appreciated in advance.

Attached couple of screens for failure and normal case client debug log (-v), and one screenshot for the TCP monitoring, it's windows ssh server send RST ack to reset the conn causing connection refused.


Hello,

Please check our event log, accessible from the Logging page of the Local Server Configuration program, for an error event at the time of the failed connection. The details of the event should tell you why the connection is failing. If you do not have any events, go to the InetD page and turn on "Record attempted connections..." this will create a warning event when the connection is made to any port monitored by InetD. If you get the expected events, then turn on Server Operation Logging on the Logging page. Set a level of 6 and configure a directory. Near the end of the log file there should be a message telling you why the server is closing.

Hi team,

Thanks for the information, we have enabled all these and reproduced the case, while for those "Connection Refused" connections, there is no any logs showing in either log file or windows event logs.

We have also followed the FAQ advise https://www.pragmasys.com/ssh-server/faq#SmallSession to update inetD desktop count and the windows registry value, this doesn't resolve the "Connection Refused" issue.

All of the "Connection Refused" error is having the same network packet pattern,
SYN -> RST, ACK, it seems before having any handshake it's refused.

And we have retried the same test cases with another open source sftp server Mina, no more "Connection Refused" issue. So wondering any config on Fortress can resolve this. Hence if  you can shed some lights on this one?



Hi Alex,

Sorry for the delay in getting back to you.

It appears you might be hitting an issue with the key exchange algorithms. I've seen this when the client is upgraded to require the rsa-sha2 key exchange algorithms. Please update to the latest build at www.pragmasys.com, which includes support for those algorithms.

Hi, Thanks a lot for advice, we have downloaded the latest version 5.0 Build10 Revision: 2122, while the previous installed one is Version: 5.0 Build 10 Revision 1563.
The upgrade approach it's by double clicking the exe then we choose upgrade fortress. (instead of uninstall/install).

Unfortunately post the upgrade we tested the same scenario and "Connection Refused" issue still existed. Am looking forward to seeing your further advise and thanks a million in advance
By Technical Support Group (TSG) - 12/28/2020 9:19:16 AM

alex qiu - 12/28/2020 8:54:51 AM
Technical Support Group (TSG) - 12/17/2020 4:45:01 PM
alex qiu - 12/14/2020 8:45:41 AM
Technical Support Group (TSG) - 11/19/2020 5:27:37 PM
alex qiu - 11/19/2020 11:19:56 AM
Hi team,

We now meet an issue when remote client sftp connect to our windows ssh server concurrently, occasionally the client get "Connection Refused" error.
We can for sure the inetd service is up-and-running, port is correct the same. sftp client is connecting to same FQDN/user/port. There shall not be firewall block in between. The current max connection in ssh server is set to 2000 and in testing env it's for sure much less than that number.

Would like to seek your advise to see how to further investigate the root cause for this? (As the connection refused error is before the connection established.)  Appreciated in advance.

Attached couple of screens for failure and normal case client debug log (-v), and one screenshot for the TCP monitoring, it's windows ssh server send RST ack to reset the conn causing connection refused.


Hello,

Please check our event log, accessible from the Logging page of the Local Server Configuration program, for an error event at the time of the failed connection. The details of the event should tell you why the connection is failing. If you do not have any events, go to the InetD page and turn on "Record attempted connections..." this will create a warning event when the connection is made to any port monitored by InetD. If you get the expected events, then turn on Server Operation Logging on the Logging page. Set a level of 6 and configure a directory. Near the end of the log file there should be a message telling you why the server is closing.

Hi team,

Thanks for the information, we have enabled all these and reproduced the case, while for those "Connection Refused" connections, there is no any logs showing in either log file or windows event logs.

We have also followed the FAQ advise https://www.pragmasys.com/ssh-server/faq#SmallSession to update inetD desktop count and the windows registry value, this doesn't resolve the "Connection Refused" issue.

All of the "Connection Refused" error is having the same network packet pattern,
SYN -> RST, ACK, it seems before having any handshake it's refused.

And we have retried the same test cases with another open source sftp server Mina, no more "Connection Refused" issue. So wondering any config on Fortress can resolve this. Hence if  you can shed some lights on this one?



Hi Alex,

Sorry for the delay in getting back to you.

It appears you might be hitting an issue with the key exchange algorithms. I've seen this when the client is upgraded to require the rsa-sha2 key exchange algorithms. Please update to the latest build at www.pragmasys.com, which includes support for those algorithms.

Hi, Thanks a lot for advice, we have downloaded the latest version 5.0 Build10 Revision: 2122, while the previous installed one is Version: 5.0 Build 10 Revision 1563.
The upgrade approach it's by double clicking the exe then we choose upgrade fortress. (instead of uninstall/install).

Unfortunately post the upgrade we tested the same scenario and "Connection Refused" issue still existed. Am looking forward to seeing your further advise and thanks a million in advance

I'm sorry that the upgrade did not fix the issue. Can you confirm that you are NOT getting any events from the InetD service stating that it accepted a connection from the client? If not, then it is a network issue and outside of our program. If you are then, we you should get some information from the Server Operation Logging. Another thing to check is whether the sshd process (could be either sshd.exe or sshd64.exe) is launching. If not, you can turn on process auditing to see why the process is failing to start.