Pragma Systems Technical Forum

Client reports "Unexpected end of sftp stream" - and a question about logging

https://forums.pragmasys.com/Topic327.aspx

By crrivers - 10/22/2015 5:05:05 AM

Hello,

We are running Pragma 5.0 Build 9 Revision 3380.

We have a user reporting frequent "Unexpected end of sftp stream" errors.  They are using a Linux host (using a java-based application using SFTPv3Client) which establishes multiple connections to transmit a large number of very small files. (each under 1kb)  We do not have "user session limit" enabled.

Aside from the default connection and troubleshooting logs, I've enabled the sftp debugging logs via registry modification.  Sever operation logging is set to 6 and "log output from server" and "log input from client" are checked.  

There are other users transmitting/picking up files from the same Linux host.  (usually no more than 2 or 3)  There are other users from other hosts connecting in around the same time.  (usually no more than 5 to 10 and typically are short lived)

Log-wise, I am not finding anything tell tale yet.  (still digging through the verbosity)

Have you encountered the "unexpected end of sftp stream" error?  Any guidance or areas to focus specifically that may help troubleshoot?

Thank you,

Chris
By bethredd - 10/22/2015 8:24:48 AM

Hello,

Thank you for your thorough troubleshooting. 

I recommend that you turn off "log output from server" and "log input from client" as they are not helpful in troubleshooting sftp issues and they will slow down performance.

Please check the Event Log under Applications and Services Logs -> Pragma Fortress SSH Server for any error events from the SFTP service. Also check the Application Event Log for any Application Manager events that involve sftp-server.exe, sshd.exe, or sshd64.exe.

Any error event will give you the best information as to what could be occurring.
By crrivers - 10/28/2015 7:58:07 AM

Hi Beth! 


I've dug into this a bit and made the recommended changes for "log output from server" and "log input from client".

We got another report of this issue this morning and here's what I am observing.

On the Application event log (not the Pragma specific one - it was overwritten by the time I got to the issue):

Faulting application name: sshd64.exe, version: 5.0.9.3380, time stamp: 0x55dc8ae5
Faulting module name: sshd64.exe, version: 5.0.9.3380, time stamp: 0x55dc8ae5
Exception code: 0xc0000005
Fault offset: 0x000000000001f140
Faulting process id: 0x10e4
Faulting application start time: 0x01d11156f85cbcf9
Faulting application path: E:\pragma\Fortress\sshd64.exe
Faulting module path: E:\pragma\Fortress\sshd64.exe
Report Id: 3650cf79-7d4a-11e5-80c8-005056a9023a
Faulting package full name: 
Faulting package-relative application ID: 

AND

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: sshd64.exe
P2: 5.0.9.3380
P3: 55dc8ae5
P4: sshd64.exe
P5: 5.0.9.3380
P6: 55dc8ae5
P7: c0000005
P8: 000000000001f140
P9: 
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_sshd64.exe_5d7457ad999c18f342eb8a2aa38d9659d1ba7e8_25e5a5c9_385b5a95

Analysis symbol: 
Rechecking for solution: 0
Report Id: 3650cf79-7d4a-11e5-80c8-005056a9023a
Report Status: 0
Hashed bucket: 



From the other Pragma-based logs, here's what I am seeing at the time this last occurred:

In Pragma Connection logs I see a log with no content inside.  

In troubleshooting logs, I see a correlating log there, but it's very verbose and may need some assistance in reading through it.  Have a decoder ring? :-)

In the debug sftp logs, I see where the user has logged on, is (scriptomagically) reading directories and trying to download some simple text files, then logs off.  The downloads fail.  (Just says "Transfer FAILED. X bytes transferred.")  Permissions-wise (NTFS related) all checks out - the user has modify rights where they're requesting directory read or file read access.
By crrivers - 11/4/2015 9:42:36 AM

Beth,
It's been about a week and I thought I'd check back in to see if there's any feedback here.  If I need to reach out through more formal support channels I can do so.  We continue to have the same set of errors. 

Thank you.
By bethredd - 11/4/2015 10:09:40 AM

Sorry for the delay. Unfortunately, your forum notification post was lost. I will try to be more stringent with monitoring it. You can always email support@pragmasys.com for a quicker response.

I wonder if you have a virus scanner that might be monitoring our application. Sometime virus scanners will interfere with network traffic and could cause the application fault you are seeing. At the point of file transfer, the only thing the sshd process is doing is sending packets between the sftp client and server. This is what leads me to believe it might be a 3rd party software monitoring the transfer as well.

Please send me your server operation log to the support@pragmasys.com email as well, so that I can view the output with my decoder ring.