|
alex qiu
|
|
|
Group: Forum Members
Posts: 17,
Visits: 85
|
Hi team,
We are using "Pragma Fortress 5.0.10.1563" version and now in production it occasionally hang during file transfer with external sftp host.
In the software program it will run the batch sftp commands file to fetch multiple files, sample content is listed below, during the incident, we found out that the sftp command hang at the BOLD one (with underline).
We can see the 123456.tmp file is there in C:\tmp but the content is incomplete. While the second file 123457.tmp file existed but with 0 byte length.
We have checked with network team and all is fine at that moment, we are using proxy to go to outside external sftp host.
For other scenario, some times none of the tmp files there created under C:\tmp and we can see in windows task manager "Process" tab, that a sftp process is running there running the batch sftp commands file, seemingly hang.
Am wondering if you can shed some light on us e.g. any advice for workaround or strategic solution on such? BTW, to avoid posting actual business info, am all using dummy file names.
Batch sftp commands file content :
ls /reports/xxxxx.txt
get "/reports/xxxxx.txt" "C:\tmp\123456.tmp"
ls /reports/xxxxx1.txt
get "/reports/xxxxx1.txt" "C:\tmp\123457.tmp"
|
|
|
|
|
Technical Support Group...
|
|
|
Group: Moderators
Posts: 136,
Visits: 639
|
+xHi team, We are using "Pragma Fortress 5.0.10.1563" version and now in production it occasionally hang during file transfer with external sftp host. In the software program it will run the batch sftp commands file to fetch multiple files, sample content is listed below, during the incident, we found out that the sftp command hang at the BOLD one (with underline). We can see the 123456.tmp file is there in C:\tmp but the content is incomplete. While the second file 123457.tmp file existed but with 0 byte length. We have checked with network team and all is fine at that moment, we are using proxy to go to outside external sftp host. For other scenario, some times none of the tmp files there created under C:\tmp and we can see in windows task manager "Process" tab, that a sftp process is running there running the batch sftp commands file, seemingly hang. Am wondering if you can shed some light on us e.g. any advice for workaround or strategic solution on such? BTW, to avoid posting actual business info, am all using dummy file names. Batch sftp commands file content : ls /reports/xxxxx.txt get "/reports/xxxxx.txt" "C:\tmp\123456.tmp"ls /reports/xxxxx1.txt get "/reports/xxxxx1.txt" "C:\tmp\123457.tmp" Hello, I need some more information on your setup. Are you using both our server and client for the download? Do you have any issues if you run the commands from an interactive sftp session? Are you running the batch file manually or is it a scheduled task? Have you tried logging the output for any reported errors. You can log by adding the -L option to the sftp command.
Pragma Systems Technical Support 13809 Research Blvd, #675 Austin, TX 78750 http://www.pragmasys.com
|
|
|
|
|
alex qiu
|
|
|
Group: Forum Members
Posts: 17,
Visits: 85
|
+x+xHi team, We are using "Pragma Fortress 5.0.10.1563" version and now in production it occasionally hang during file transfer with external sftp host. In the software program it will run the batch sftp commands file to fetch multiple files, sample content is listed below, during the incident, we found out that the sftp command hang at the BOLD one (with underline). We can see the 123456.tmp file is there in C:\tmp but the content is incomplete. While the second file 123457.tmp file existed but with 0 byte length. We have checked with network team and all is fine at that moment, we are using proxy to go to outside external sftp host. For other scenario, some times none of the tmp files there created under C:\tmp and we can see in windows task manager "Process" tab, that a sftp process is running there running the batch sftp commands file, seemingly hang. Am wondering if you can shed some light on us e.g. any advice for workaround or strategic solution on such? BTW, to avoid posting actual business info, am all using dummy file names. Batch sftp commands file content : ls /reports/xxxxx.txt get "/reports/xxxxx.txt" "C:\tmp\123456.tmp"ls /reports/xxxxx1.txt get "/reports/xxxxx1.txt" "C:\tmp\123457.tmp" Hello, I need some more information on your setup. Are you using both our server and client for the download? Do you have any issues if you run the commands from an interactive sftp session? Are you running the batch file manually or is it a scheduled task? Have you tried logging the output for any reported errors. You can log by adding the -L option to the sftp command. Thanks for the prompt reply. We have both Pragma fortress server and client there installed in the machine, while during our Java program execution, it shall use sftp client only I believe. We don't have the issue when I manually execute commands in interactive sftp session. We run sftp batch file in a Java program which execute regularly with CRON expression config. As it's for production, haven't enabled the -L option, but we will try this in testing env to see if possible to reproduce. Also one thing supplement here, in last couple of months e.g. Jan to May, such case happened once, but recent two weeks twice it occurred.
|
|
|
|
|
alex qiu
|
|
|
Group: Forum Members
Posts: 17,
Visits: 85
|
+x+x+xHi team, We are using "Pragma Fortress 5.0.10.1563" version and now in production it occasionally hang during file transfer with external sftp host. In the software program it will run the batch sftp commands file to fetch multiple files, sample content is listed below, during the incident, we found out that the sftp command hang at the BOLD one (with underline). We can see the 123456.tmp file is there in C:\tmp but the content is incomplete. While the second file 123457.tmp file existed but with 0 byte length. We have checked with network team and all is fine at that moment, we are using proxy to go to outside external sftp host. For other scenario, some times none of the tmp files there created under C:\tmp and we can see in windows task manager "Process" tab, that a sftp process is running there running the batch sftp commands file, seemingly hang. Am wondering if you can shed some light on us e.g. any advice for workaround or strategic solution on such? BTW, to avoid posting actual business info, am all using dummy file names. Batch sftp commands file content : ls /reports/xxxxx.txt get "/reports/xxxxx.txt" "C:\tmp\123456.tmp"ls /reports/xxxxx1.txt get "/reports/xxxxx1.txt" "C:\tmp\123457.tmp" Hello, I need some more information on your setup. Are you using both our server and client for the download? Do you have any issues if you run the commands from an interactive sftp session? Are you running the batch file manually or is it a scheduled task? Have you tried logging the output for any reported errors. You can log by adding the -L option to the sftp command. Thanks for the prompt reply. We have both Pragma fortress server and client there installed in the machine, while during our Java program execution, it shall use sftp client only I believe. We don't have the issue when I manually execute commands in interactive sftp session. We run sftp batch file in a Java program which execute regularly with CRON expression config. As it's for production, haven't enabled the -L option, but we will try this in testing env to see if possible to reproduce. Also one thing supplement here, in last couple of months e.g. Jan to May, such case happened once, but recent two weeks twice it occurred. Few more update here, the latest monitoring in testing env it looks like that the sftp command will trigger another sub-process like below: #1 The sftp command sample: sftp -oUser=xxxxx1 -oConnectionTimeout=120s -oPort=22 -oIdentityfile2=blahblahblah external-sftp-host.com #2 The above sftp command will trigger a subprocess like below ssh ssh -o User="xxxxx1" -oConnectionTimeout="120s" -o Port="22" -o Identityfile2="blahblahblah" external-sftp-host.com -v -s oProtocol=2 sftp If it's true, any scenario that the ssh sub-process will get hang (e.g. wait for input signal) or "keep running" ... etc?
|
|
|
|
|
Technical Support Group...
|
|
|
Group: Moderators
Posts: 136,
Visits: 639
|
+x+x+x+xHi team, We are using "Pragma Fortress 5.0.10.1563" version and now in production it occasionally hang during file transfer with external sftp host. In the software program it will run the batch sftp commands file to fetch multiple files, sample content is listed below, during the incident, we found out that the sftp command hang at the BOLD one (with underline). We can see the 123456.tmp file is there in C:\tmp but the content is incomplete. While the second file 123457.tmp file existed but with 0 byte length. We have checked with network team and all is fine at that moment, we are using proxy to go to outside external sftp host. For other scenario, some times none of the tmp files there created under C:\tmp and we can see in windows task manager "Process" tab, that a sftp process is running there running the batch sftp commands file, seemingly hang. Am wondering if you can shed some light on us e.g. any advice for workaround or strategic solution on such? BTW, to avoid posting actual business info, am all using dummy file names. Batch sftp commands file content : ls /reports/xxxxx.txt get "/reports/xxxxx.txt" "C:\tmp\123456.tmp"ls /reports/xxxxx1.txt get "/reports/xxxxx1.txt" "C:\tmp\123457.tmp" Hello, I need some more information on your setup. Are you using both our server and client for the download? Do you have any issues if you run the commands from an interactive sftp session? Are you running the batch file manually or is it a scheduled task? Have you tried logging the output for any reported errors. You can log by adding the -L option to the sftp command. Thanks for the prompt reply. We have both Pragma fortress server and client there installed in the machine, while during our Java program execution, it shall use sftp client only I believe. We don't have the issue when I manually execute commands in interactive sftp session. We run sftp batch file in a Java program which execute regularly with CRON expression config. As it's for production, haven't enabled the -L option, but we will try this in testing env to see if possible to reproduce. Also one thing supplement here, in last couple of months e.g. Jan to May, such case happened once, but recent two weeks twice it occurred. Few more update here, the latest monitoring in testing env it looks like that the sftp command will trigger another sub-process like below: #1 The sftp command sample: sftp -oUser=xxxxx1 -oConnectionTimeout=120s -oPort=22 -oIdentityfile2=blahblahblah external-sftp-host.com #2 The above sftp command will trigger a subprocess like below ssh ssh -o User="xxxxx1" -oConnectionTimeout="120s" -o Port="22" -o Identityfile2="blahblahblah" external-sftp-host.com -v -s oProtocol=2 sftp If it's true, any scenario that the ssh sub-process will get hang (e.g. wait for input signal) or "keep running" ... etc? What you are seeing is the norm for the sftp protocol. The sftp client launches the ssh client which handles all of the authentication and encryption for the file transfer. If the ssh process closes, the sftp process should close as well. If the ssh process is running the sftp process should be running as well. If the "hang" is in during the file transfer it is most likely an issue with the sftp portion file manipulation and not the network or ssh portion of the process. Have you tried running the same command line you run in cron from the command line to try to duplicate? It might be a timing issue or possibly a user context issue of who is running the job, for example if you are running the cron job as system, you may not have access to any necessary configuration files that are used when the command is run as a particular user.
Pragma Systems Technical Support 13809 Research Blvd, #675 Austin, TX 78750 http://www.pragmasys.com
|
|
|
|
|
alex qiu
|
|
|
Group: Forum Members
Posts: 17,
Visits: 85
|
+x+x+x+x+xHi team, We are using "Pragma Fortress 5.0.10.1563" version and now in production it occasionally hang during file transfer with external sftp host. In the software program it will run the batch sftp commands file to fetch multiple files, sample content is listed below, during the incident, we found out that the sftp command hang at the BOLD one (with underline). We can see the 123456.tmp file is there in C:\tmp but the content is incomplete. While the second file 123457.tmp file existed but with 0 byte length. We have checked with network team and all is fine at that moment, we are using proxy to go to outside external sftp host. For other scenario, some times none of the tmp files there created under C:\tmp and we can see in windows task manager "Process" tab, that a sftp process is running there running the batch sftp commands file, seemingly hang. Am wondering if you can shed some light on us e.g. any advice for workaround or strategic solution on such? BTW, to avoid posting actual business info, am all using dummy file names. Batch sftp commands file content : ls /reports/xxxxx.txt get "/reports/xxxxx.txt" "C:\tmp\123456.tmp"ls /reports/xxxxx1.txt get "/reports/xxxxx1.txt" "C:\tmp\123457.tmp" Hello, I need some more information on your setup. Are you using both our server and client for the download? Do you have any issues if you run the commands from an interactive sftp session? Are you running the batch file manually or is it a scheduled task? Have you tried logging the output for any reported errors. You can log by adding the -L option to the sftp command. Thanks for the prompt reply. We have both Pragma fortress server and client there installed in the machine, while during our Java program execution, it shall use sftp client only I believe. We don't have the issue when I manually execute commands in interactive sftp session. We run sftp batch file in a Java program which execute regularly with CRON expression config. As it's for production, haven't enabled the -L option, but we will try this in testing env to see if possible to reproduce. Also one thing supplement here, in last couple of months e.g. Jan to May, such case happened once, but recent two weeks twice it occurred. Few more update here, the latest monitoring in testing env it looks like that the sftp command will trigger another sub-process like below: #1 The sftp command sample: sftp -oUser=xxxxx1 -oConnectionTimeout=120s -oPort=22 -oIdentityfile2=blahblahblah external-sftp-host.com #2 The above sftp command will trigger a subprocess like below ssh ssh -o User="xxxxx1" -oConnectionTimeout="120s" -o Port="22" -o Identityfile2="blahblahblah" external-sftp-host.com -v -s oProtocol=2 sftp If it's true, any scenario that the ssh sub-process will get hang (e.g. wait for input signal) or "keep running" ... etc? What you are seeing is the norm for the sftp protocol. The sftp client launches the ssh client which handles all of the authentication and encryption for the file transfer. If the ssh process closes, the sftp process should close as well. If the ssh process is running the sftp process should be running as well. If the "hang" is in during the file transfer it is most likely an issue with the sftp portion file manipulation and not the network or ssh portion of the process. Have you tried running the same command line you run in cron from the command line to try to duplicate? It might be a timing issue or possibly a user context issue of who is running the job, for example if you are running the cron job as system, you may not have access to any necessary configuration files that are used when the command is run as a particular user. Hi team, Thanks a lot for the update, we have tried running the same command line manually (hmm... many times) but unfortunately no able to reproduce, and we believe it's not a privilege related issue, since it only occasionally happens. And the production on-site investigation that I have described earlier above, that first file content is half done and the second file is 0 byte, this seems that it's hang during file transfer (in the middle of the sftp command batch file). As to remediate this, we have changed our code and if the sftp process is continue running for e.g. 5 mins (since the file is not big one), we force destroy it and with alert raised to support. Also with the retry mechanism in app, it's a tactical fix on this issue from our end. This only happens when connecting to external via proxy, all intra-net files transfer work like a charm with even heavier workload (for years). If later on there is new version on such enhancement, kindly let me know.
|
|
|
|
|
Technical Support Group...
|
|
|
Group: Moderators
Posts: 136,
Visits: 639
|
+x+x+x+x+x+xHi team, We are using "Pragma Fortress 5.0.10.1563" version and now in production it occasionally hang during file transfer with external sftp host. In the software program it will run the batch sftp commands file to fetch multiple files, sample content is listed below, during the incident, we found out that the sftp command hang at the BOLD one (with underline). We can see the 123456.tmp file is there in C:\tmp but the content is incomplete. While the second file 123457.tmp file existed but with 0 byte length. We have checked with network team and all is fine at that moment, we are using proxy to go to outside external sftp host. For other scenario, some times none of the tmp files there created under C:\tmp and we can see in windows task manager "Process" tab, that a sftp process is running there running the batch sftp commands file, seemingly hang. Am wondering if you can shed some light on us e.g. any advice for workaround or strategic solution on such? BTW, to avoid posting actual business info, am all using dummy file names. Batch sftp commands file content : ls /reports/xxxxx.txt get "/reports/xxxxx.txt" "C:\tmp\123456.tmp"ls /reports/xxxxx1.txt get "/reports/xxxxx1.txt" "C:\tmp\123457.tmp" Hello, I need some more information on your setup. Are you using both our server and client for the download? Do you have any issues if you run the commands from an interactive sftp session? Are you running the batch file manually or is it a scheduled task? Have you tried logging the output for any reported errors. You can log by adding the -L option to the sftp command. Thanks for the prompt reply. We have both Pragma fortress server and client there installed in the machine, while during our Java program execution, it shall use sftp client only I believe. We don't have the issue when I manually execute commands in interactive sftp session. We run sftp batch file in a Java program which execute regularly with CRON expression config. As it's for production, haven't enabled the -L option, but we will try this in testing env to see if possible to reproduce. Also one thing supplement here, in last couple of months e.g. Jan to May, such case happened once, but recent two weeks twice it occurred. Few more update here, the latest monitoring in testing env it looks like that the sftp command will trigger another sub-process like below: #1 The sftp command sample: sftp -oUser=xxxxx1 -oConnectionTimeout=120s -oPort=22 -oIdentityfile2=blahblahblah external-sftp-host.com #2 The above sftp command will trigger a subprocess like below ssh ssh -o User="xxxxx1" -oConnectionTimeout="120s" -o Port="22" -o Identityfile2="blahblahblah" external-sftp-host.com -v -s oProtocol=2 sftp If it's true, any scenario that the ssh sub-process will get hang (e.g. wait for input signal) or "keep running" ... etc? What you are seeing is the norm for the sftp protocol. The sftp client launches the ssh client which handles all of the authentication and encryption for the file transfer. If the ssh process closes, the sftp process should close as well. If the ssh process is running the sftp process should be running as well. If the "hang" is in during the file transfer it is most likely an issue with the sftp portion file manipulation and not the network or ssh portion of the process. Have you tried running the same command line you run in cron from the command line to try to duplicate? It might be a timing issue or possibly a user context issue of who is running the job, for example if you are running the cron job as system, you may not have access to any necessary configuration files that are used when the command is run as a particular user. Hi team, Thanks a lot for the update, we have tried running the same command line manually (hmm... many times) but unfortunately no able to reproduce, and we believe it's not a privilege related issue, since it only occasionally happens. And the production on-site investigation that I have described earlier above, that first file content is half done and the second file is 0 byte, this seems that it's hang during file transfer (in the middle of the sftp command batch file). As to remediate this, we have changed our code and if the sftp process is continue running for e.g. 5 mins (since the file is not big one), we force destroy it and with alert raised to support. Also with the retry mechanism in app, it's a tactical fix on this issue from our end. This only happens when connecting to external via proxy, all intra-net files transfer work like a charm with even heavier workload (for years). If later on there is new version on such enhancement, kindly let me know. Hello, Have you tried any logging on the client side using the -L option? This will give us more information.
Pragma Systems Technical Support 13809 Research Blvd, #675 Austin, TX 78750 http://www.pragmasys.com
|
|
|
|
|
alex qiu
|
|
|
Group: Forum Members
Posts: 17,
Visits: 85
|
+x+x+x+x+x+x+xHi team, We are using "Pragma Fortress 5.0.10.1563" version and now in production it occasionally hang during file transfer with external sftp host. In the software program it will run the batch sftp commands file to fetch multiple files, sample content is listed below, during the incident, we found out that the sftp command hang at the BOLD one (with underline). We can see the 123456.tmp file is there in C:\tmp but the content is incomplete. While the second file 123457.tmp file existed but with 0 byte length. We have checked with network team and all is fine at that moment, we are using proxy to go to outside external sftp host. For other scenario, some times none of the tmp files there created under C:\tmp and we can see in windows task manager "Process" tab, that a sftp process is running there running the batch sftp commands file, seemingly hang. Am wondering if you can shed some light on us e.g. any advice for workaround or strategic solution on such? BTW, to avoid posting actual business info, am all using dummy file names. Batch sftp commands file content : ls /reports/xxxxx.txt get "/reports/xxxxx.txt" "C:\tmp\123456.tmp"ls /reports/xxxxx1.txt get "/reports/xxxxx1.txt" "C:\tmp\123457.tmp" Hello, I need some more information on your setup. Are you using both our server and client for the download? Do you have any issues if you run the commands from an interactive sftp session? Are you running the batch file manually or is it a scheduled task? Have you tried logging the output for any reported errors. You can log by adding the -L option to the sftp command. Thanks for the prompt reply. We have both Pragma fortress server and client there installed in the machine, while during our Java program execution, it shall use sftp client only I believe. We don't have the issue when I manually execute commands in interactive sftp session. We run sftp batch file in a Java program which execute regularly with CRON expression config. As it's for production, haven't enabled the -L option, but we will try this in testing env to see if possible to reproduce. Also one thing supplement here, in last couple of months e.g. Jan to May, such case happened once, but recent two weeks twice it occurred. Few more update here, the latest monitoring in testing env it looks like that the sftp command will trigger another sub-process like below: #1 The sftp command sample: sftp -oUser=xxxxx1 -oConnectionTimeout=120s -oPort=22 -oIdentityfile2=blahblahblah external-sftp-host.com #2 The above sftp command will trigger a subprocess like below ssh ssh -o User="xxxxx1" -oConnectionTimeout="120s" -o Port="22" -o Identityfile2="blahblahblah" external-sftp-host.com -v -s oProtocol=2 sftp If it's true, any scenario that the ssh sub-process will get hang (e.g. wait for input signal) or "keep running" ... etc? What you are seeing is the norm for the sftp protocol. The sftp client launches the ssh client which handles all of the authentication and encryption for the file transfer. If the ssh process closes, the sftp process should close as well. If the ssh process is running the sftp process should be running as well. If the "hang" is in during the file transfer it is most likely an issue with the sftp portion file manipulation and not the network or ssh portion of the process. Have you tried running the same command line you run in cron from the command line to try to duplicate? It might be a timing issue or possibly a user context issue of who is running the job, for example if you are running the cron job as system, you may not have access to any necessary configuration files that are used when the command is run as a particular user. Hi team, Thanks a lot for the update, we have tried running the same command line manually (hmm... many times) but unfortunately no able to reproduce, and we believe it's not a privilege related issue, since it only occasionally happens. And the production on-site investigation that I have described earlier above, that first file content is half done and the second file is 0 byte, this seems that it's hang during file transfer (in the middle of the sftp command batch file). As to remediate this, we have changed our code and if the sftp process is continue running for e.g. 5 mins (since the file is not big one), we force destroy it and with alert raised to support. Also with the retry mechanism in app, it's a tactical fix on this issue from our end. This only happens when connecting to external via proxy, all intra-net files transfer work like a charm with even heavier workload (for years). If later on there is new version on such enhancement, kindly let me know. Hello, Have you tried any logging on the client side using the -L option? This will give us more information. hi there, Just get back on this one providing an update, we have applied the changed program in prod. When same issue happened, program forcing the process to end, it retry after 5 mins. Two cases happened post impl till now, and proven the solution works. As it's prod env we haven't enable the -L option in command since it will globally affect all sftp commands within the program (Including internal/external).
|
|
|
|