Hi,

I've been wrestling with a problem, for a few months, now, where a host (running Windows 7 Pro, Pragma FortressSSH and Filezilla Server) that I inherited admin duties on becomes unavailable -- attempts to RDP into it fail, SFTP and FTP connection requests fail -- and only a restart of the host brings it back online.  It will then function for a number of days and sometimes even weeks, before going down, again.

Referencing notes on this page;

https://docs.microsoft.com/en-us/archive/blogs/askds/port-exhaustion-and-you-or-why-the-netstat-tool-is-your-friend

I have been looking into the question of whether or not the trouble might be port exhaustion.  I have applied all of the latest Windows updates and have even gone as far as applying patches such as this one, that seem to directly address the trouble that I've been seeing:

https://forum.filezilla-project.org/viewtopic.php?t=49308&start=15

None of this has helped, though.

Pasted to the bottom of this message is the text of two small scripts that I've put together:
1.) tcp_port_summary.sh simply runs the referenced netstat call to gather info on port usage on the problem win7 host, and redirects output from that call to a file, with the current time/date stamp in its name.
2.) tcp_connections_counter.sh, run against output from tcp_port_summary.sh, will produce a simple report showing the local and foreign IP addresses that appear in the output, with counts for each. 

I setup a task_manager job to run tcp_port_summary.sh, daily, once every hour.  Then, I restarted the host.  So, I periodically compare output of tcp_connections_counter.sh, run against the file generated by the first, post-restart run of tcp_port_summary.sh with output from the same, run against more recent tcp_port_summary.sh files.

I find that connections from a single foreign IP to the win7 host's port 22 are increasing, daily, in what seems, so far, more an exponential than linear fashion.  The PID associated with the relevant entries belongs to "Pragma InetD". 

At no point, since I started logging the IP counts, does the number of connections from that single foreign IP go down, between host restarts.

Has anyone else out there encountered this issue?  'Anyone have thoughts on how it might be resolved?

Thanks,

-Anthony
 







==> tcp_port_summary.sh


@echo off

REm C:\Users\sftp_admin\Downloads>C:\Windows\System32\netstat -anob > ports_02112020_1148am.txt


for /f "tokens=*" %%i in ('tzutil /g') do set CTZ=%%i
tzutil /s UTC
set UTC=
for /f "skip=1 delims=" %%i in ('WMIC OS GET LocalDateTime') do if not defined UTC set UTC=%%i
tzutil /s "%CTZ%"
set UTC=%UTC:~0,4%-%UTC:~4,2%-%UTC:~6,3%T%UTC:~8,13%
REM echo %UTC%


netstat -anob > ports_%UTC%.txt

==> tcp_port_summary.sh (END)


==>  tcp_connections_counter.sh

#!/bin/bash

## tcp_connections_counter.sh



file_to_process=$1
target_host_name=cais-sftp-prd01


if [[ -n "$file_to_process" ]]; then
#echo "file_to_process was specified"
        if [[ ! -f "$file_to_process" ]]; then
            echo "file specified for processing does not exist."
            exit
        fi
else
echo "you must supply at least one file to process"
exit
fi


strings_to_filter='active|w32time|ikeext|dnscache|ssdpsrv|ownership|lanman|policy|Schedule|eventlog|CryptSvc|CryptSvc|RpcSs|gpsvc|proto'
remove_blank_lines='^[[:space:]]*$'


echo -e "## 'Local Address' counts:\n"

cat $file_to_process | grep -v "\[" | egrep -v -i $strings_to_filter | grep -v -e $remove_blank_lines | tr -s ' ' | sort -k 2 | cut -d ' ' -f3 | cut -d':' -f1 | grep -v "\*" | grep -v -e $remove_blank_lines | while read line; do echo "ip count for $line = $(grep -o $line $file_to_process | wc -l)"; done | sort -k 6 | uniq

echo -e "\n## 'Foreign Address' counts:\n"
cat $file_to_process | grep -v "\[" | egrep -v -i $strings_to_filter | grep -v -e $remove_blank_lines | tr -s ' ' | sort -k 2 | cut -d ' ' -f4 | cut -d':' -f1 | grep -v "\*" | grep -v -e $remove_blank_lines | while read line; do echo "ip count for $line = $(grep -o $line $file_to_process | wc -l)"; done | sort -k 6 | uniq

==> tcp_connections_counter.sh

+x

 

abgenerette - 2/12/2020 5:25:14 PM

Hi,

I've been wrestling with a problem, for a few months, now, where a host (running Windows 7 Pro, Pragma FortressSSH and Filezilla Server) that I inherited admin duties on becomes unavailable -- attempts to RDP into it fail, SFTP and FTP connection requests fail -- and only a restart of the host brings it back online.  It will then function for a number of days and sometimes even weeks, before going down, again.

Referencing notes on this page;

https://docs.microsoft.com/en-us/archive/blogs/askds/port-exhaustion-and-you-or-why-the-netstat-tool-is-your-friend

I have been looking into the question of whether or not the trouble might be port exhaustion.  I have applied all of the latest Windows updates and have even gone as far as applying patches such as this one, that seem to directly address the trouble that I've been seeing:

https://forum.filezilla-project.org/viewtopic.php?t=49308&start=15

None of this has helped, though.

Pasted to the bottom of this message is the text of two small scripts that I've put together:
1.) tcp_port_summary.sh simply runs the referenced netstat call to gather info on port usage on the problem win7 host, and redirects output from that call to a file, with the current time/date stamp in its name.
2.) tcp_connections_counter.sh, run against output from tcp_port_summary.sh, will produce a simple report showing the local and foreign IP addresses that appear in the output, with counts for each. 

I setup a task_manager job to run tcp_port_summary.sh, daily, once every hour.  Then, I restarted the host.  So, I periodically compare output of tcp_connections_counter.sh, run against the file generated by the first, post-restart run of tcp_port_summary.sh with output from the same, run against more recent tcp_port_summary.sh files.

I find that connections from a single foreign IP to the win7 host's port 22 are increasing, daily, in what seems, so far, more an exponential than linear fashion.  The PID associated with the relevant entries belongs to "Pragma InetD". 

At no point, since I started logging the IP counts, does the number of connections from that single foreign IP go down, between host restarts.

Has anyone else out there encountered this issue?  'Anyone have thoughts on how it might be resolved?

Thanks,

-Anthony
 







==> tcp_port_summary.sh


@echo off

REm C:\Users\sftp_admin\Downloads>C:\Windows\System32\netstat -anob > ports_02112020_1148am.txt


for /f "tokens=*" %%i in ('tzutil /g') do set CTZ=%%i
tzutil /s UTC
set UTC=
for /f "skip=1 delims=" %%i in ('WMIC OS GET LocalDateTime') do if not defined UTC set UTC=%%i
tzutil /s "%CTZ%"
set UTC=%UTC:~0,4%-%UTC:~4,2%-%UTC:~6,3%T%UTC:~8,13%
REM echo %UTC%


netstat -anob > ports_%UTC%.txt

==> tcp_port_summary.sh (END)


==>  tcp_connections_counter.sh

#!/bin/bash

## tcp_connections_counter.sh



file_to_process=$1
target_host_name=cais-sftp-prd01


if [[ -n "$file_to_process" ]]; then
#echo "file_to_process was specified"
        if [[ ! -f "$file_to_process" ]]; then
            echo "file specified for processing does not exist."
            exit
        fi
else
echo "you must supply at least one file to process"
exit
fi


strings_to_filter='active|w32time|ikeext|dnscache|ssdpsrv|ownership|lanman|policy|Schedule|eventlog|CryptSvc|CryptSvc|RpcSs|gpsvc|proto'
remove_blank_lines='^[[:space:]]*$'


echo -e "## 'Local Address' counts:\n"

cat $file_to_process | grep -v "\[" | egrep -v -i $strings_to_filter | grep -v -e $remove_blank_lines | tr -s ' ' | sort -k 2 | cut -d ' ' -f3 | cut -d':' -f1 | grep -v "\*" | grep -v -e $remove_blank_lines | while read line; do echo "ip count for $line = $(grep -o $line $file_to_process | wc -l)"; done | sort -k 6 | uniq

echo -e "\n## 'Foreign Address' counts:\n"
cat $file_to_process | grep -v "\[" | egrep -v -i $strings_to_filter | grep -v -e $remove_blank_lines | tr -s ' ' | sort -k 2 | cut -d ' ' -f4 | cut -d':' -f1 | grep -v "\*" | grep -v -e $remove_blank_lines | while read line; do echo "ip count for $line = $(grep -o $line $file_to_process | wc -l)"; done | sort -k 6 | uniq

==> tcp_connections_counter.sh

It sounds like you might be getting a Denial of Service attack that blocks your Windows network ability to answer any port, since all ports block. I recommend adding the foreign IP to our block IP address, so that we will ignore any connections from that IP. If you have a specific range of IPs that you expect your clients to use, you can also add an allowed range of IP addresses and block all other addresses. You add IP restriction on the InetD -> Fortress page of the Local Server Configuration program.


Pragma Systems Technical Support
13809 Research Blvd, #675
Austin, TX 78750
http://www.pragmasys.com

+x

 

Technical Support Group (TSG) - 2/13/2020 7:35:23 PM

+x

 

abgenerette - 2/12/2020 5:25:14 PM

Hi,

I've been wrestling with a problem, for a few months, now, where a host (running Windows 7 Pro, Pragma FortressSSH and Filezilla Server) that I inherited admin duties on becomes unavailable -- attempts to RDP into it fail, SFTP and FTP connection requests fail -- and only a restart of the host brings it back online.  It will then function for a number of days and sometimes even weeks, before going down, again.

Referencing notes on this page;

https://docs.microsoft.com/en-us/archive/blogs/askds/port-exhaustion-and-you-or-why-the-netstat-tool-is-your-friend

I have been looking into the question of whether or not the trouble might be port exhaustion.  I have applied all of the latest Windows updates and have even gone as far as applying patches such as this one, that seem to directly address the trouble that I've been seeing:

https://forum.filezilla-project.org/viewtopic.php?t=49308&start=15

None of this has helped, though.

Pasted to the bottom of this message is the text of two small scripts that I've put together:
1.) tcp_port_summary.sh simply runs the referenced netstat call to gather info on port usage on the problem win7 host, and redirects output from that call to a file, with the current time/date stamp in its name.
2.) tcp_connections_counter.sh, run against output from tcp_port_summary.sh, will produce a simple report showing the local and foreign IP addresses that appear in the output, with counts for each. 

I setup a task_manager job to run tcp_port_summary.sh, daily, once every hour.  Then, I restarted the host.  So, I periodically compare output of tcp_connections_counter.sh, run against the file generated by the first, post-restart run of tcp_port_summary.sh with output from the same, run against more recent tcp_port_summary.sh files.

I find that connections from a single foreign IP to the win7 host's port 22 are increasing, daily, in what seems, so far, more an exponential than linear fashion.  The PID associated with the relevant entries belongs to "Pragma InetD". 

At no point, since I started logging the IP counts, does the number of connections from that single foreign IP go down, between host restarts.

Has anyone else out there encountered this issue?  'Anyone have thoughts on how it might be resolved?

Thanks,

-Anthony
 







==> tcp_port_summary.sh


@echo off

REm C:\Users\sftp_admin\Downloads>C:\Windows\System32\netstat -anob > ports_02112020_1148am.txt


for /f "tokens=*" %%i in ('tzutil /g') do set CTZ=%%i
tzutil /s UTC
set UTC=
for /f "skip=1 delims=" %%i in ('WMIC OS GET LocalDateTime') do if not defined UTC set UTC=%%i
tzutil /s "%CTZ%"
set UTC=%UTC:~0,4%-%UTC:~4,2%-%UTC:~6,3%T%UTC:~8,13%
REM echo %UTC%


netstat -anob > ports_%UTC%.txt

==> tcp_port_summary.sh (END)


==>  tcp_connections_counter.sh

#!/bin/bash

## tcp_connections_counter.sh



file_to_process=$1
target_host_name=cais-sftp-prd01


if [[ -n "$file_to_process" ]]; then
#echo "file_to_process was specified"
        if [[ ! -f "$file_to_process" ]]; then
            echo "file specified for processing does not exist."
            exit
        fi
else
echo "you must supply at least one file to process"
exit
fi


strings_to_filter='active|w32time|ikeext|dnscache|ssdpsrv|ownership|lanman|policy|Schedule|eventlog|CryptSvc|CryptSvc|RpcSs|gpsvc|proto'
remove_blank_lines='^[[:space:]]*$'


echo -e "## 'Local Address' counts:\n"

cat $file_to_process | grep -v "\[" | egrep -v -i $strings_to_filter | grep -v -e $remove_blank_lines | tr -s ' ' | sort -k 2 | cut -d ' ' -f3 | cut -d':' -f1 | grep -v "\*" | grep -v -e $remove_blank_lines | while read line; do echo "ip count for $line = $(grep -o $line $file_to_process | wc -l)"; done | sort -k 6 | uniq

echo -e "\n## 'Foreign Address' counts:\n"
cat $file_to_process | grep -v "\[" | egrep -v -i $strings_to_filter | grep -v -e $remove_blank_lines | tr -s ' ' | sort -k 2 | cut -d ' ' -f4 | cut -d':' -f1 | grep -v "\*" | grep -v -e $remove_blank_lines | while read line; do echo "ip count for $line = $(grep -o $line $file_to_process | wc -l)"; done | sort -k 6 | uniq

==> tcp_connections_counter.sh

It sounds like you might be getting a Denial of Service attack that blocks your Windows network ability to answer any port, since all ports block. I recommend adding the foreign IP to our block IP address, so that we will ignore any connections from that IP. If you have a specific range of IPs that you expect your clients to use, you can also add an allowed range of IP addresses and block all other addresses. You add IP restriction on the InetD -> Fortress page of the Local Server Configuration program.

Hi,

I neglected to mention this, in my initial posting, but that foreign IP that I referred to actually looks to be a valid one: ARIN/WHOIS shows it as belonging to one of the companies that we do business with and all of the connections from it are on port 22, which is expected.

So, it's like their SFTP connections are just not being closed, properly, or something along those lines.

-Anthony

It sounds like you might be getting a Denial of Service attack that blocks your Windows network ability to answer any port, since all ports block. I recommend adding the foreign IP to our block IP address, so that we will ignore any connections from that IP. If you have a specific range of IPs that you expect your clients to use, you can also add an allowed range of IP addresses and block all other addresses. You add IP restriction on the InetD -> Fortress page of the Local Server Configuration program.