Server stops responding to HTTP randomly - can't figure out

Hello,

We have a server that about once a week, stops responding to HTTP requests.

When we telnet to the IP on port 80, the second you hit any key on the
keyboard it disconnects. In IE, you just get page cannot be displayed. Even
when trying locally on the server itself, with public & 127.0.0.1.

IIS Reset does nothing. There is nothing in any event viewer log that
indicates any problems.

The only way to restore HTTP access is by rebooting.

This is a Windows Server 2003 Enterprise Edition server with SP2/R2.


How can we troubleshoot why the server stops responding to HTTP requests?

How can we troubleshoot this when it is not happening? (just rebooted to
fix)


I've put network monitor on the server but it doesn't appear to register any
connections on the IP, netstat doesn't show anything either.


Any insight would be appreciated,
Thanks,
-B

 

Hello Brock,

How is your network setup? Is the machine in the local domain or in the public
network? Please post an unedited ipconfig /all from the server and the DNS
server that it uses.

Best regards

Meinolf Weber
Disclaimer: This posting is provided "AS IS" with no warranties, and confers
no rights.
** Please do NOT email, only reply to Newsgroups
** HELP us help YOU!!! http://www.blakjak.demon.co.uk/mul_crss.htm


> Hello,
>
> We have a server that about once a week, stops responding to HTTP
> requests.
>
> When we telnet to the IP on port 80, the second you hit any key on the
> keyboard it disconnects. In IE, you just get page cannot be displayed.
> Even when trying locally on the server itself, with public &
> 127.0.0.1.
>
> IIS Reset does nothing. There is nothing in any event viewer log that
> indicates any problems.
>
> The only way to restore HTTP access is by rebooting.
>
> This is a Windows Server 2003 Enterprise Edition server with SP2/R2.
>
> How can we troubleshoot why the server stops responding to HTTP
> requests?
>
> How can we troubleshoot this when it is not happening? (just rebooted
> to fix)
>
> I've put network monitor on the server but it doesn't appear to
> register any connections on the IP, netstat doesn't show anything
> either.
>
> Any insight would be appreciated,
> Thanks,
> -B

 

Hello,

The server is part of an Active Directory Domain and is the Active server in
an Active/Passive Failover SQL Cluster.

=============================================
C:\Documents and Settings\Administrator.XX-XX>ipconfig /all

Windows IP Configuration

Host Name . . . . . . . . . . . . : XXXX
Primary Dns Suffix . . . . . . . : XX-XX.LOCAL
Node Type . . . . . . . . . . . . : Unknown
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : XX-XX.LOCAL

Ethernet adapter PublicNetwork:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) PRO/1000 PL Network
Connection
Physical Address. . . . . . . . . : 00-80-14-AE-2A-C1
DHCP Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : 170.11.88.199
Subnet Mask . . . . . . . . . . . : 255.255.255.128
IP Address. . . . . . . . . . . . : 170.11.88.198
Subnet Mask . . . . . . . . . . . : 255.255.255.128
IP Address. . . . . . . . . . . . : 170.11.88.197
Subnet Mask . . . . . . . . . . . : 255.255.255.128
IP Address. . . . . . . . . . . . : 170.11.88.196
Subnet Mask . . . . . . . . . . . : 255.255.255.128
IP Address. . . . . . . . . . . . : 170.11.55.135
Subnet Mask . . . . . . . . . . . : 255.255.255.224
IP Address. . . . . . . . . . . . : 170.11.88.194
Subnet Mask . . . . . . . . . . . : 255.255.255.128
Default Gateway . . . . . . . . . : 170.11.55.129
DNS Servers . . . . . . . . . . . : 10.17.116.74
10.0.80.12

Ethernet adapter PrivateNetwork:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) PRO/1000 PM Network
Connection
Physical Address. . . . . . . . . : 00-80-14-AE-2A-C1
DHCP Enabled. . . . . . . . . . . : No
IP Address. . . . . . . . . . . . : 10.17.117.101
Subnet Mask . . . . . . . . . . . : 255.255.255.128
IP Address. . . . . . . . . . . . : 10.17.117.100
Subnet Mask . . . . . . . . . . . : 255.255.255.128
IP Address. . . . . . . . . . . . : 10.17.116.76
Subnet Mask . . . . . . . . . . . : 255.255.255.192
IP Address. . . . . . . . . . . . : 10.17.117.98
Subnet Mask . . . . . . . . . . . : 255.255.255.128
Default Gateway . . . . . . . . . :
DNS Servers . . . . . . . . . . . : 10.17.116.74
10.0.80.12
=============================================

-B

"Meinolf Weber [MVP-DS]" <meiweb(nospam)@gmx.de> wrote in message
news:ff16fb66205b68cb8fe33864bbfc@msnews.microsoft.com...
> Hello Brock,
>
> How is your network setup? Is the machine in the local domain or in the
> public network? Please post an unedited ipconfig /all from the server and
> the DNS server that it uses.
>
> Best regards
>
> Meinolf Weber
> Disclaimer: This posting is provided "AS IS" with no warranties, and
> confers no rights.
> ** Please do NOT email, only reply to Newsgroups
> ** HELP us help YOU!!! http://www.blakjak.demon.co.uk/mul_crss.htm
>
>> Hello,
>>
>> We have a server that about once a week, stops responding to HTTP
>> requests.
>>
>> When we telnet to the IP on port 80, the second you hit any key on the
>> keyboard it disconnects. In IE, you just get page cannot be displayed.
>> Even when trying locally on the server itself, with public &
>> 127.0.0.1.
>>
>> IIS Reset does nothing. There is nothing in any event viewer log that
>> indicates any problems.
>>
>> The only way to restore HTTP access is by rebooting.
>>
>> This is a Windows Server 2003 Enterprise Edition server with SP2/R2.
>>
>> How can we troubleshoot why the server stops responding to HTTP
>> requests?
>>
>> How can we troubleshoot this when it is not happening? (just rebooted
>> to fix)
>>
>> I've put network monitor on the server but it doesn't appear to
>> register any connections on the IP, netstat doesn't show anything
>> either.
>>
>> Any insight would be appreciated,
>> Thanks,
>> -B
>
>

 

Hello Brock,

I think this is related to the DNS setup on the different cluster adapters
used for internal and public access. I will crosspost this to the cluster
NG, think this will be the better place.



> Hello,
>
> The server is part of an Active Directory Domain and is the Active
> server in an Active/Passive Failover SQL Cluster.
>
> =============================================
> C:\Documents and Settings\Administrator.XX-XX>ipconfig /all
> Windows IP Configuration
>
> Host Name . . . . . . . . . . . . : XXXX
> Primary Dns Suffix . . . . . . . : XX-XX.LOCAL
> Node Type . . . . . . . . . . . . : Unknown
> IP Routing Enabled. . . . . . . . : No
> WINS Proxy Enabled. . . . . . . . : No
> DNS Suffix Search List. . . . . . : XX-XX.LOCAL
> Ethernet adapter PublicNetwork:
>
> Connection-specific DNS Suffix . :
> Description . . . . . . . . . . . : Intel(R) PRO/1000 PL Network
> Connection
> Physical Address. . . . . . . . . : 00-80-14-AE-2A-C1
> DHCP Enabled. . . . . . . . . . . : No
> IP Address. . . . . . . . . . . . : 170.11.88.199
> Subnet Mask . . . . . . . . . . . : 255.255.255.128
> IP Address. . . . . . . . . . . . : 170.11.88.198
> Subnet Mask . . . . . . . . . . . : 255.255.255.128
> IP Address. . . . . . . . . . . . : 170.11.88.197
> Subnet Mask . . . . . . . . . . . : 255.255.255.128
> IP Address. . . . . . . . . . . . : 170.11.88.196
> Subnet Mask . . . . . . . . . . . : 255.255.255.128
> IP Address. . . . . . . . . . . . : 170.11.55.135
> Subnet Mask . . . . . . . . . . . : 255.255.255.224
> IP Address. . . . . . . . . . . . : 170.11.88.194
> Subnet Mask . . . . . . . . . . . : 255.255.255.128
> Default Gateway . . . . . . . . . : 170.11.55.129
> DNS Servers . . . . . . . . . . . : 10.17.116.74
> 10.0.80.12
> Ethernet adapter PrivateNetwork:
>
> Connection-specific DNS Suffix . :
> Description . . . . . . . . . . . : Intel(R) PRO/1000 PM Network
> Connection
> Physical Address. . . . . . . . . : 00-80-14-AE-2A-C1
> DHCP Enabled. . . . . . . . . . . : No
> IP Address. . . . . . . . . . . . : 10.17.117.101
> Subnet Mask . . . . . . . . . . . : 255.255.255.128
> IP Address. . . . . . . . . . . . : 10.17.117.100
> Subnet Mask . . . . . . . . . . . : 255.255.255.128
> IP Address. . . . . . . . . . . . : 10.17.116.76
> Subnet Mask . . . . . . . . . . . : 255.255.255.192
> IP Address. . . . . . . . . . . . : 10.17.117.98
> Subnet Mask . . . . . . . . . . . : 255.255.255.128
> Default Gateway . . . . . . . . . :
> DNS Servers . . . . . . . . . . . : 10.17.116.74
> 10.0.80.12
> =============================================
> -B
>
> "Meinolf Weber [MVP-DS]" <meiweb(nospam)@gmx.de> wrote in message
> news:ff16fb66205b68cb8fe33864bbfc@msnews.microsoft.com...
>
>> Hello Brock,
>>
>> How is your network setup? Is the machine in the local domain or in
>> the public network? Please post an unedited ipconfig /all from the
>> server and the DNS server that it uses.
>>
>> Best regards
>>
>> Meinolf Weber
>> Disclaimer: This posting is provided "AS IS" with no warranties, and
>> confers no rights.
>> ** Please do NOT email, only reply to Newsgroups
>> ** HELP us help YOU!!! http://www.blakjak.demon.co.uk/mul_crss.htm
>>> Hello,
>>>
>>> We have a server that about once a week, stops responding to HTTP
>>> requests.
>>>
>>> When we telnet to the IP on port 80, the second you hit any key on
>>> the keyboard it disconnects. In IE, you just get page cannot be
>>> displayed. Even when trying locally on the server itself, with
>>> public & 127.0.0.1.
>>>
>>> IIS Reset does nothing. There is nothing in any event viewer log
>>> that indicates any problems.
>>>
>>> The only way to restore HTTP access is by rebooting.
>>>
>>> This is a Windows Server 2003 Enterprise Edition server with SP2/R2.
>>>
>>> How can we troubleshoot why the server stops responding to HTTP
>>> requests?
>>>
>>> How can we troubleshoot this when it is not happening? (just
>>> rebooted to fix)
>>>
>>> I've put network monitor on the server but it doesn't appear to
>>> register any connections on the IP, netstat doesn't show anything
>>> either.
>>>
>>> Any insight would be appreciated,
>>> Thanks,
>>> -B

Best regards

Meinolf Weber
Disclaimer: This posting is provided "AS IS" with no warranties, and confers
no rights.
** Please do NOT email, only reply to Newsgroups
** HELP us help YOU!!! http://www.blakjak.demon.co.uk/mul_crss.htm

 

Hi,

> The server is part of an Active Directory Domain and is the Active server
> in an Active/Passive Failover SQL Cluster.
>
> =============================================
> C:\Documents and Settings\Administrator.XX-XX>ipconfig /all
>
> Windows IP Configuration
>
> Host Name . . . . . . . . . . . . : XXXX
> Primary Dns Suffix . . . . . . . : XX-XX.LOCAL
> Node Type . . . . . . . . . . . . : Unknown
> IP Routing Enabled. . . . . . . . : No
> WINS Proxy Enabled. . . . . . . . : No
> DNS Suffix Search List. . . . . . : XX-XX.LOCAL
[....]

It might me unrelated but... there are now problems with domain names ending
in .local
It seems .local is now in use by some auto mechanisms.

Bonno Bloksma

 

Hello Bonno,

The only problems i heard about when using .local are with MAC machines.

Best regards

Meinolf Weber
Disclaimer: This posting is provided "AS IS" with no warranties, and confers
no rights.
** Please do NOT email, only reply to Newsgroups
** HELP us help YOU!!! http://www.blakjak.demon.co.uk/mul_crss.htm


> Hi,
>
>> The server is part of an Active Directory Domain and is the Active
>> server in an Active/Passive Failover SQL Cluster.
>>
>> =============================================
>> C:\Documents and Settings\Administrator.XX-XX>ipconfig /all
>> Windows IP Configuration
>>
>> Host Name . . . . . . . . . . . . : XXXX
>> Primary Dns Suffix . . . . . . . : XX-XX.LOCAL
>> Node Type . . . . . . . . . . . . : Unknown
>> IP Routing Enabled. . . . . . . . : No
>> WINS Proxy Enabled. . . . . . . . : No
>> DNS Suffix Search List. . . . . . : XX-XX.LOCAL
> [....]
>
> It might me unrelated but... there are now problems with domain names
> ending
> in .local
> It seems .local is now in use by some auto mechanisms.
> Bonno Bloksma
>

 

TCP Receive Side Scaling may cause the problem.
(HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableRSS)

http://support.microsoft.com/kb/927695

Best regards,

"Brock Hensley" wrote:

> Hello,
>
> We have a server that about once a week, stops responding to HTTP requests.
>
> When we telnet to the IP on port 80, the second you hit any key on the
> keyboard it disconnects. In IE, you just get page cannot be displayed. Even
> when trying locally on the server itself, with public & 127.0.0.1.
>
> IIS Reset does nothing. There is nothing in any event viewer log that
> indicates any problems.
>
> The only way to restore HTTP access is by rebooting.
>
> This is a Windows Server 2003 Enterprise Edition server with SP2/R2.
>
>
> How can we troubleshoot why the server stops responding to HTTP requests?
>
> How can we troubleshoot this when it is not happening? (just rebooted to
> fix)
>
>
> I've put network monitor on the server but it doesn't appear to register any
> connections on the IP, netstat doesn't show anything either.
>
>
> Any insight would be appreciated,
> Thanks,
> -B
>

 

Hello,

Thank you all for your help. I believe I have found the problem and
resolution.

When the server stops accepting HTTP requests, the error
"1_Connections_Refused" is logged in the Systems HTTPERR log file.

This issue occurs if less than 20 megabytes (MB) of nonpaged pool memory is
available on the server. When less than 20 megabytes (MB) of nonpaged pool
memory is available, the Http.sys kernel mode driver stops accepting new
connections.

This issue may occur in situations in which the /3GB switch is enabled in
the Boot.ini file on the server.

Few options on how to proceed from here.

[option 1]
We can work around this issue and add the "EnableAggressiveMemoryUsage"
registry entry to the registry, essentially lowering the out of memory limit
from 20MB to 8MB, which can be applied with a simple IISReset.

[option 2]
We can also leave the server as it is and wait for the issue to occur again.
Once the issue occurs, we can run a program called "Poolmon" to capture the
servers current non-paged pool memory usage to find out what process is
taking all the memory. We can then address the actual issue with the
offending process.

[option 3]
We can also remove the /3GB flag from the boot.ini file to increase the
Kernel Non Paged Pool (NPP) memory limit from 128MB to 250MB, however,
decreasing virtual address space of processes and applications from 3GB to
2GB.

Hope this helps someone else in the future!
-B

"neothwin" <neothwin@discussions.microsoft.com> wrote in message
news:CFEF09B0-9A4B-49F2-926A-007FF2A9F4D5@microsoft.com...
> TCP Receive Side Scaling may cause the problem.
> (HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableRSS)
>
> http://support.microsoft.com/kb/927695
>
> Best regards,
>
> "Brock Hensley" wrote:
>
>> Hello,
>>
>> We have a server that about once a week, stops responding to HTTP
>> requests.
>>
>> When we telnet to the IP on port 80, the second you hit any key on the
>> keyboard it disconnects. In IE, you just get page cannot be displayed.
>> Even
>> when trying locally on the server itself, with public & 127.0.0.1.
>>
>> IIS Reset does nothing. There is nothing in any event viewer log that
>> indicates any problems.
>>
>> The only way to restore HTTP access is by rebooting.
>>
>> This is a Windows Server 2003 Enterprise Edition server with SP2/R2.
>>
>>
>> How can we troubleshoot why the server stops responding to HTTP requests?
>>
>> How can we troubleshoot this when it is not happening? (just rebooted to
>> fix)
>>
>>
>> I've put network monitor on the server but it doesn't appear to register
>> any
>> connections on the IP, netstat doesn't show anything either.
>>
>>
>> Any insight would be appreciated,
>> Thanks,
>> -B
>>