Defragging a very large, heavily fragmented drive

I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
RAID 5 with a hot spare) that I think is very fragmented. Recently I have
started getting the error "Not enough server storage is available to process
this command" when I try to access certain files or directories on the disk
over the network or locally. From what I have read it seems that heavy
fragmentation can be a problem that causes the disk to act like it is full
when it is not. This server is over 2 years old and has never been
defragged. The disk is 8% empty right now, but was as little as 7% recently
when the problem started. I'm wondering what you thinks is the best course
of action. There is no backup of a lot of the data so that is not an option.
I am currently moving as many files as I can, but that is going very slowly.
The array is not the boot drive on the server. The server is Windows Server
2003 Standard (32-bit). There are two main directory trees on the disk. One
has about 16 million JPGs in it (in a tree, not all in one directory) that
average 670K in size and the other has about 60 million JPGs that average
around 30K. The directory with the smaller images is not critical because
they can be regenerated.

 

"Brad" <Brad@discussions.microsoft.com> wrote in message
news:5231BB06-0F0D-4B75-9ED5-3D94CEAE484F@microsoft.com...
>I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
> RAID 5 with a hot spare) that I think is very fragmented. Recently I have
> started getting the error "Not enough server storage is available to
> process
> this command" when I try to access certain files or directories on the
> disk
> over the network or locally. From what I have read it seems that heavy
> fragmentation can be a problem that causes the disk to act like it is full
> when it is not. This server is over 2 years old and has never been
> defragged. The disk is 8% empty right now, but was as little as 7%
> recently
> when the problem started. I'm wondering what you thinks is the best
> course
> of action. There is no backup of a lot of the data so that is not an
> option.
> I am currently moving as many files as I can, but that is going very
> slowly.
> The array is not the boot drive on the server. The server is Windows
> Server
> 2003 Standard (32-bit). There are two main directory trees on the disk.
> One
> has about 16 million JPGs in it (in a tree, not all in one directory) that
> average 670K in size and the other has about 60 million JPGs that average
> around 30K. The directory with the smaller images is not critical because
> they can be regenerated.

When you say "The disk is 8% empty right now", do you mean to say that the
disk is 92% full? If so then I would describe it as an overly full disk.
IMHO disk should never be allowed to go beyond the 80% limit. When you're
above this limit then defragging will be extremely slow and quite
ineffective.

Note also that defragging will NOT increase the amount of free disk space.
It merely re-arranges the various file fragments.

From other posts I have collected the notes below in relation to the error
message "Not enough storage . . .":
- Usually resolved by applying the latest service pack
- Sometimes caused by NAV or IBM AV:
http://support.microsoft.com/?scid=kb;en-us;177078
- Set
HKLM\SYSTEM\CurrentControlSet\Services\lanmanserver\parameter\IRPStackSize
to 15..21 decimal
- Sometimes requires server service configuration and tuning
http://support.microsoft.com/default.aspx?scid=kb;[LN];128167
- PoolUsageMaximum set too low:
http://support.microsoft.com/default.aspx?scid=kb;EN-US;304101

 

In article <5231BB06-0F0D-4B75-9ED5-3D94CEAE484F@microsoft.com>,
=?Utf-8?B?QnJhZA==?= <Brad@discussions.microsoft.com> wrote:
>I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
>RAID 5 with a hot spare) that I think is very fragmented. Recently I have
>started getting the error "Not enough server storage is available to process
>this command"


Read the message carefully, look at the event code if there is one and
google it.

It's been a long time but there used to be a message that was related
to OS process data structures (not disks and disk space) and the fix,
in my case, was to re-run the latest service pack to fix skewed
versions of DLLs on the system.



--
Al Dykes
News is something someone wants to suppress, everything else is advertising.
- Lord Northcliffe, publisher of the Daily Mail

 

I realize that the disk is way too full. We are not writing any data to it
at this point, but I am moving files off of it and some files are being read
at the same time. I guess I'm just looking for a best recommendation at this
point.

One of the weird issues is that if I issue a Dir D:\01\23\45\ I get the "Not
enough server storage" error, but if I call Dir D:\01\23\45 I get
Directory of \\server\D$\01\23\45

09/26/2008 08:40 AM <DIR> 00
0 File(s) 0 bytes
1 Dir(s) 1,324,973,830,144 bytes free

instead of getting what is inside the directory. The OS doesn't seem to be
able to look inside because of some kind of corruption.

"Pegasus [MVP]" wrote:

>
> "Brad" <Brad@discussions.microsoft.com> wrote in message
> news:5231BB06-0F0D-4B75-9ED5-3D94CEAE484F@microsoft.com...
> >I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
> > RAID 5 with a hot spare) that I think is very fragmented. Recently I have
> > started getting the error "Not enough server storage is available to
> > process
> > this command" when I try to access certain files or directories on the
> > disk
> > over the network or locally. From what I have read it seems that heavy
> > fragmentation can be a problem that causes the disk to act like it is full
> > when it is not. This server is over 2 years old and has never been
> > defragged. The disk is 8% empty right now, but was as little as 7%
> > recently
> > when the problem started. I'm wondering what you thinks is the best
> > course
> > of action. There is no backup of a lot of the data so that is not an
> > option.
> > I am currently moving as many files as I can, but that is going very
> > slowly.
> > The array is not the boot drive on the server. The server is Windows
> > Server
> > 2003 Standard (32-bit). There are two main directory trees on the disk.
> > One
> > has about 16 million JPGs in it (in a tree, not all in one directory) that
> > average 670K in size and the other has about 60 million JPGs that average
> > around 30K. The directory with the smaller images is not critical because
> > they can be regenerated.
>
> When you say "The disk is 8% empty right now", do you mean to say that the
> disk is 92% full? If so then I would describe it as an overly full disk.
> IMHO disk should never be allowed to go beyond the 80% limit. When you're
> above this limit then defragging will be extremely slow and quite
> ineffective.
>
> Note also that defragging will NOT increase the amount of free disk space.
> It merely re-arranges the various file fragments.
>
> From other posts I have collected the notes below in relation to the error
> message "Not enough storage . . .":
> - Usually resolved by applying the latest service pack
> - Sometimes caused by NAV or IBM AV:
> http://support.microsoft.com/?scid=kb;en-us;177078
> - Set
> HKLM\SYSTEM\CurrentControlSet\Services\lanmanserver\parameter\IRPStackSize
> to 15..21 decimal
> - Sometimes requires server service configuration and tuning
> http://support.microsoft.com/default.aspx?scid=kb;[LN];128167
> - PoolUsageMaximum set too low:
> http://support.microsoft.com/default.aspx?scid=kb;EN-US;304101
>
>
>

 

"Brad" <Brad@discussions.microsoft.com> wrote in message
news0846A3F-47E7-4367-A86A-4275F202D2B4@microsoft.com...
>I realize that the disk is way too full. We are not writing any data to it
> at this point, but I am moving files off of it and some files are being
> read
> at the same time. I guess I'm just looking for a best recommendation at
> this
> point.
>
> One of the weird issues is that if I issue a Dir D:\01\23\45\ I get the
> "Not
> enough server storage" error, but if I call Dir D:\01\23\45 I get
> Directory of \\server\D$\01\23\45
>
> 09/26/2008 08:40 AM <DIR> 00
> 0 File(s) 0 bytes
> 1 Dir(s) 1,324,973,830,144 bytes free
>
> instead of getting what is inside the directory. The OS doesn't seem to
> be
> able to look inside because of some kind of corruption.

You can, of course, run chkdsk.exe but I'm always reluctant to do this
unless I have a confirmed saved copy of all files. Chkdsk might thrash a few
files or folder that are perfectly readable right now.

About the disk getting way too full: With servers it is a good idea to
generate weekly management reports about a number of key parameters - and to
read them!

 

I'm not an advocate for defragging. Quite contrary.
IMO,
a) If you have enough free storage space, fragmentation is not the issue.
b) If you don't have enough free storage space, fragmentation is the issue,
but defragging will not help. The only solution is to increase free storage
space.
c) If you start defragging the full disk, you risk loss of data (Exchange
database was known to be sensitive to defragging).
d) Defragging the full disk increases the disk I/O so that other services
suffer significantly.

In your particular case I'd use "divide and conquer" strategy.
I would split this large space to something more manageable.For example you
can use mounting points where you can mount smaller storage units into your
directory tree structure.

"Brad" <Brad@discussions.microsoft.com> wrote in message
news:5231BB06-0F0D-4B75-9ED5-3D94CEAE484F@microsoft.com...
>I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
> RAID 5 with a hot spare) that I think is very fragmented. Recently I have
> started getting the error "Not enough server storage is available to
> process
> this command" when I try to access certain files or directories on the
> disk
> over the network or locally. From what I have read it seems that heavy
> fragmentation can be a problem that causes the disk to act like it is full
> when it is not. This server is over 2 years old and has never been
> defragged. The disk is 8% empty right now, but was as little as 7%
> recently
> when the problem started. I'm wondering what you thinks is the best
> course
> of action. There is no backup of a lot of the data so that is not an
> option.
> I am currently moving as many files as I can, but that is going very
> slowly.
> The array is not the boot drive on the server. The server is Windows
> Server
> 2003 Standard (32-bit). There are two main directory trees on the disk.
> One
> has about 16 million JPGs in it (in a tree, not all in one directory) that
> average 670K in size and the other has about 60 million JPGs that average
> around 30K. The directory with the smaller images is not critical because
> they can be regenerated.

 

In article <5231BB06-0F0D-4B75-9ED5-3D94CEAE484F@microsoft.com>,
Brad@discussions.microsoft.com says...
> I have a 15TB disk array (3ware 24-port array contoller with 750GB disks in
> RAID 5 with a hot spare) that I think is very fragmented. Recently I have
> started getting the error "Not enough server storage is available to process
> this command" when I try to access certain files or directories on the disk
> over the network or locally. From what I have read it seems that heavy
> fragmentation can be a problem that causes the disk to act like it is full
> when it is not. This server is over 2 years old and has never been
> defragged. The disk is 8% empty right now, but was as little as 7% recently
> when the problem started. I'm wondering what you thinks is the best course
> of action. There is no backup of a lot of the data so that is not an option.
> I am currently moving as many files as I can, but that is going very slowly.
> The array is not the boot drive on the server. The server is Windows Server
> 2003 Standard (32-bit). There are two main directory trees on the disk. One
> has about 16 million JPGs in it (in a tree, not all in one directory) that
> average 670K in size and the other has about 60 million JPGs that average
> around 30K. The directory with the smaller images is not critical because
> they can be regenerated.
>

Download JKDefrag and let it run overnight, it's free and does a good
job on overly filled drives.

--
- Igitur qui desiderat pacem, praeparet bellum.
- Calling an illegal alien an "undocumented worker" is like calling a
drug dealer an "unlicensed pharmacist"
spam999free@rrohio.com (remove 999 for proper email address)