Xcopy: Insufficient Memory
Here is an interesting issue I ran into today. I have a script which creates a local copy of one of the important folders on the server using the xcopy command. To make the long story short, after several “accidents” with people messing up or deleting files on the server, we figured that letting people to make local mirrors on their own desktops might be slightly better than restoring stuff back from the tape 4 times a day.
It was something along the lines of:
rmdir /S /Q "%HOMEPATH%\My Documents\Important"
xcopy /E /R /Y /I G:\Important "%HOMEPATH%\My Documents\Important"
The script worked fine for almost a year now. Today however it decided to fail. After copying 1 or 2 folders it would simply stop. After closer inspection of the output I noticed that the last line was:
Insufficient Memory
I immediately checked the memory. The machine had 512 MB of RAM, and over 20 GB of free disk space. The folder I was attempting to copy had only 128 MB. This could not be correct.
Quick google confirmed this - the “insufficient memory” message is very misleading. It appears that this message shows up when the fully qualified (ie. with path) name of the copied file is longer than 254 characters which seems to be Windows maximum path length. Yes - even in WinXP SP2 and Win 2k3. Thank you Microsoft!
So I’m guessing the message doesn’t really refer to the actual free memory, but to the amount of space left in the file name buffer… Or at least I hope that’s what it means. Either that or MS just decided to make “insufficient memory” one of those lovely have a “catch all exceptions and errors here” clauses.
There are essentially 3 workarounds to this:
- fix your paths - you can try using virtual drives, junctions and all that jazz to make the source and destination paths shorter. Still, this does not mean that a trigger happy user won’t build a crazy long directory tree in your important folder in the future
- Use xxcopy - a 3rd party xcopy clone. It was suggested by David at blogdom.org. This actually worked for me but I’m not entirely happy with this solution. The good part is that xxcopy implements almost all the regular xcopy switches. The bad part is that to use it for commercial purposes you are supposed to pay up $20 or more.
- Use robocopy - which seems to be the optimal solution. Robocopy is a part of Windows Server 2003 Resource Kit. The kit will only install on Windows XP and 2003 but robocopy will work on any NT compatible machine. Just install the kit on some machine and copy robocopy.exe from it to all the workstations that need it
The added benefit of using robocopy is that it actually simplified my script:
robocopy \\server\share\Important "%HOMEPATH%\My Documents\Important"
Robocopy can snag the copies directly from the network share, so I do not have to worry about having correct drive letters assigned to the correct shares. This was never really a problem for me since I use login scripts to mount shares on our network, but it is a good thing nevertheless.
In addition robocopy does true mirroring - it will purge all the files in the destination folder which do not appear in the source, saving me the headache of deleting the whole folder prior to copying.
In conclusion - xxcopy works, but robocopy works better. Get it and use it.
Related Posts:
May 5th, 2008 at 5:28 pm (8968) [Quote]
I have the same problem with xcopy on xp sp2+ copying over a network mapped drive. It misses files. I suspect long path names or certain system files?
Posted usingMay 5th, 2008 at 6:58 pm (8969) [Quote]
Yup - long file paths are usually the cause here. I recommend using Robocopy instead.
Posted usingMay 5th, 2008 at 6:59 pm (8970) [Quote]
SOLUTION:
The source drive D: was FAT32 the Target is an xp box share drive formatted with NTFS mapped to Z:\ permissons Everyone Full Control. Both xp pc’s are running xp pro sp2.
The files that failed to copy were in a stored xp user profile folder (no in use and not in \documents and setings\ so there were no file lock issues. I ziped the user profile folder which was about 10 megabytes and the problem went away.
I then moved the two user profiles into a dir called d:\delme and created a bunch of really long folder names like “zzzzzzzzzzzzzzzzzzz33″ and everything copied find, total files matched and so did size to the byte.
–> Microsoft does not acknowledge the problem on their web site.
SOLUTION:
I copied D:\DRIVERS (FAT32) to E:\DRIVERS (NTFS), both D: and E: are normal local SATA drives / partitions then I performed the copy again and it worked. The total number of files was: 3,438 and ~ 652 megabytes. The copy took an unusually long time 3 or 4 minutes when it should have taken less than 20 seconds. Usually, the copy speed is 30 to 40 MB/sec.
The problem seems to be related to the network redirector and or aka mapped drive’s inability to properly handle some file/folder name lengths or types going from FAT32 to NTFS.
Posted using