Long Path Madness

Most people never run into the MAX_PATH issue when using windows. I run into it every other week. Windows has thins funky little quirk about long file paths. The fully qualified path of any file on your NTFS file system is supposed to be under 256 characters. It’s a weird limitation, and hell – if your path is 250 characters long, you are probably doing something very wrong, right?

No, not really. For example, consider this sample path:

C:\Documents and Settings\Johnathan Smith\My Documents\Work Documents\

Let’s say our imaginary friend John Smith keeps all his work related documents in a folder called Work Documents in his My Documents directory. Seems reasonable, isn’t it? Note that path to his work folder is already 70 characters. I’m not doing anything outrageous here – that’s just how it is.

Let’s say John is very organized and he keeps various documents in aptly named sub folders. For example he has a folder named “Reports” and inside couple of sub folders like “For Review” and “Reviewed”. Let’s say he dumps new reports in the “For Review” folder as he gets them. Since he gets several of them each day he keeps them organized by month and company name. So a typical path he might be working with on any given day could be something like this:

C:\Documents and Settings\Johnathan Smith\My Documents\Work Documents\Reports\For Review\Jan 2008\Initrode International Corporation\

If you are counting this is already whopping 134 characters. This is still a very simple hierarchy – no exceedingly huge folder names, no redundant sub folders and nothing out of oridinary. And yet, it is already huge. Now let’s assume that John gets a report in a zip file aptly named something like:

Initrode International Corp Field Report Jan-13-07 by JB and RF.zip

Yes, it’s a little bit long, but this is not an outrageous file naming convention. Some managers do insist on using a scheme like [project name] [document type] [date] [name of submitter] for all the data you send into the office. John diligently extracts this zip file and gets another hierarchy unrolled in his work folder. For example the file he is interested in might be located in following set of folders:

\Workpapers\Accounts Receivable\Dec 2007\Initrode Accounts Receivable Aging Schedule – Projected 12-07.doc

Nothing out of usual here – granted, we could shorten the file name, and instead of spelling out Accounts Receivable we could just say AR, but honest to goodness I do see file paths like that at work all the time. So let’s put the two paths together and see where it takes us:

C:\Documents and Settings\Johnathan Smith\My Documents\Work Documents\Reports\For Review\Jan 2008\Initrode International Corporation\Initrode International Corp Field Report Jan-13-07 by JB and RF\Workpapers\Accounts Receivable\Dec 2007\Initrode Accounts Receivable Aging Schedule – Projected 12-07.doc

Vim tells me that this path is exactly 302 characters, and Vim usually doesn’t lie. This puts us way beyond the MAX_PATH but windows won’t really care. It will let us create this path by unzipping, and it won’t complain. The fun starts if our friend John decides to open this file in Word and edit it. He won’t be able to save it, no matter how he tries and he will naturally call the IT department at exactly 5:59 pm on a Friday and cry that if he can’t save this file he will loose at least 7 hours of work if not more. I know this, because it happens all the time – and always on Firday as I’m walking out the door.

There is a quick workaround of course – save the file in another folder for example. And according to MSDN you can actually prepend \\?\ to the file name to force windows to use a Unicode based API call which supports path lengths up to 32,000 characters. And of course you can use the eight-dot-three path which is naturally shorter. None of this done by default though.

Part of the problem here is that prerequisite My Documents path which usually weighs in around 50-60 characters. Compare it to the default userspace path in linux which is usually /home/username and can be abreviated as ~. I hear Vista puts user files in a little bit saner directory structure.

Still, while 256 seems like a lot, it is clearly not. A simple folder hierarchy that is only 7 levels deep (at least from John’s perspective) can easily, and without warning hit that magic boundary. I’m not advocating creating complex hierarchies – they do cause problems, but that is a topic for an entirely separate rant. The point is that in the real world using verbose names for files and folders is not something uncommon. In fact it is often considered a desirable practice that improves readability and organization. Microsoft is obviously aware of this issue, since they build in that \\?\ workaround. I wonder why isn’t there some sort of a fall-back mechanism here.

Is it really this hard to catch the MAX_PATH error at the OS level, and then re-try it using the Unicode extended call? This way most of the file system is served with the legacy 256 character capable mechanic, which I suspect might be faster, or more direct. Those special cases would then be handled with the secondary call – completely in the background and transparently to the user. But I guess this is to much to ask. :P

[tags]windows, max_path, maximum path length, windows path length, windows xp[/tags]

This entry was posted in sysadmin notes and tagged , . Bookmark the permalink.



9 Responses to Long Path Madness

  1. Matt` UNITED KINGDOM Mozilla Firefox Windows Terminalist says:

    Well that’s silly…

    Is there a limit on path lengths under Linux? The problem is lessened by the pared back default folder names (seems like every important folder is a 3 letter contraction) but the monster-length file path you came up with is only 9 characters short of 256 if you take out everything up to and including “My Documents”.

    Really should be handled automatically, so I’d be surprised if someone hasn’t already written up some code for Linux to do so.

    Reply  |  Quote
  2. Zack Sloane UNITED STATES Mozilla Firefox Windows says:

    Yes, Vista does do it differently. They use a slight different hierarchy. It’s setup as C:\Users\[Username]\Documents and C:\Users\[Username]\Pictures, etc etc. for all documents, music, pictures, favorites, desktop and whatnot. I believe it is almost the exact same layout as OSx if I am not mistaken. I remember commenting on it when I first got my Vista computer. Noting all the things that MS “tried” to copy out of OSx

    Reply  |  Quote
  3. Luke Maciak UNITED STATES Mozilla Firefox Windows Terminalist says:

    @Matt` – depends on the file system but standard ones like ext2 and ext3 or Raiser-FS have no path size limit defined. Neither does FAT32 and FAT16 – so this path restriction is purely artificial NTFS quirk.

    @Zack – that’s what I thought I read somewhere but I wasn’t sure. It’s nice to know they are adopting saner practices. I guess they had to update the user directory structure with the new security model in place and all. And the fact they ripped it off Mac is not surprise – they have been doing this since the very begging.

    If you run strings on certain applications (like http://ftp.exe) and parts of the TCP/IP stack you will see BSD copyright license notes. Granted – this is perfectly legal under BSD license but it does show you that Windows is basically amalgamation of ideas and code that was developed elsewhere and then poorly ported to work with the rest of the OS. :)

    Reply  |  Quote
  4. Muhammad SINGAPORE Mozilla Firefox Windows Terminalist says:

    Instead of running long paths in Windows, is it possible to truncate it using %SYSTEMROOT% or %APPDATA%? Or does it make no difference?

    Reply  |  Quote
  5. Luke Maciak UNITED STATES Mozilla Firefox Windows Terminalist says:

    I suspect it makes no difference internally, since these system variables simply expand to appropriate paths – they are not symlinks or junctions.

    You could possibly truncate the path by say creating a junction from:

    C:\reports\

    to

    C:\Documents and Settings\Johnathan Smith\Work Files\Reports 07\

    Then each year you can just re-junction the reports folder to another one. Or you could skip My Documents and just keep your files on the root drive. But again, not a perfect solution. :(

    Reply  |  Quote
  6. My new thing with blogs is tl/dr versions are needed ;)

    Reply  |  Quote
  7. Starhawk UNITED STATES Mozilla Firefox Windows says:

    Ya gotta love Windows. lol. I wonder how long we gonna continue to be cursed by the 2 to the eight thing anyway. That maybe made some kinda sense in the MS Dos days but is retarded in a modern OS.

    The simplest solution would be to move the My Documents folder to a better location. It’s not too hard, see How to Change the Default Location of the My Documents Folder. I’ve never did it but it should work ok and btw the same method works for Win 2000 pro. I don’t think it works on Win 98 and stuff like that but I could be wrong. Doubt many ppl still use these OSes anyway. If they do I think a registry hack will do the trick.

    Reply  |  Quote
  8. Luke Maciak UNITED STATES Mozilla Firefox Ubuntu Linux Terminalist says:

    @Travis – tl/dr version: the MAX_PATH in windows is 256 characters and it sucks! lol

    @Starhawk – yes, you could relocate My Documents to a better place. I recommend a separate partition or a secondary drive for example which has added benefit of separating your data from your OS and application files. :)

    Reply  |  Quote
  9. Muhammad SINGAPORE Mozilla Firefox Windows Terminalist says:

    Just right-clicking My Documents and hit properties. From there you can relocate your My Documents folder. No need for registry hacking.

    It’s good to relocate this folder, for the exact reasons Luke has said above.

    Reply  |  Quote

Leave a Reply

Your email address will not be published. Required fields are marked *