Files with quotes in the filename can't be checked out

[robotroll]

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.17.0.windows.1
cpu: x86_64
built from commit: e7621d891d081acff6acd1f0ba6ae0adce06dd09
sizeof-long: 4

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 6.1.7601] Windows 7 x64

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: VIM
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

No

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Issue present in Git Bash, CMD and Powershell.
Issue not present in msys 2 from CYGWIN_NT-6.1-WOW 1.7.35(0.287/5/3) 2015-03-04 12:07 i686 Cygwin

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

$ git clone https://github.com/mhagger/cvs2svn
Cloning into 'cvs2svn'...
remote: Counting objects: 18417, done.
remote: Total 18417 (delta 0), reused 0 (delta 0), pack-reused 18417
Receiving objects: 100% (18417/18417), 10.60 MiB | 3.60 MiB/s, done.
Resolving deltas: 100% (11987/11987), done.
error: unable to create file test-data/main-cvsrepos/single-files/"double-double-quotes",v: Invalid argument
error: unable to create file test-data/main-cvsrepos/single-files/single-double-quote",v: Invalid argument
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

• What did you expect to occur after running these commands?

The repository is cloned and checked out, omitting files that can not be checked out because of errors.

• What actually happened instead?

The check out is incomplete.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

https://github.com/mhagger/cvs2svn

[Osse]

MSYS2 seems to convert " like cygwin does. Cloning this repo with MSYS2 git works just fine, for some definition of fine. One possible way to deal with this is to let Git for Windows do the same, but maybe it would be difficult to copy that behaviour from the MSYS2 emulation layer.

[dscho]

MSYS2 seems to convert " like cygwin does.

Indeed. It replaces characters that are illegal in Windows file names by Unicode characters from a private page.

One possible way to deal with this is to let Git for Windows do the same, but maybe it would be difficult to copy that behaviour from the MSYS2 emulation layer.

That is not the reason we don't do that. The reason is that literally no Windows software other than Cygwin and MSYS2 understand that this private Unicode page characters need to be mapped back to the ASCII page. Quotes are simply not allowed in file names on Windows (among other characters).

[drizzd]

Would it fix your issue if you could define a mapping like tr '"' _ in .git/config? Git could apply such filename translation filters similarly to how to gitattributes filter works for file contents. The user could then decide what kind of mapping makes sense for them.

The user could also be made responsible for making sure that the mapping is injective, so that Git does not have to worry about the possibility that two different tree entries will be mapped to the same worktree file.

[dscho]

@drizzd that's a nice idea in theory, but in practice file names never stand on their own, they are also used. So if you have file names in a project that cannot be checked out, but that are part of, say, the build process, all of a sudden the build process would not find them any longer!

So I am afraid that it is not as easy as introducing a filename translation: we could only possibly fix the Git side of things, and there are so many other sides left.

What could potentially help would be a way to help users rename files upon checkout, i.e. not only translate the filenames, but perform the implicit git mv at the same time, so that you end up with something that was checked out as expected and already has staged changes. The user could then try to ensure that (build) scripts and other consumers of the checked-out files would be adjusted accordingly, before committing.

Something like that could even allow Windows users to check out linux.git (and automatically stage a fixup for the filename aux.c that is illegal on Windows).

At this point, I am unsure about the UI of such a feature. An option to git checkout or git clone? Or should it be a hook? The pre-checkout hook does not exist, but we could introduce it. That would be relatively cumbersome a feature to use, though, as it would require Unix shell scripting (even for users who rightfully want to stay as far away from that as possible).

Of course, it could already be scripted thusly:

#!/bin/sh

die () {
    echo "$*" >&2
    exit 1
}

test $# -ge 1 && test $# -le 3 || die "Usage: $0 <url> [<directory> [<branch-name>]]"
url=$1
directory=${2:-${url##*/}}
branch=${3:-master}

git clone -n "$url" "$directory" &&
cd "$directory" ||
die "Could not clone $url into $directory"

git reset origin/$branch ||
die "Could not update the index to origin/$branch"

git ls-files -z |
xargs -0r -n 1 bash -c \
    'f() { case "$1" in *\"*) git mv --cached "$1" "${1//\"/_}";; esac; }; f "$@"' _ ||
die "Could not rename files with quotes in their filename"

git checkout-index -ua ||
die "Could not check out files"

As I said: not very user-friendly.

[drizzd]

We should just enable file renaming on checkout by default. After all, the only alternative is to fail entirely.

[dscho]

We should just enable file renaming on checkout by default. After all, the only alternative is to fail entirely.

@drizzd I do not think that renaming on checkout would serve anybody: that way, we would pretend that we succeeded to check out the files as promised by the Git repository. But we did not! And pretty much all other software, including, say, any build system, will get utterly confused, and the users will get angry.

I really think that the only alternative, namely: to fail, is really the only option.

What we can do is help the user get out of their fix, e.g. by providing an option to git clone and/or git checkout as I hinted earlier.

[dscho]

This ticket has languished long enough. Time to put it to rest.

[PhilipOakley]

@dscho What are you thoughts regarding how we could somehow 'capture' these interesting but closed issues as potential projects for others to pick up? (in a way that avoids any extra work for you)

I was thinking that perhaps one option is that we should add the "up-for-grabs" label to any interesting (i.e. potential project) issue fairly early in it's life (e.g. if the owner has gone quiet for a week or so, or some timescale that is a balance between letting the owner prepare their contribution, and requesting help via the label).

The other half of the idea is to have an issue that is simply "Projects" which directs folk to search for the "up for grabs" (particularly 'closed ones') issues. As an 'open' issue, title "Projects" it should come to folks attention moderately easily (which is the hard part). I doubt that an independent 'projects' page on the wiki (or most other places) would raise that much interest.

Or maybe I should just (also) create a "Potential Project" label if the up-for grabs doesn't feel a best fit.

The "can't be checked out (on Windows) file names" is probably the broader project topic, rather than just quotes, anyway...

[dscho]

What are you thoughts regarding how we could somehow 'capture' these interesting but closed issues as potential projects for others to pick up?

My first, and biggest, thought is: based on past experience, I expect nobody to be interested.

maybe I should just (also) create a "Potential Project" label if the up-for grabs doesn't feel a best fit.

I think that would be a splendid idea.

The "can't be checked out (on Windows) file names" is probably the broader project topic, rather than just quotes, anyway...

Indeed. And there is of course the totally unclear issue how to resolve this. Just refuse? That's probably not helpful. Try to check them out under a different name, complaining about it, and staging the rename? Maybe.

It would probably make a ton of sense to bring this up on the Git mailing list.

[PhilipOakley]

create a "Potential Project" label if the up-for grabs doesn't feel a best fit.

I think that would be a splendid idea.

Done.

Indeed. And there is of course the totally unclear issue how to resolve this. Just refuse? That's probably not helpful. Try to check them out under a different name, complaining about it, and staging the rename? Maybe.

I think that when it happens we need to offer the user a range of choices (let them make decisions that suit them). I'm not sure upstream are the best judge of the what the right solution would be for the Windows fanbase ;-).

In some ways the 'refuse [don't] checkout' option is a partial checkout capability, where the user selects/marks the irrelevant files and only gets the ones (of the conflict set) they need to edit being checked out (only one per conflict!) while the other files are static/immutable in the index to be part of the next commit.

It maybe that we [also] need a smudge-clean filter suitable for filename munging (which ends up an upstream discussion) so that cases where certain file name characters are meant to be banned (and this maybe a compatibility policy in a community on the Linux side) are munged in some way to a valid filename. This (if automatic) still has the possibility of conflict between the files 1::, 1"", \\, etc. (and all the naff Linux spoof file names with spaces, tabs, bells, etc. etc.)

So I think we need some combination of three things, clean-smudge filenames, sparse checkout integration, and some way of offering user choice at checkout time (and # 4:Willing Devs...)

[drizzd]

How about a git normalize-paths command which will modify all paths in such a way that they can be checked out on Windows and Linux and Mac. This command should work on Windows even if some files cannot currently be checked out. It will commit the change abd a subsequent git reset --hard will checkout all files successfully.

This is good enough to help users checkout files which they otherwise cannot access easily.