Best practice for migrating an existing Git repo to support LFS?

[strich]

I'd like to give Git LFS a good crack with our existing ~25GB Git repo but I'm at a bit of a loss as to how we should go about migrating it to support LFS.
I'm happy to loose all history of the existing large files I'd like to track with LFS.

Do I need to write a script to find and purge all the filetypes I want to track with LFS from the existing Git history, untrack them, then retrack them with LFS?

[andyneff]

I'm HOPING there is a better way to do this... BUT after redacted units of time, I found a way where you will NOT lose any history of existing large files. It is probably SLOW but it works.

DISCLAIMER I'm just a stubborn user, can't promise this will work out for you, BACKUP!

I have run into a similar situation myself, but in my case, all my large files were sitting in submodules. So I was able to remove the submodules, track files, and add them back (without the submodules, HORRAY!) This way I can say I did maintain all my history, only some of the history is in the submodules. I'm guessing this is not similar to your situation, and that everything is in one big repo.

1. Filter-branch to convert all tracking over to lfs

git filter-branch --prune-empty --tree-filter '
git config -f .gitconfig lfs.url "http://127.0.0.1:8080/user/repo"
git lfs track "*.npy"
git add .gitattributes .gitconfig

for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do
  echo "Processing ${file}"

  git rm -f --cached ${file}
  echo "Adding $file lfs style"
  git add ${file}
done' --tag-name-filter cat -- --all

• You have to use tree-filter here, and not index-filter because we are adding objects back.
• I just added track *.npy as an example, but add all of your track commands in there. Hopefully you can use the SAME set of rules for EVERY commit
• The .gitconfig files and git config lines are so that if you have an lfs server in a separate location than the git repo, everything works. IF you have them at the same url, you can skip that step

2. Push changes to whatever remote or remotes you have, of course you have to use the -f option, which can (and WILL!) have many implications to all the other users using the repo. Make sure no one else pushes or references the commits you just rebased, or else you will have a mess. This is when all the large files are sent to the lfs server.

git push -f origin master

(Optional) Collect garbage to shrink currently checked out repo

rm .git/refs/original -rf
git -c gc.reflogExpireUnreachable=0 -c gc.pruneExpire=now gc

• The rm command remove a "backup copy" of the the original refs before the filter-branch command, just to CYA. but when you are done with them, WIPE EM!
• If this doesn't do enough gc, check out http://stackoverflow.com/a/14728706

(Optional) Collect garbage on a bare repo (on the remotes)

git -c gc.reflogExpireUnreachable=0 -c gc.pruneExpire=now gc

I hope this helps!

Untested ideas
Instead Step 1, just add

git lfs track "*.foo"
git config -f .gitconfig lfs.url "http://127.0.0.1:8080/user/repo"
git add .gitattributes .gitconfig

inside the for loop on step two. It should save that wasted initial rebase, and prevent the need for the git add .

Tested using git-lfs 0.5.1 and git 1.9.4.msysgit.1 (Yes, on windows 64 bit)

[tlbtlbtlb]

bfg is much more efficient than git-filter-branch, especially for long histories.
[https://rtyley.github.io/bfg-repo-cleaner/]

Here's roughly what I did to convert all my .mov files to lfs:

• cp *.mov (and a few other large blob types) ~/tmp
• git rm *.mov
• git commit
• git lfs track *.mov
• git add .gitattributes
• git commit; git push

In a fresh directory:

• git clone --mirror $remote; cd repo
• bfg --delete-files '*.mov'
• git reflog expire --expire=now --all && git gc --prune=now --aggressive
• git push

Back to my src directory:

• mv repo repo.bloated
• git clone $remote; cd repo
• cp ~/tmp/*.mov .
• git add *.mov (it now puts them in lfs)
• git commit; git push

Kind of a chore to figure out, but now my repo is small and zippy.
Tip: do this on a cloud machine instead of your laptop, since most of the time is pulling/pushing data to github.

[andyneff]

@tlbtlbtlb in your solution, when you check out all the previous versions of your history, are all the .mov files correctly there, or are the missing everywhere except the newest commits?

[tlbtlbtlb]

They're gone from previous versions. Which is what's necessary to cut the size of the repository.

[andyneff]

I guess I was being stubborn and trying to give anyone an option where they DON'T loose the history of all large files, even though it was stated he'd be happy without them.

I guess the slower build-in equivalent to bfg --delete-files '*.mov' would be

git filter-branch --prune-empty --index-filter 'git rm --ignore-unmatch --cached "*.mov"'

[strich]

Although I'm happy to loose the history of the large files, I would need them to continue to exist in previous commits though - No point having any history at all if every commit previous is broken with missing files.

[andyneff]

@strich I agree with the second statement, I'm a little confused about the first. Are you saying that you are ok with losing the contents AND history of any large files NOT on the tip of your current branch, but the version of the large files that are there, you want them to exist in previous commits up until that version of the file, where before that it will just not exist (be a missing file)?

Example:

If I understand correctly, are you saying that if I have something like this,

commit	what happened	big contents
master	nothing new with big files	big1.bin(version 2)
master~1	update big1.bin to version 2, delete big2.bin	big1.bin(version 2)
master~2	add big2.bin	big1.bin(version 1) and big2.bin
master~3	add big1.bin (version 1)	big1.bin (version 1)

You want to make sure that master and master~1 both point to big1.bin(version 2), but you are ok with big2.bin and big1.bin(version 1) just disappearing?

[strich]

Thanks for continuing to reply @andyneff - Yes that is exactly what I'd like to achieve if it is possible.

[andyneff]

Oh, its possible... My current solution actually preserves ALL the files (so big1.bin(version 1), big1.bin(version 2) AND big2.bin). You you are actually asking for a less, more limited version. It is possible to just use the latest commit if you add a little more

git lfs track "*.npy"
export KEEP_FILE="$(git rev-parse --show-toplevel)/.git/.keep"
git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/" > ${KEEP_FILE}
rm .gitattributes
git checkout .gitattributes  || :

git filter-branch --prune-empty --tree-filter '
git config -f .gitconfig lfs.url "http://127.0.0.1:8080/user/repo"
git lfs track "*.npy"
git add .gitattributes .gitconfig

for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do
  keep_file=0
  while read keep; do
    if [ "${keep}" == "${file}" ]; then
      keep_file=1
      break
    fi
  done < ${KEEP_FILE}
  if [[ ${keep_file} == 1 ]]; then
    git rm --cached ${file}
    git add ${file}
  else
    git rm -f ${file}
  fi
done' --tag-name-filter cat -- --all

rm ${KEEP_FILE}
rm .git/refs/original -rv
git -c gc.reflogExpireUnreachable=0 -c gc.pruneExpire=now gc

This will remove all files matching the lfs track patterns, (in this case, *.npy) except for those listed in the KEEP_FILE file. The files listed in KEEP_FILE and their history will be maintained.

Of course, you can replace the first 5 lines with anything you want to get the list of files you want to keeps.

Side effect: It is possible that was two separate commit will be merged into one, if the only difference was a file that is now gone. This can also change the topology of your branches too. It will still be "as correct as possible", only some commit messages would disappear. Of course, it is unlikely it will happen, but just an FYI

[andyneff]

Other things I've tried that did NOT work out

• I had this idea of using git-replace. This has an advantage of efficiently creating all the lfs pointers once and only once. This WORKS (after a filter-branch to update .gitconfig/.gitattribute), but I have a hard time making the changes permanent. git-filter-branch does not appear to support blob replacement (only commit replacement).
  while read line; do
  LINE=($line)
  OBJECT_HASH=${LINE[0]}
  FILENAME="${LINE[@]:1}"
  MATCH=$(git check-attr filter ${FILENAME} | grep 'filter: lfs' | sed -r 's/(.): filter: lfs/\1/')
  if [ ! "${MATCH}" == "" ]; then
  POINTER_HASH=$(git show ${OBJECT_HASH} | git lfs clean | git hash-object -w --stdin)
  echo New file ${FILENAME} ${OBJECT_HASH} ${POINTER_HASH}
  git replace ${OBJECT_HASH} ${POINTER_HASH}
  fi
  done < <(git rev-list --objects --all | \grep "[0-9a-f] .")
• Solutions involving git rebase --interactive, this is possible, but only works for one branch. Plus it is prone to merge conflicts in cases where your repo uses .gitattributes
  env GIT_SEQUENCE_EDITOR="sed -i s:^pick:edit:" git rebase --interactive --root
  while : ; do
  git lfs ls-files > .gitlfs_converted
  for file in $(git ls-files | xargs git check-attr filter | grep "filter: lfs" | sed -r "s/(.*): filter: lfs/\1/"); do
  new_file=1
  while read converted; do
  if [ "${converted}" == "${file}" ]; then
  new_file=0
  fi
  done < .gitlfs_converted
  if [[ ${new_file} == 1 ]]; then
  echo "Processing ${file}"
  git rm --cached ${file}
  echo "Adding $file"
  git add ${file}
  fi
  done
  git commit --no-edit --amend
  rm .gitlfs_converted
  git rebase --continue || break
  done

[bozaro]

I write simple java code which can convert repository for LFS usage: https://github.com/bozaro/git-lfs-migrate
May be it be usefull for somebody.

[vmrob]

Version of the first script that works on OS X with files that contains spaces (but not newline characters):

git filter-branch --prune-empty --tree-filter '
git lfs track "*.zip"
git lfs track "*.exe"
git add .gitattributes

git ls-files -z | xargs -0 git check-attr filter | grep "filter: lfs" | sed -E "s/(.*): filter: lfs/\1/" | tr "\n" "\0" | while read -r -d $'"'\0'"' file; do
    echo "Processing ${file}"

    git rm -f --cached "${file}"
    echo "Adding $file lfs style"
    git add "${file}"
done

' --tag-name-filter cat -- --all

The most unusual part is while read -r -d $'"'\0'"'

The parameter to read, -d, is $'\0', but to escape the single quote inside the block that is already single quoted, we end the quote, open a new double quote, then use single quotes, close the double quote, and then open a single quote back up. Shell escaping wasn't always the easiest..

[rtyley]

As @tlbtlbtlb mentioned, the BFG is much faster than git filter-branch for rewriting history - and I've added explicit Git LFS support with BFG v1.12.5:

$ bfg --convert-to-git-lfs '*.{exe,dll}' --no-blob-protection

Incidentally, the git-lfs-migrate code by @bozaro is quite interesting - it looks like it does an equivalent job, and maybe at equivalent speed - will check it out when I get a chance.

[bozaro]

@rtyley

I think both of these projects need to work roughly the same speed.
By the way, I'll stand in the common part of my projects (git-as-svn and git-lfs-migrate) in a separate git-lfs-java project.