As a server Shortcuts/Optimisations? #627
Description
Hi
I'm trying to write a caching proxy server that we can use to cache locally various repos from github (and other places) that we use heavily but I'm running into a performance issue
I'm more than happy to concede that we won't beat github for speed but I'm finding this to be a great deal slower.
This is a greatly simplified version of what I'm running in the main codebase
package main
import (
"compress/gzip"
"io"
"net/http"
"os"
"path"
"strings"
"gopkg.in/src-d/go-git.v4/plumbing/protocol/packp"
"gopkg.in/src-d/go-git.v4/plumbing/transport"
"gopkg.in/src-d/go-git.v4/plumbing/transport/server"
)
func main() {
wd, _ := os.Getwd()
http.ListenAndServe(":8822", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "no-cache")
s := strings.SplitN(strings.TrimLeft(r.URL.Path, "/"), "/", 2)
ep, _ := transport.NewEndpoint(path.Join("file://", wd, s[0]))
ups, _ := server.DefaultServer.NewUploadPackSession(ep, nil)
if strings.Contains(r.URL.Path, "info") {
advs, _ := ups.AdvertisedReferences()
advs.Prefix = [][]byte{
[]byte("# service=git-upload-pack"),
[]byte(""),
}
w.Header().Set("Content-Type", "application/x-git-upload-pack-advertisement")
advs.Encode(w)
return
}
defer r.Body.Close()
var rdr io.ReadCloser = r.Body
if r.Header.Get("Content-Encoding") == "gzip" {
rdr, _ = gzip.NewReader(r.Body)
}
upakreq := packp.NewUploadPackRequest()
upakreq.Decode(rdr)
up, _ := ups.UploadPack(r.Context(), upakreq)
w.Header().Set("Content-Type", "application/x-git-upload-pack-result")
up.Encode(w)
}))
}
if you clone the aws/aws-sdk-go repo git clone --quiet --mirror https://github.com/aws/aws-sdk-go aws-sdk-go into the same directory as you put this go file
Then go run main.go you can do the following tests
$ time git clone --quiet https://github.com/aws/aws-sdk-go
real 0m23.816s
user 0m4.047s
sys 0m1.289s
$ time git clone --quiet http://localhost:8822/aws-sdk-go
real 2m18.718s
user 0m3.793s
sys 0m1.165s
The time cost here is entirely in cloning from scratch, pulling seems plenty fast.
I did profile my code and I found it spent most of it's time in encoder.go
Can anyone think of any way to shortcut this process for a clone if not optimise the code?
If instead of using git clone --mirror to create the aws-sdk-dir and I use go-git to clone it I even get pre-packed files
./objects/pack/pack-41174c775d8b7f517d5db3c20d52b0e5379fe9de.idx
./objects/pack/pack-41174c775d8b7f517d5db3c20d52b0e5379fe9de.pack
Perhaps for a fresh clone I can just ship that?