Git.exe Processes Hanging / Not Cleaning Up

[atom0s]

Gitea Version

1.16.1

Git Version

2.35.1.windows.2

Operating System

Windows Server 2012 R2

How are you running Gitea?

• Using the prebuilt package for 1.16.1 - gitea-1.16.1-windows-4.0-amd64.exe
• Running via IIS with a forward proxy to pass traffic to the Gitea instance.
• Gitea is running as a system service.

Database

MySQL

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Description

When running for an extended period of time, we are noticing that git.exe instances are left open/hanging. After about 25-30 instances are left open, the site just starts throwing 500 errors. It becomes impossible to do anything on the site outside of viewing the main page and admin panel.

This issue seems to be an exact copy of this: #17138

I am seeing the same thing.

Monitor page shows tons of entries for things like this:

C:\Program Files\Git\cmd\git.exe -c credential.helper= -c protocol.version=2 -c uploadpack.allowfilter=true check-attr --stdin -z --cached linguist-vendored linguist-generated linguist-language gitlab-language -- [repo_path: PATH\TO\REPO\HERE.git]

There are also other monitored processes stuck:

GET: /PATH/TO/REPO/HERE/commits/commit/COMMIT_HASH_HERE/path/to/file.h?lang=pt-BR
3 hours ago
  C:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: PATH\TO\REPO\HERE.git] (modules/context/repo.go:559)
  3 hours ago
  C:\Program Files\Git\cmd\git.exe cat-file --batch [repo_path: PATH\TO\REPO\HERE.git] (modules/context/repo.go:559)
  3 hours ago

GET: /PATH/TO/REPO/HERE/commit/COMMIT_HASH_HERE.diff
2 hours ago
  C:\Program Files\Git\cmd\git.exe cat-file --batch [repo_path: PATH\TO\REPO\HERE.git] (modules/context/repo.go:559)
  2 hours ago
  C:\Program Files\Git\cmd\git.exe cat-file --batch-check [repo_path: PATH\TO\REPO\HERE.git] (modules/context/repo.go:559)
  2 hours ago

C:\Program Files\Git\cmd\git.exe cat-file --batch [repo_path: PATH\TO\REPO\HERE.git] (modules/git/repo_base_nogogit.go:41)

And so on. The main one is the first reported entry, there are lots of those ones.

This is a bit rough to diagnose as it does not happen immediately. It takes a few hours before we start seeing the instances begin hanging.

Screenshots

No response

[zeripath]

My suspicion is that it's not that the git files are hanging - the request opening them is.

If you press the cancel button although they'll stay in the process hierarchy on monitor the command they've opened should cancel.

The problem is that the request is hanging somewhere.

[atom0s]

Not sure if it is helpful to figuring this out, but this wasn't happening before updating last night.

We were a bit outdated, running on 1.13.6 before. Updated to 1.16.1 (latest) last night and it started happening then. Before, we had no issues at all aside from a weird read lock bug that was only affecting 1 repo. We opt'd to update to try to fix that problem and now running into this new one.

git.exe is stuck open though either way. There are like 30 processes just sitting idle.

If you press the cancel button although they'll stay in the process hierarchy on monitor the command they've opened should cancel.

This isn't really an option to sit and babysit the site 24/7. This is constantly happening.

[CrysMia]

The problem is that the request is hanging somewhere.

Seems to be related to the index file in the temp folder:

2022/02/12 18:39:57 ...es/git/repo_index.go:59:func1() [E] failed to remove tmp index file: remove C:\Users\o3o\AppData\Local\Temp\index3379606138: The process cannot access the file because it is being used by another process.

Can be reproduced on every push.

[atom0s]

This is what we have seen in our log file, error wise:

Most of the errors are this: (hundreds of them with different temp file names.)

...es/git/repo_index.go:59:func1() [E] failed to remove tmp index file: remove C:\Windows\TEMP\index489361462: The process cannot access the file because it is being used by another process.

Unsure on these ones:

...s/context/context.go:290:PlainTextBytes() [E] PlainTextBytes: Not found.

...ules/context/repo.go:908:func1() [E] RepoRef invalid repo: branch or tag not exist: 

...web/repo/download.go:94:SingleDownload() [E] ServeBlob: write tcp 127.0.0.1:3031->127.0.0.1:12079: wsasend: An existing connection was forcibly closed by the remote host.

Bunch of name lookup failures, not harmful, just noting they are happening:

.../web/user/profile.go:35:GetUserByName() [E] GetUserByName: user redirect does not exist [name: NAME_HERE]

If I had to guess as well, the ones from git/repo_index.go:59:func1() appear to be happening the most and potentially causing the problem.

[atom0s]

After a restart and flushed log, here is what's present as the git.exe's start piling up now:

	Line 4332: 2022/02/12 12:57:53 ...ules/context/repo.go:908:func1() [E] RepoRef invalid repo: branch or tag not exist: 
	Line 4907: 2022/02/12 13:00:29 ...s/context/context.go:290:PlainTextBytes() [E] PlainTextBytes: Not found.
	Line 5070: 2022/02/12 13:00:42 ...es/git/repo_index.go:59:func1() [E] failed to remove tmp index file: remove C:\Windows\TEMP\index2623699774: The process cannot access the file because it is being used by another process.
	Line 5391: 2022/02/12 13:01:18 .../web/user/profile.go:35:GetUserByName() [E] GetUserByName: user redirect does not exist [name: robots.txt]
	Line 5392: 2022/02/12 13:01:18 ...s/context/context.go:290:PlainTextBytes() [E] PlainTextBytes: Not found.
	Line 5738: 2022/02/12 13:02:31 ...es/git/repo_index.go:59:func1() [E] failed to remove tmp index file: remove C:\Windows\TEMP\index245458179: The process cannot access the file because it is being used by another process.
	Line 5908: 2022/02/12 13:03:01 .../web/user/profile.go:35:GetUserByName() [E] GetUserByName: user redirect does not exist [name: apogee]
	Line 6289: 2022/02/12 13:03:59 ...es/git/repo_index.go:59:func1() [E] failed to remove tmp index file: remove C:\Windows\TEMP\index1505406057: The process cannot access the file because it is being used by another process.
	Line 6300: 2022/02/12 13:04:00 .../web/user/profile.go:35:GetUserByName() [E] GetUserByName: user redirect does not exist [name: serviceworker.js]
	Line 6301: 2022/02/12 13:04:00 ...s/context/context.go:290:PlainTextBytes() [E] PlainTextBytes: Not found.

There are currently 3 stuck instances, which aligns to the ...es/git/repo_index.go:59:func1() tmp index file error counts.

[uncled1023]

I'm also seeing a bunch of processes piling on, one for each repo. As well, also seeing the unable to remove tmp index file errors.

After a few hours, every repo is a 500, but a restart fixes it all like above.

These are the processes i'm seeing being backed up: C:\Program Files\Git\cmd\git.exe -c credential.helper= check-attr --stdin -z --cached linguist-vendored linguist-generated linguist-language gitlab-language -- [repo_path: C:\Path\To\Repositories\my_repo.git]

[zeripath]

Ok it looks like there has to be a deadlock here.

The logs above aren't really pointing very clearly to what the problem could be or is.

---

Hmm...

Any chance you enable pprof?

[server]
...
ENABLE_PPROF=true

Then whilst there is one of these blocked processes could you run on the server: (and preferably nothing else going on)

curl http://127.0.0.1:6060/debug/pprof/goroutine?debug=2

This should help us to figure out where a deadlock could possibly be occuring by examining the goroutines stack.

[atom0s]

Here's a pprof log:

PS C:\Users\Administrator> curl http://127.0.0.1:6060/debug/pprof/goroutine?debug=2


StatusCode        : 200
StatusDescription : OK
Content           : goroutine 4140 [running]:
                    runtime/pprof.writeGoroutineStacks({0x4521b80, 0xc000492460})
                        /usr/local/go/src/runtime/pprof/pprof.go:693 +0x70
                    runtime/pprof.writeGoroutine({0x4521b80, 0xc000492460}, 0x0)...
RawContent        : HTTP/1.1 200 OK
                    X-Content-Type-Options: nosniff
                    Transfer-Encoding: chunked
                    Content-Type: text/plain; charset=utf-8
                    Date: Sun, 13 Feb 2022 09:50:00 GMT

                    goroutine 4140 [running]:
                    runtime/pprof.wr...
Forms             : {}
Headers           : {[X-Content-Type-Options, nosniff], [Transfer-Encoding, chunked], [Content-Type, text/plain;
                    charset=utf-8], [Date, Sun, 13 Feb 2022 09:50:00 GMT]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 28942

Here is also level 1 trace info:

goroutine profile: total 35
4 @ 0xd3d196 0xd4c892 0x1e1fa15 0xd6c3a1
#	0x1e1fa14	github.com/blevesearch/bleve_index_api.AnalysisWorker+0x94	/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:46

3 @ 0xd3d196 0xd32b53 0xd65a29 0xde4ff2 0xde6625 0xde743f 0xe34749 0xe45e25 0x1083ecd 0xf179a3 0xf1856f 0xf187c7 0x101dad9 0x107f3d9 0x107f3da 0x1085285 0x1089625 0xd6c3a1
#	0xd65a28	internal/poll.runtime_pollWait+0x88		/usr/local/go/src/runtime/netpoll.go:234
#	0xde4ff1	internal/poll.(*pollDesc).wait+0x31		/usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#	0xde6624	internal/poll.execIO+0xe4			/usr/local/go/src/internal/poll/fd_windows.go:175
#	0xde743e	internal/poll.(*FD).Read+0x25e			/usr/local/go/src/internal/poll/fd_windows.go:441
#	0xe34748	net.(*netFD).Read+0x28				/usr/local/go/src/net/fd_posix.go:56
#	0xe45e24	net.(*conn).Read+0x44				/usr/local/go/src/net/net.go:183
#	0x1083ecc	net/http.(*connReader).Read+0x16c		/usr/local/go/src/net/http/server.go:780
#	0xf179a2	bufio.(*Reader).fill+0x102			/usr/local/go/src/bufio/bufio.go:101
#	0xf1856e	bufio.(*Reader).ReadSlice+0x2e			/usr/local/go/src/bufio/bufio.go:360
#	0xf187c6	bufio.(*Reader).ReadLine+0x26			/usr/local/go/src/bufio/bufio.go:389
#	0x101dad8	net/textproto.(*Reader).readLineSlice+0x98	/usr/local/go/src/net/textproto/reader.go:57
#	0x107f3d8	net/textproto.(*Reader).ReadLine+0x78		/usr/local/go/src/net/textproto/reader.go:38
#	0x107f3d9	net/http.readRequest+0x79			/usr/local/go/src/net/http/request.go:1029
#	0x1085284	net/http.(*conn).readRequest+0x224		/usr/local/go/src/net/http/server.go:966
#	0x1089624	net/http.(*conn).serve+0x864			/usr/local/go/src/net/http/server.go:1856

1 @ 0xd0484a 0xd687d4 0xf0ab85 0x151f629 0x1522ef6 0xd6c3a1
#	0xd687d3	syscall.Syscall+0xf3							/usr/local/go/src/runtime/syscall_windows.go:483
#	0xf0ab84	golang.org/x/sys/windows.StartServiceCtrlDispatcher+0x64		/go/pkg/mod/golang.org/x/sys@v0.0.0-20211117180635-dee7805ff2e1/windows/zsyscall_windows.go:1241
#	0x151f628	golang.org/x/sys/windows/svc.Run+0x128					/go/pkg/mod/golang.org/x/sys@v0.0.0-20211117180635-dee7805ff2e1/windows/svc/service.go:293
#	0x1522ef5	code.gitea.io/gitea/modules/graceful.(*Manager).start.func1+0x35	/source/modules/graceful/manager_windows.go:100

1 @ 0xd0484a 0xd6893a 0xdbf41e 0xdb7ece 0xde7394 0xdf0fbe 0xdf0fb6 0xdafdd2 0xf2f45a 0xf2f446 0xf303e5 0xd6c3a1
#	0xd68939	syscall.Syscall6+0xf9				/usr/local/go/src/runtime/syscall_windows.go:497
#	0xdbf41d	syscall.ReadFile+0xbd				/usr/local/go/src/syscall/zsyscall_windows.go:1024
#	0xdb7ecd	syscall.Read+0x2d				/usr/local/go/src/syscall/syscall_windows.go:380
#	0xde7393	internal/poll.(*FD).Read+0x1b3			/usr/local/go/src/internal/poll/fd_windows.go:427
#	0xdf0fbd	os.(*File).read+0x5d				/usr/local/go/src/os/file_posix.go:32
#	0xdf0fb5	os.(*File).Read+0x55				/usr/local/go/src/os/file.go:119
#	0xdafdd1	io.copyBuffer+0x1b1				/usr/local/go/src/io/io.go:423
#	0xf2f459	io.Copy+0x39					/usr/local/go/src/io/io.go:382
#	0xf2f445	os/exec.(*Cmd).writerDescriptor.func1+0x25	/usr/local/go/src/os/exec/exec.go:311
#	0xf303e4	os/exec.(*Cmd).Start.func1+0x24			/usr/local/go/src/os/exec/exec.go:441

1 @ 0xd0484a 0xd6893a 0xdbf41e 0xdb7ece 0xde7394 0xdf0fbe 0xdf0fb6 0xe55ad8 0xdafd6b 0xf2f45a 0xf2f446 0xf303e5 0xd6c3a1
#	0xd68939	syscall.Syscall6+0xf9				/usr/local/go/src/runtime/syscall_windows.go:497
#	0xdbf41d	syscall.ReadFile+0xbd				/usr/local/go/src/syscall/zsyscall_windows.go:1024
#	0xdb7ecd	syscall.Read+0x2d				/usr/local/go/src/syscall/syscall_windows.go:380
#	0xde7393	internal/poll.(*FD).Read+0x1b3			/usr/local/go/src/internal/poll/fd_windows.go:427
#	0xdf0fbd	os.(*File).read+0x5d				/usr/local/go/src/os/file_posix.go:32
#	0xdf0fb5	os.(*File).Read+0x55				/usr/local/go/src/os/file.go:119
#	0xe55ad7	bytes.(*Buffer).ReadFrom+0x97			/usr/local/go/src/bytes/buffer.go:204
#	0xdafd6a	io.copyBuffer+0x14a				/usr/local/go/src/io/io.go:409
#	0xf2f459	io.Copy+0x39					/usr/local/go/src/io/io.go:382
#	0xf2f445	os/exec.(*Cmd).writerDescriptor.func1+0x25	/usr/local/go/src/os/exec/exec.go:311
#	0xf303e4	os/exec.(*Cmd).Start.func1+0x24			/usr/local/go/src/os/exec/exec.go:441

1 @ 0xd3d196 0xd07a0c 0xd07438 0xf30608 0x13c54f6 0x13e1a3b 0x13e1875 0x1d54845 0xd6c3a1
#	0xf30607	os/exec.(*Cmd).Wait+0x187								/usr/local/go/src/os/exec/exec.go:515
#	0x13c54f5	code.gitea.io/gitea/modules/git.(*Command).RunWithContext+0x6b5				/source/modules/git/command.go:189
#	0x13e1a3a	code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutEnvFullPipelineFunc+0x25a	/source/modules/git/command.go:113
#	0x13e1874	code.gitea.io/gitea/modules/git.(*CheckAttributeReader).Run+0x94			/source/modules/git/repo_attribute.go:185
#	0x1d54844	code.gitea.io/gitea/services/gitdiff.GetDiff.func1+0x44					/source/services/gitdiff/gitdiff.go:1415

1 @ 0xd3d196 0xd07a0c 0xd07478 0x1d9ab11 0xd6c3a1
#	0x1d9ab10	code.gitea.io/gitea/modules/queue.(*WrappedQueue).Run.func1+0x70	/source/modules/queue/queue_wrapped.go:233

1 @ 0xd3d196 0xd32b53 0xd65a29 0xde4ff2 0xde6625 0xde743f 0xe34749 0xe45e25 0x1083a1f 0xd6c3a1
#	0xd65a28	internal/poll.runtime_pollWait+0x88		/usr/local/go/src/runtime/netpoll.go:234
#	0xde4ff1	internal/poll.(*pollDesc).wait+0x31		/usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#	0xde6624	internal/poll.execIO+0xe4			/usr/local/go/src/internal/poll/fd_windows.go:175
#	0xde743e	internal/poll.(*FD).Read+0x25e			/usr/local/go/src/internal/poll/fd_windows.go:441
#	0xe34748	net.(*netFD).Read+0x28				/usr/local/go/src/net/fd_posix.go:56
#	0xe45e24	net.(*conn).Read+0x44				/usr/local/go/src/net/net.go:183
#	0x1083a1e	net/http.(*connReader).backgroundRead+0x3e	/usr/local/go/src/net/http/server.go:672

1 @ 0xd3d196 0xd32b53 0xd65a29 0xde4ff2 0xde6625 0xde9dcd 0xdea136 0xe35ea5 0xe4c628 0xe4b55d 0x1524d07 0x108e2d4 0x152478b 0x15242e5 0x152593d 0x28b48a5 0x28b48a6 0x28b4033 0x2281828 0x2282612 0x227f9e5 0x28eb193 0xd3cdf7 0xd6c3a1
#	0xd65a28	internal/poll.runtime_pollWait+0x88					/usr/local/go/src/runtime/netpoll.go:234
#	0xde4ff1	internal/poll.(*pollDesc).wait+0x31					/usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#	0xde6624	internal/poll.execIO+0xe4						/usr/local/go/src/internal/poll/fd_windows.go:175
#	0xde9dcc	internal/poll.(*FD).acceptOne+0x6c					/usr/local/go/src/internal/poll/fd_windows.go:810
#	0xdea135	internal/poll.(*FD).Accept+0x1d5					/usr/local/go/src/internal/poll/fd_windows.go:844
#	0xe35ea4	net.(*netFD).accept+0x64						/usr/local/go/src/net/fd_windows.go:139
#	0xe4c627	net.(*TCPListener).accept+0x27						/usr/local/go/src/net/tcpsock_posix.go:140
#	0xe4b55c	net.(*TCPListener).AcceptTCP+0x3c					/usr/local/go/src/net/tcpsock.go:249
#	0x1524d06	code.gitea.io/gitea/modules/graceful.(*wrappedListener).Accept+0x46	/source/modules/graceful/server.go:180
#	0x108e2d3	net/http.(*Server).Serve+0x393						/usr/local/go/src/net/http/server.go:3002
#	0x152478a	code.gitea.io/gitea/modules/graceful.(*Server).Serve+0x12a		/source/modules/graceful/server.go:133
#	0x15242e4	code.gitea.io/gitea/modules/graceful.(*Server).ListenAndServe+0x1c4	/source/modules/graceful/server.go:95
#	0x152593c	code.gitea.io/gitea/modules/graceful.HTTPListenAndServe+0x3c		/source/modules/graceful/server_http.go:33
#	0x28b48a4	code.gitea.io/gitea/cmd.runHTTP+0x424					/source/cmd/web_graceful.go:19
#	0x28b48a5	code.gitea.io/gitea/cmd.listen+0x425					/source/cmd/web.go:224
#	0x28b4032	code.gitea.io/gitea/cmd.runWeb+0x832					/source/cmd/web.go:171
#	0x2281827	github.com/urfave/cli.HandleAction+0xa7					/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:524
#	0x2282611	github.com/urfave/cli.Command.Run+0x651					/go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:173
#	0x227f9e4	github.com/urfave/cli.(*App).Run+0x704					/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:277
#	0x28eb192	main.main+0xcf2								/source/main.go:115
#	0xd3cdf6	runtime.main+0x216							/usr/local/go/src/runtime/proc.go:255

1 @ 0xd3d196 0xd32b53 0xd65a29 0xde4ff2 0xde6625 0xde9dcd 0xdea136 0xe35ea5 0xe4c628 0xe4b6bd 0x108e2d4 0x108defd 0x28b6e85 0x28b6e86 0xd6c3a1
#	0xd65a28	internal/poll.runtime_pollWait+0x88		/usr/local/go/src/runtime/netpoll.go:234
#	0xde4ff1	internal/poll.(*pollDesc).wait+0x31		/usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#	0xde6624	internal/poll.execIO+0xe4			/usr/local/go/src/internal/poll/fd_windows.go:175
#	0xde9dcc	internal/poll.(*FD).acceptOne+0x6c		/usr/local/go/src/internal/poll/fd_windows.go:810
#	0xdea135	internal/poll.(*FD).Accept+0x1d5		/usr/local/go/src/internal/poll/fd_windows.go:844
#	0xe35ea4	net.(*netFD).accept+0x64			/usr/local/go/src/net/fd_windows.go:139
#	0xe4c627	net.(*TCPListener).accept+0x27			/usr/local/go/src/net/tcpsock_posix.go:140
#	0xe4b6bc	net.(*TCPListener).Accept+0x3c			/usr/local/go/src/net/tcpsock.go:262
#	0x108e2d3	net/http.(*Server).Serve+0x393			/usr/local/go/src/net/http/server.go:3002
#	0x108defc	net/http.(*Server).ListenAndServe+0x7c		/usr/local/go/src/net/http/server.go:2931
#	0x28b6e84	net/http.ListenAndServe+0x64			/usr/local/go/src/net/http/server.go:3185
#	0x28b6e85	code.gitea.io/gitea/cmd.runWeb.func2+0x65	/source/cmd/web.go:146

1 @ 0xd3d196 0xd4c892 0x10c2a11 0xd6c3a1
#	0x10c2a10	code.gitea.io/gitea/modules/log.(*ChannelledLog).Start+0x90	/source/modules/log/event.go:70

1 @ 0xd3d196 0xd4c892 0x10c3dfc 0xd6c3a1
#	0x10c3dfb	code.gitea.io/gitea/modules/log.(*MultiChannelledLog).Start+0x27b	/source/modules/log/event.go:300

1 @ 0xd3d196 0xd4c892 0x11d8f6d 0xd6c3a1
#	0x11d8f6c	github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain+0xcc	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206

1 @ 0xd3d196 0xd4c892 0x11fe41e 0xd6c3a1
#	0x11fe41d	github.com/syndtr/goleveldb/leveldb.(*DB).compactionError+0x15d	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90

1 @ 0xd3d196 0xd4c892 0x12037f9 0xd6c3a1
#	0x12037f8	github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction+0x118	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762

1 @ 0xd3d196 0xd4c892 0x1204025 0xd6c3a1
#	0x1204024	github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction+0x684	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825

1 @ 0xd3d196 0xd4c892 0x120794e 0xd6c3a1
#	0x120794d	github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain+0xad	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101

1 @ 0xd3d196 0xd4c892 0x12e69b0 0xd6c3a1
#	0x12e69af	github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1+0xaf	/go/pkg/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:614

1 @ 0xd3d196 0xd4c892 0x151f288 0xd572d4 0xd04d9a 0xd04a26 0xd6c157 0xd6c3a1
#	0x151f287	golang.org/x/sys/windows/svc.serviceMain+0x427	/go/pkg/mod/golang.org/x/sys@v0.0.0-20211117180635-dee7805ff2e1/windows/svc/service.go:245
#	0xd572d3	runtime.callbackWrap+0x133			/usr/local/go/src/runtime/syscall_windows.go:378
#	0xd04d99	runtime.cgocallbackg1+0x299			/usr/local/go/src/runtime/cgocall.go:306
#	0xd04a25	runtime.cgocallbackg+0x105			/usr/local/go/src/runtime/cgocall.go:232
#	0xd6c156	runtime.cgocallback+0xd6			/usr/local/go/src/runtime/asm_amd64.s:915

1 @ 0xd3d196 0xd4c892 0x152314f 0x151f4c5 0xd6c3a1
#	0x152314e	code.gitea.io/gitea/modules/graceful.(*Manager).Execute+0x22e	/source/modules/graceful/manager_windows.go:133
#	0x151f4c4	golang.org/x/sys/windows/svc.serviceMain.func2+0x44		/go/pkg/mod/golang.org/x/sys@v0.0.0-20211117180635-dee7805ff2e1/windows/svc/service.go:233

1 @ 0xd3d196 0xd4c892 0x15252b1 0xd6c3a1
#	0x15252b0	code.gitea.io/gitea/modules/graceful.(*Server).awaitShutdown+0xb0	/source/modules/graceful/server_hooks.go:16

1 @ 0xd3d196 0xd4c892 0x1d933fa 0x1d92ed7 0x1d9a9ee 0x1520a65 0xd6c3a1
#	0x1d933f9	code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).readToChan+0x319		/source/modules/queue/queue_bytefifo.go:151
#	0x1d92ed6	code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).Run+0x256			/source/modules/queue/queue_bytefifo.go:120
#	0x1d9a9ed	code.gitea.io/gitea/modules/queue.(*WrappedQueue).Run+0x2cd			/source/modules/queue/queue_wrapped.go:242
#	0x1520a64	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownFns+0x124	/source/modules/graceful/manager.go:83

1 @ 0xd3d196 0xd4c892 0x21fa68c 0x15213f0 0xd6c3a1
#	0x21fa68b	code.gitea.io/gitea/services/webhook.DeliverHooks+0x28b				/source/services/webhook/deliver.go:227
#	0x15213ef	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext+0xaf	/source/modules/graceful/manager.go:139

1 @ 0xd3d196 0xd4c892 0x25cce9f 0x15213f0 0xd6c3a1
#	0x25cce9e	code.gitea.io/gitea/modules/eventsource.(*Manager).Run+0xde			/source/modules/eventsource/manager_run.go:32
#	0x15213ef	code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext+0xaf	/source/modules/graceful/manager.go:139

1 @ 0xd3d196 0xd4c892 0x25e35e5 0xd6c3a1
#	0x25e35e4	github.com/gogs/cron.(*Cron).run+0x5a4	/go/pkg/mod/github.com/gogs/cron@v0.0.0-20171120032916-9f6c956d3e14/cron.go:197

1 @ 0xd3d196 0xd4c892 0x27b45c9 0x25dbec5 0x108a7ef 0x1df1776 0x108a7ef 0x25dc6a6 0x108a7ef 0x25dc6a6 0x108a7ef 0x25dc6a6 0x108a7ef 0x260afc5 0x108a7ef 0x25dc6a6 0x108a7ef 0x1e02d11 0x108a7ef 0x27b9f11 0x108a7ef 0x1def5ad 0x1df0f5c 0x108a7ef 0x1df1776 0x108a7ef 0x27bcc1b 0x108a7ef 0x1bf1ede 0x108a7ef 0x1c9fa9c 0x108a7ef
#	0x27b45c8	code.gitea.io/gitea/routers/web/events.Events+0x6c8			/source/routers/web/events/events.go:78
#	0x25dbec4	code.gitea.io/gitea/modules/web.Wrap.func1+0x1c4			/source/modules/web/route.go:74
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x1df1775	github.com/go-chi/chi/v5.(*Mux).routeHTTP+0x215				/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:442
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x25dc6a5	code.gitea.io/gitea/modules/web.Middle.func1.1+0xe5			/source/modules/web/route.go:113
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x25dc6a5	code.gitea.io/gitea/modules/web.Middle.func1.1+0xe5			/source/modules/web/route.go:113
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x25dc6a5	code.gitea.io/gitea/modules/web.Middle.func1.1+0xe5			/source/modules/web/route.go:113
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x260afc4	github.com/go-chi/chi/v5/middleware.GetHead.func1+0x1e4			/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/middleware/get_head.go:37
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x25dc6a5	code.gitea.io/gitea/modules/web.Middle.func1.1+0xe5			/source/modules/web/route.go:113
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x1e02d10	code.gitea.io/gitea/modules/context.Contexter.func1.1+0x1990		/source/modules/context/context.go:731
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x27b9f10	github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1+0x290	/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:342
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x1def5ac	github.com/go-chi/chi/v5.(*Mux).ServeHTTP+0x48c				/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:71
#	0x1df0f5b	github.com/go-chi/chi/v5.(*Mux).Mount.func1+0x19b			/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:314
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x1df1775	github.com/go-chi/chi/v5.(*Mux).routeHTTP+0x215				/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:442
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x27bcc1a	code.gitea.io/gitea/routers/web.Recovery.func1.1+0x9a			/source/routers/web/base.go:169
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x1bf1edd	gitea.com/go-chi/session.Sessioner.func1.1+0x2fd			/go/pkg/mod/gitea.com/go-chi/session@v0.0.0-20211218221615-e3605d8b28b8/session.go:257
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047
#	0x1c9fa9b	code.gitea.io/gitea/modules/public.AssetsHandler.func1.1+0x3bb		/source/modules/public/public.go:42
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e					/usr/local/go/src/net/http/server.go:2047

1 @ 0xd3d196 0xd4c892 0xe1ea5d 0xd6c3a1
#	0xe1ea5c	database/sql.(*DB).connectionCleaner+0xbc	/usr/local/go/src/database/sql/sql.go:1068

1 @ 0xd3d196 0xd4c892 0xe1f533 0xd6c3a1
#	0xe1f532	database/sql.(*DB).connectionOpener+0x92	/usr/local/go/src/database/sql/sql.go:1196

1 @ 0xd655e5 0x259bcd5 0x259baed 0x2598c8b 0x25c981a 0x25ca3ce 0x108a7ef 0x108c0e9 0x108dd5b 0x10898c8 0xd6c3a1
#	0xd655e4	runtime/pprof.runtime_goroutineProfileWithLabels+0x24	/usr/local/go/src/runtime/mprof.go:746
#	0x259bcd4	runtime/pprof.writeRuntimeProfile+0xb4			/usr/local/go/src/runtime/pprof/pprof.go:724
#	0x259baec	runtime/pprof.writeGoroutine+0x4c			/usr/local/go/src/runtime/pprof/pprof.go:684
#	0x2598c8a	runtime/pprof.(*Profile).WriteTo+0x14a			/usr/local/go/src/runtime/pprof/pprof.go:331
#	0x25c9819	net/http/pprof.handler.ServeHTTP+0x499			/usr/local/go/src/net/http/pprof/pprof.go:253
#	0x25ca3cd	net/http/pprof.Index+0x12d				/usr/local/go/src/net/http/pprof/pprof.go:371
#	0x108a7ee	net/http.HandlerFunc.ServeHTTP+0x2e			/usr/local/go/src/net/http/server.go:2047
#	0x108c0e8	net/http.(*ServeMux).ServeHTTP+0x148			/usr/local/go/src/net/http/server.go:2425
#	0x108dd5a	net/http.serverHandler.ServeHTTP+0x43a			/usr/local/go/src/net/http/server.go:2879
#	0x10898c7	net/http.(*conn).serve+0xb07				/usr/local/go/src/net/http/server.go:1930

1 @ 0xd6c3a1

[atom0s]

Here's the full goroutine stack dump as well:

goroutine 4692 [running]:
runtime/pprof.writeGoroutineStacks({0x4521b80, 0xc00946e620})
	/usr/local/go/src/runtime/pprof/pprof.go:693 +0x70
runtime/pprof.writeGoroutine({0x4521b80, 0xc00946e620}, 0x0)
	/usr/local/go/src/runtime/pprof/pprof.go:682 +0x2b
runtime/pprof.(*Profile).WriteTo(0x3044540, {0x4521b80, 0xc00946e620}, 0xc)
	/usr/local/go/src/runtime/pprof/pprof.go:331 +0x14b
net/http/pprof.handler.ServeHTTP({0xc0038720a1, 0x5c259e0}, {0x4537278, 0xc00946e620}, 0xc003872094)
	/usr/local/go/src/net/http/pprof/pprof.go:253 +0x49a
net/http/pprof.Index({0x4537278, 0xc00946e620}, 0xc0069de600)
	/usr/local/go/src/net/http/pprof/pprof.go:371 +0x12e
net/http.HandlerFunc.ServeHTTP(0x0, {0x4537278, 0xc00946e620}, 0x0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc0038720ab, {0x4537278, 0xc00946e620}, 0xc0069de600)
	/usr/local/go/src/net/http/server.go:2425 +0x149
net/http.serverHandler.ServeHTTP({0xc003855a10}, {0x4537278, 0xc00946e620}, 0xc0069de600)
	/usr/local/go/src/net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc00a1560a0, {0x4550ca8, 0xc002458de0})
	/usr/local/go/src/net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3034 +0x4e8

goroutine 1 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0xd7696474c8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0x200000003, 0xd0ea94, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.execIO(0xc0079ccc98, 0xc00a487128)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe5
internal/poll.(*FD).acceptOne(0xc0079ccc80, 0x514, {0xc004a8ec30, 0x0, 0x0}, 0x0)
	/usr/local/go/src/internal/poll/fd_windows.go:810 +0x6d
internal/poll.(*FD).Accept(0xc0079ccc80, 0xc00a487300)
	/usr/local/go/src/internal/poll/fd_windows.go:844 +0x1d6
net.(*netFD).accept(0xc0079ccc80)
	/usr/local/go/src/net/fd_windows.go:139 +0x65
net.(*TCPListener).accept(0xc0049550b0)
	/usr/local/go/src/net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).AcceptTCP(0xc0049550b0)
	/usr/local/go/src/net/tcpsock.go:249 +0x3d
code.gitea.io/gitea/modules/graceful.(*wrappedListener).Accept(0xc007b221c0)
	/source/modules/graceful/server.go:180 +0x47
net/http.(*Server).Serve(0xc00250cd20, {0x4535118, 0xc007b221c0})
	/usr/local/go/src/net/http/server.go:3002 +0x394
code.gitea.io/gitea/modules/graceful.(*Server).Serve(0xc0032b9ce0, 0xc0079f1900)
	/source/modules/graceful/server.go:133 +0x12b
code.gitea.io/gitea/modules/graceful.(*Server).ListenAndServe(0xc0032b9ce0, 0x15208a0)
	/source/modules/graceful/server.go:95 +0x1c5
code.gitea.io/gitea/modules/graceful.HTTPListenAndServe({0x3212349, 0x19}, {0xc007afa5e0, 0x14}, {0x320f766, 0x1}, {0x451d620, 0xc0048b8880})
	/source/modules/graceful/server_http.go:33 +0x3d
code.gitea.io/gitea/cmd.runHTTP(...)
	/source/cmd/web_graceful.go:19
code.gitea.io/gitea/cmd.listen({0x451d620, 0xc0048b8880}, 0x1)
	/source/cmd/web.go:224 +0x425
code.gitea.io/gitea/cmd.runWeb(0xc00251e840)
	/source/cmd/web.go:171 +0x833
github.com/urfave/cli.HandleAction({0x2e0e360, 0x40034f0}, 0x3)
	/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:524 +0xa8
github.com/urfave/cli.Command.Run({{0x32127ae, 0x3}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x337f040, 0x16}, {0x0, ...}, ...}, ...)
	/go/pkg/mod/github.com/urfave/cli@v1.22.5/command.go:173 +0x652
github.com/urfave/cli.(*App).Run(0xc00254a000, {0xc0000d4000, 0x4, 0x4})
	/go/pkg/mod/github.com/urfave/cli@v1.22.5/app.go:277 +0x705
main.main()
	/source/main.go:115 +0xcf3

goroutine 17 [select, 9 minutes, locked to thread]:
golang.org/x/sys/windows/svc.serviceMain(0x2, 0xd762819538)
	/go/pkg/mod/golang.org/x/sys@v0.0.0-20211117180635-dee7805ff2e1/windows/svc/service.go:245 +0x428

goroutine 20 [select]:
code.gitea.io/gitea/modules/log.(*MultiChannelledLog).Start(0xc000120100)
	/source/modules/log/event.go:300 +0x27c
created by code.gitea.io/gitea/modules/log.(*MultiChannelledLog).AddLogger
	/source/modules/log/event.go:190 +0x17b

goroutine 5 [select, 9 minutes]:
github.com/blevesearch/bleve_index_api.AnalysisWorker({0xc00240f8c0, 0xc00240f920})
	/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:46 +0x95
created by github.com/blevesearch/bleve_index_api.NewAnalysisQueue
	/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:38 +0x97

goroutine 6 [select, 9 minutes]:
github.com/blevesearch/bleve_index_api.AnalysisWorker({0xc00240f8c0, 0xc00240f920})
	/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:46 +0x95
created by github.com/blevesearch/bleve_index_api.NewAnalysisQueue
	/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:38 +0x97

goroutine 7 [select, 9 minutes]:
github.com/blevesearch/bleve_index_api.AnalysisWorker({0xc00240f8c0, 0xc00240f920})
	/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:46 +0x95
created by github.com/blevesearch/bleve_index_api.NewAnalysisQueue
	/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:38 +0x97

goroutine 8 [select, 9 minutes]:
github.com/blevesearch/bleve_index_api.AnalysisWorker({0xc00240f8c0, 0xc00240f920})
	/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:46 +0x95
created by github.com/blevesearch/bleve_index_api.NewAnalysisQueue
	/go/pkg/mod/github.com/blevesearch/bleve_index_api@v1.0.1/analysis.go:38 +0x97

goroutine 10 [select, 9 minutes]:
code.gitea.io/gitea/modules/graceful.(*Manager).Execute(0xc002502100, {0x0, 0xd43e85, 0xc002499fd0}, 0xc002464060, 0xc000f828a0)
	/source/modules/graceful/manager_windows.go:133 +0x22f
golang.org/x/sys/windows/svc.serviceMain.func2()
	/go/pkg/mod/golang.org/x/sys@v0.0.0-20211117180635-dee7805ff2e1/windows/svc/service.go:233 +0x45
created by golang.org/x/sys/windows/svc.serviceMain
	/go/pkg/mod/golang.org/x/sys@v0.0.0-20211117180635-dee7805ff2e1/windows/svc/service.go:232 +0x2b9

goroutine 26 [syscall, 9 minutes, locked to thread]:
syscall.Syscall(0x7fff0f19f5f0, 0x1, 0xc002495f80, 0x0, 0x0)
	/usr/local/go/src/runtime/syscall_windows.go:483 +0xf4
golang.org/x/sys/windows.StartServiceCtrlDispatcher(0xc002495f80)
	/go/pkg/mod/golang.org/x/sys@v0.0.0-20211117180635-dee7805ff2e1/windows/zsyscall_windows.go:1241 +0x65
golang.org/x/sys/windows/svc.Run({0x3228551, 0x5}, {0x451d300, 0xc002502100})
	/go/pkg/mod/golang.org/x/sys@v0.0.0-20211117180635-dee7805ff2e1/windows/svc/service.go:293 +0x129
code.gitea.io/gitea/modules/graceful.(*Manager).start.func1()
	/source/modules/graceful/manager_windows.go:100 +0x36
created by code.gitea.io/gitea/modules/graceful.(*Manager).start
	/source/modules/graceful/manager_windows.go:99 +0x34f

goroutine 4349 [IO wait]:
internal/poll.runtime_pollWait(0xd769646f58, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc0023c1568, 0xc0023c1578, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.execIO(0xc00039c798, 0x4009a30)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe5
internal/poll.(*FD).Read(0xc00039c780, {0xc004e1d000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_windows.go:441 +0x25f
net.(*netFD).Read(0xc00039c780, {0xc004e1d000, 0x4521b80, 0xc00250c540})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00a1c8090, {0xc004e1d000, 0xd44ad1, 0xc0093cf3e8})
	/usr/local/go/src/net/net.go:183 +0x45
net/http.(*connReader).Read(0xc0093cf3e0, {0xc004e1d000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:780 +0x16d
bufio.(*Reader).fill(0xc007ada2a0)
	/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc007ada2a0, 0x1)
	/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).ReadLine(0xc007ada2a0)
	/usr/local/go/src/bufio/bufio.go:389 +0x27
net/textproto.(*Reader).readLineSlice(0xc008349dd0)
	/usr/local/go/src/net/textproto/reader.go:57 +0x99
net/textproto.(*Reader).ReadLine(...)
	/usr/local/go/src/net/textproto/reader.go:38
net/http.readRequest(0xc00a1c8090)
	/usr/local/go/src/net/http/request.go:1029 +0x79
net/http.(*conn).readRequest(0xc007fa6280, {0x4550c00, 0xc005b5ecc0})
	/usr/local/go/src/net/http/server.go:966 +0x225
net/http.(*conn).serve(0xc007fa6280, {0x4550ca8, 0xc002458de0})
	/usr/local/go/src/net/http/server.go:1856 +0x865
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3034 +0x4e8

goroutine 28 [IO wait]:
internal/poll.runtime_pollWait(0xd769647698, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc0055752e0, 0xd0ea94, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.execIO(0xc001081418, 0xc00249dbb0)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe5
internal/poll.(*FD).acceptOne(0xc001081400, 0x3c8, {0xc0038451d0, 0xdc147f, 0x0}, 0x0)
	/usr/local/go/src/internal/poll/fd_windows.go:810 +0x6d
internal/poll.(*FD).Accept(0xc001081400, 0xc00249dd88)
	/usr/local/go/src/internal/poll/fd_windows.go:844 +0x1d6
net.(*netFD).accept(0xc001081400)
	/usr/local/go/src/net/fd_windows.go:139 +0x65
net.(*TCPListener).accept(0xc002446e40)
	/usr/local/go/src/net/tcpsock_posix.go:140 +0x28
net.(*TCPListener).Accept(0xc002446e40)
	/usr/local/go/src/net/tcpsock.go:262 +0x3d
net/http.(*Server).Serve(0xc0000e0d20, {0x4537098, 0xc002446e40})
	/usr/local/go/src/net/http/server.go:3002 +0x394
net/http.(*Server).ListenAndServe(0xc0000e0d20)
	/usr/local/go/src/net/http/server.go:2931 +0x7d
net/http.ListenAndServe(...)
	/usr/local/go/src/net/http/server.go:3185
code.gitea.io/gitea/cmd.runWeb.func2()
	/source/cmd/web.go:146 +0x65
created by code.gitea.io/gitea/cmd.runWeb
	/source/cmd/web.go:144 +0x6f0

goroutine 14 [select]:
code.gitea.io/gitea/modules/log.(*ChannelledLog).Start(0xc0025a6140)
	/source/modules/log/event.go:70 +0x91
created by code.gitea.io/gitea/modules/log.NewChannelledLog
	/source/modules/log/event.go:61 +0x245

goroutine 912 [select, 9 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc00049c680)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x15e
created by github.com/syndtr/goleveldb/leveldb.openDB
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x4ef

goroutine 16 [select, 9 minutes]:
database/sql.(*DB).connectionOpener(0xc0032fe4e0, {0x4550c00, 0xc00255fdc0})
	/usr/local/go/src/database/sql/sql.go:1196 +0x93
created by database/sql.OpenDB
	/usr/local/go/src/database/sql/sql.go:794 +0x188

goroutine 904 [chan receive, 9 minutes]:
os/exec.(*Cmd).Wait(0xc00010f340)
	/usr/local/go/src/os/exec/exec.go:515 +0x188
code.gitea.io/gitea/modules/git.(*Command).RunWithContext(0xc004b48f00, 0xc004a7ded8)
	/source/modules/git/command.go:189 +0x6b6
code.gitea.io/gitea/modules/git.(*Command).RunInDirTimeoutEnvFullPipelineFunc(...)
	/source/modules/git/command.go:113
code.gitea.io/gitea/modules/git.(*CheckAttributeReader).Run(0xc0000f3ef0)
	/source/modules/git/repo_attribute.go:185 +0x25b
code.gitea.io/gitea/services/gitdiff.GetDiff.func1()
	/source/services/gitdiff/gitdiff.go:1415 +0x45
created by code.gitea.io/gitea/services/gitdiff.GetDiff
	/source/services/gitdiff/gitdiff.go:1414 +0x1267

goroutine 914 [select, 9 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc00049c680)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x685
created by github.com/syndtr/goleveldb/leveldb.openDB
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x5a7

goroutine 913 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc00049c680)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0xae
created by github.com/syndtr/goleveldb/leveldb.openDB
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x531

goroutine 911 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc0000e00e0)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xcd
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x19b

goroutine 980 [chan receive, 9 minutes]:
code.gitea.io/gitea/modules/queue.(*WrappedQueue).Run.func1()
	/source/modules/queue/queue_wrapped.go:233 +0x71
created by code.gitea.io/gitea/modules/queue.(*WrappedQueue).Run
	/source/modules/queue/queue_wrapped.go:232 +0x2af

goroutine 979 [select, 9 minutes]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc00049c680)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0x119
created by github.com/syndtr/goleveldb/leveldb.openDB
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x5e9

goroutine 906 [syscall, 9 minutes, locked to thread]:
syscall.Syscall6(0x7fff0e661de0, 0x5, 0x318, 0xc005637600, 0x200, 0xc004827d5c, 0x0, 0x0)
	/usr/local/go/src/runtime/syscall_windows.go:497 +0xfa
syscall.ReadFile(0x0, {0xc005637600, 0x200, 0x800000}, 0xc004827d5c, 0x0)
	/usr/local/go/src/syscall/zsyscall_windows.go:1024 +0xbe
syscall.Read(0xc007caaa00, {0xc005637600, 0xc000053000, 0xc004827db0})
	/usr/local/go/src/syscall/syscall_windows.go:380 +0x2e
internal/poll.(*FD).Read(0xc007caaa00, {0xc005637600, 0x200, 0x200})
	/usr/local/go/src/internal/poll/fd_windows.go:427 +0x1b4
os.(*File).read(...)
	/usr/local/go/src/os/file_posix.go:32
os.(*File).Read(0xc0078c2c40, {0xc005637600, 0xd3d196, 0xc004827ea0})
	/usr/local/go/src/os/file.go:119 +0x5e
bytes.(*Buffer).ReadFrom(0xc007bf1260, {0x4521d20, 0xc0078c2c40})
	/usr/local/go/src/bytes/buffer.go:204 +0x98
io.copyBuffer({0x451d040, 0xc007bf1260}, {0x4521d20, 0xc0078c2c40}, {0x0, 0x0, 0x0})
	/usr/local/go/src/io/io.go:409 +0x14b
io.Copy(...)
	/usr/local/go/src/io/io.go:382
os/exec.(*Cmd).writerDescriptor.func1()
	/usr/local/go/src/os/exec/exec.go:311 +0x3a
os/exec.(*Cmd).Start.func1(0xc00750ba00)
	/usr/local/go/src/os/exec/exec.go:441 +0x25
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:440 +0x8e5

goroutine 873 [select, 9 minutes]:
code.gitea.io/gitea/modules/graceful.(*Server).awaitShutdown(0xc007991400)
	/source/modules/graceful/server_hooks.go:16 +0xb1
created by code.gitea.io/gitea/modules/graceful.(*Server).ListenAndServe
	/source/modules/graceful/server.go:83 +0x65

goroutine 4699 [runnable]:
net/http.(*connReader).startBackgroundRead·dwrap·75()
	/usr/local/go/src/net/http/server.go:668
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1
created by net/http.(*connReader).startBackgroundRead
	/usr/local/go/src/net/http/server.go:668 +0xcf

goroutine 905 [syscall, 9 minutes, locked to thread]:
syscall.Syscall6(0x7fff0e661de0, 0x5, 0x308, 0xc007e00000, 0x8000, 0xc00496fdb4, 0x0, 0x0)
	/usr/local/go/src/runtime/syscall_windows.go:497 +0xfa
syscall.ReadFile(0x10, {0xc007e00000, 0x8000, 0x800000}, 0xc00496fdb4, 0x0)
	/usr/local/go/src/syscall/zsyscall_windows.go:1024 +0xbe
syscall.Read(0xc007caa500, {0xc007e00000, 0xb, 0xc00496fea0})
	/usr/local/go/src/syscall/syscall_windows.go:380 +0x2e
internal/poll.(*FD).Read(0xc007caa500, {0xc007e00000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_windows.go:427 +0x1b4
os.(*File).read(...)
	/usr/local/go/src/os/file_posix.go:32
os.(*File).Read(0xc0078c2c28, {0xc007e00000, 0xdf, 0x8000})
	/usr/local/go/src/os/file.go:119 +0x5e
io.copyBuffer({0xd76ae8cdc0, 0xc007c80600}, {0x4521d20, 0xc0078c2c28}, {0x0, 0x0, 0x0})
	/usr/local/go/src/io/io.go:423 +0x1b2
io.Copy(...)
	/usr/local/go/src/io/io.go:382
os/exec.(*Cmd).writerDescriptor.func1()
	/usr/local/go/src/os/exec/exec.go:311 +0x3a
os/exec.(*Cmd).Start.func1(0xc004a06e00)
	/usr/local/go/src/os/exec/exec.go:441 +0x25
created by os/exec.(*Cmd).Start
	/usr/local/go/src/os/exec/exec.go:440 +0x8e5

goroutine 145 [select, 9 minutes]:
github.com/gogs/cron.(*Cron).run(0xc0022bba40)
	/go/pkg/mod/github.com/gogs/cron@v0.0.0-20171120032916-9f6c956d3e14/cron.go:197 +0x5a5
created by github.com/gogs/cron.(*Cron).Start
	/go/pkg/mod/github.com/gogs/cron@v0.0.0-20171120032916-9f6c956d3e14/cron.go:150 +0x65

goroutine 267 [select]:
code.gitea.io/gitea/modules/eventsource.(*Manager).Run(0xc002446ac8, {0x4550c00, 0xc00255e1c0})
	/source/modules/eventsource/manager_run.go:32 +0xdf
code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext(0xc002502100, 0xc0046f74c0)
	/source/modules/graceful/manager.go:139 +0xb0
created by code.gitea.io/gitea/modules/eventsource.(*Manager).Init
	/source/modules/eventsource/manager_run.go:23 +0xe5

goroutine 263 [select, 9 minutes]:
code.gitea.io/gitea/services/webhook.DeliverHooks({0x4550c00, 0xc00255e1c0})
	/source/services/webhook/deliver.go:227 +0x28c
code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownContext(0xc002502100, 0x4005dd8)
	/source/modules/graceful/manager.go:139 +0xb0
created by code.gitea.io/gitea/services/webhook.InitDeliverHooks
	/source/services/webhook/deliver.go:311 +0x1d1

goroutine 4696 [select]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1()
	/go/pkg/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:614 +0xb0
created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher
	/go/pkg/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:611 +0x105

goroutine 360 [select, 9 minutes]:
code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).readToChan(0xc007b42460)
	/source/modules/queue/queue_bytefifo.go:151 +0x31a
code.gitea.io/gitea/modules/queue.(*ByteFIFOQueue).Run(0xc007b42460, 0xc0013fb6c0, 0xc0013fb6d0)
	/source/modules/queue/queue_bytefifo.go:120 +0x257
code.gitea.io/gitea/modules/queue.(*WrappedQueue).Run(0xc0046c5e00, 0x2f55240, 0x1123301)
	/source/modules/queue/queue_wrapped.go:242 +0x2ce
code.gitea.io/gitea/modules/graceful.(*Manager).RunWithShutdownFns(0xc002502100, 0xc00593ad38)
	/source/modules/graceful/manager.go:83 +0x125
created by code.gitea.io/gitea/modules/indexer/issues.InitIssueIndexer.func2
	/source/modules/indexer/issues/indexer.go:196 +0x56f

goroutine 1484 [select]:
code.gitea.io/gitea/routers/web/events.Events(0xc009d2f380)
	/source/routers/web/events/events.go:78 +0x6c9
code.gitea.io/gitea/modules/web.Wrap.func1({0x4535088, 0xc005b41940}, 0xc006457800)
	/source/modules/web/route.go:74 +0x1c5
net/http.HandlerFunc.ServeHTTP(0x2ea33a0, {0x4535088, 0xc005b41940}, 0xc00a2259a0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc007a9cba0, {0x4535088, 0xc005b41940}, 0xc006457800)
	/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:442 +0x216
net/http.HandlerFunc.ServeHTTP(0xc00a257620, {0x4535088, 0xc005b41940}, 0x10000c0001475a0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
code.gitea.io/gitea/modules/web.Middle.func1.1({0x3036500, 0x0}, 0xc005b41940)
	/source/modules/web/route.go:113 +0xe6
net/http.HandlerFunc.ServeHTTP(0xc009d2f380, {0x4535088, 0xc005b41940}, 0x10)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
code.gitea.io/gitea/modules/web.Middle.func1.1({0x3036500, 0x0}, 0xc005b41940)
	/source/modules/web/route.go:113 +0xe6
net/http.HandlerFunc.ServeHTTP(0xc009d2f380, {0x4535088, 0xc005b41940}, 0x550000c00a39e5f0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
code.gitea.io/gitea/modules/web.Middle.func1.1({0xc00a243e60, 0xc00a03eda8}, 0x1e00f7c)
	/source/modules/web/route.go:113 +0xe6
net/http.HandlerFunc.ServeHTTP(0x3245d23, {0x4535088, 0xc005b41940}, 0xc009d2f380)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/go-chi/chi/v5/middleware.GetHead.func1({0x4535088, 0xc005b41940}, 0xc005b41940)
	/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/middleware/get_head.go:37 +0x1e5
net/http.HandlerFunc.ServeHTTP(0xc009d2f380, {0x4535088, 0xc005b41940}, 0x2ea4480)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
code.gitea.io/gitea/modules/web.Middle.func1.1({0x3036500, 0x2dc9b00}, 0xc005b41940)
	/source/modules/web/route.go:113 +0xe6
net/http.HandlerFunc.ServeHTTP(0x2ea4480, {0x4535088, 0xc005b41940}, 0x8)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
code.gitea.io/gitea/modules/context.Contexter.func1.1({0x4535088, 0xc005b41940}, 0xc006457400)
	/source/modules/context/context.go:731 +0x1991
net/http.HandlerFunc.ServeHTTP(0x2dc9ce0, {0x4535088, 0xc005b41940}, 0x4)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x4535088, 0xc005b41940}, 0xc00a2259a0)
	/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:342 +0x291
net/http.HandlerFunc.ServeHTTP(0xc00a03f298, {0x4535088, 0xc005b41940}, 0xd52ed4)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc007a9cba0, {0x4535088, 0xc005b41940}, 0xc006457400)
	/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:71 +0x48d
github.com/go-chi/chi/v5.(*Mux).Mount.func1({0x4535088, 0xc005b41940}, 0xc006457400)
	/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:314 +0x19c
net/http.HandlerFunc.ServeHTTP(0x2ea33a0, {0x4535088, 0xc005b41940}, 0xc00a225890)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc0048be7e0, {0x4535088, 0xc005b41940}, 0xc006457400)
	/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:442 +0x216
net/http.HandlerFunc.ServeHTTP(0xc00a257500, {0x4535088, 0xc005b41940}, 0x88)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
code.gitea.io/gitea/routers/web.Recovery.func1.1({0x4535088, 0xc005b41940}, 0x44e5901)
	/source/routers/web/base.go:169 +0x9b
net/http.HandlerFunc.ServeHTTP(0x4550b20, {0x4535088, 0xc005b41940}, 0x44e5940)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
gitea.com/go-chi/session.Sessioner.func1.1({0x4535088, 0xc005b41940}, 0xc006457100)
	/go/pkg/mod/gitea.com/go-chi/session@v0.0.0-20211218221615-e3605d8b28b8/session.go:257 +0x2fe
net/http.HandlerFunc.ServeHTTP(0x0, {0x4535088, 0xc005b41940}, 0xc)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
code.gitea.io/gitea/modules/public.AssetsHandler.func1.1({0x4535088, 0xc005b41940}, 0xc006457100)
	/source/modules/public/public.go:42 +0x3bc
net/http.HandlerFunc.ServeHTTP(0x2e78080, {0x4535088, 0xc005b41940}, 0xd52ed4)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc0048be7e0, {0x4535088, 0xc005b41940}, 0xc006457100)
	/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:71 +0x48d
github.com/go-chi/chi/v5.(*Mux).Mount.func1({0x4535088, 0xc005b41940}, 0xc006457100)
	/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:314 +0x19c
net/http.HandlerFunc.ServeHTTP(0x2ea33a0, {0x4535088, 0xc005b41940}, 0xc00a226604)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc0048be780, {0x4535088, 0xc005b41940}, 0xc006457100)
	/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:442 +0x216
net/http.HandlerFunc.ServeHTTP(0xd13f62, {0x4535088, 0xc005b41940}, 0xc00a03f7d8)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
code.gitea.io/gitea/routers/common.Middlewares.func2.1({0x4535088, 0xc005b41940}, 0xc00a03f870)
	/source/routers/common/middleware.go:78 +0x82
net/http.HandlerFunc.ServeHTTP(0xc00a03f830, {0x4535088, 0xc005b41940}, 0x44e0e30)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/go-chi/chi/v5/middleware.StripSlashes.func1({0x4535088, 0xc005b41940}, 0xc006457100)
	/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/middleware/strip.go:30 +0x139
net/http.HandlerFunc.ServeHTTP(0xc00a2329ce, {0x4535088, 0xc005b41940}, 0xc00a2329ce)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/chi-middleware/proxy.ForwardedHeaders.func1.1({0x4535088, 0xc005b41940}, 0xc006457100)
	/go/pkg/mod/github.com/chi-middleware/proxy@v1.1.1/middleware.go:37 +0x19f
net/http.HandlerFunc.ServeHTTP(0x322d955, {0x4535088, 0xc005b41940}, 0xc00a22c768)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
code.gitea.io/gitea/routers/common.Middlewares.func1.1({0x4537278, 0xc00946e460}, 0xc006456f00)
	/source/routers/common/middleware.go:31 +0x45a
net/http.HandlerFunc.ServeHTTP(0x4550c00, {0x4537278, 0xc00946e460}, 0x2bb17d0)
	/usr/local/go/src/net/http/server.go:2047 +0x2f
github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc0048be780, {0x4537278, 0xc00946e460}, 0xc006525500)
	/go/pkg/mod/github.com/go-chi/chi/v5@v5.0.4/mux.go:88 +0x442
code.gitea.io/gitea/modules/web.(*Route).ServeHTTP(0x0, {0x4537278, 0xc00946e460}, 0xd6a055)
	/source/modules/web/route.go:328 +0x2e
net/http.serverHandler.ServeHTTP({0xc009f14390}, {0x4537278, 0xc00946e460}, 0xc006525500)
	/usr/local/go/src/net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc009bd7680, {0x4550ca8, 0xc007b24930})
	/usr/local/go/src/net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3034 +0x4e8

goroutine 1633 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0xd7696473e0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.execIO(0xc009f2a518, 0x4009a30)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe5
internal/poll.(*FD).Read(0xc009f2a500, {0xc009f143a1, 0x1, 0x1})
	/usr/local/go/src/internal/poll/fd_windows.go:441 +0x25f
net.(*netFD).Read(0xc009f2a500, {0xc009f143a1, 0x0, 0x0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc006bd3d58, {0xc009f143a1, 0x0, 0x0})
	/usr/local/go/src/net/net.go:183 +0x45
net/http.(*connReader).backgroundRead(0xc009f14390)
	/usr/local/go/src/net/http/server.go:672 +0x3f
created by net/http.(*connReader).startBackgroundRead
	/usr/local/go/src/net/http/server.go:668 +0xcf

goroutine 4698 [select]:
database/sql.(*DB).connectionCleaner(0xc0032fe4e0, 0x0)
	/usr/local/go/src/database/sql/sql.go:1068 +0xbd
created by database/sql.(*DB).startCleanerLocked
	/usr/local/go/src/database/sql/sql.go:1055 +0x105

goroutine 4374 [IO wait]:
internal/poll.runtime_pollWait(0xd769647040, 0x72)
	/usr/local/go/src/runtime/netpoll.go:234 +0x89
internal/poll.(*pollDesc).wait(0xc001c2d538, 0x108a7ef, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.execIO(0xc0042de298, 0x4009a30)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0xe5
internal/poll.(*FD).Read(0xc0042de280, {0xc0066bc000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_windows.go:441 +0x25f
net.(*netFD).Read(0xc0042de280, {0xc0066bc000, 0xc001c2d698, 0xd44ad1})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc009f7a098, {0xc0066bc000, 0xc001c2d6c8, 0xd66525})
	/usr/local/go/src/net/net.go:183 +0x45
net/http.(*connReader).Read(0xc008349da0, {0xc0066bc000, 0x1000, 0x1000})
	/usr/local/go/src/net/http/server.go:780 +0x16d
bufio.(*Reader).fill(0xc002c07aa0)
	/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).ReadSlice(0xc002c07aa0, 0x10)
	/usr/local/go/src/bufio/bufio.go:360 +0x2f
bufio.(*Reader).ReadLine(0xc002c07aa0)
	/usr/local/go/src/bufio/bufio.go:389 +0x27
net/textproto.(*Reader).readLineSlice(0xc005fab7a0)
	/usr/local/go/src/net/textproto/reader.go:57 +0x99
net/textproto.(*Reader).ReadLine(...)
	/usr/local/go/src/net/textproto/reader.go:38
net/http.readRequest(0xc00a255bd0)
	/usr/local/go/src/net/http/request.go:1029 +0x79
net/http.(*conn).readRequest(0xc007b426e0, {0x4550c00, 0xc0032c4040})
	/usr/local/go/src/net/http/server.go:966 +0x225
net/http.(*conn).serve(0xc007b426e0, {0x4550ca8, 0xc007b24930})
	/usr/local/go/src/net/http/server.go:1856 +0x865
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:3034 +0x4e8

[atom0s]

I've narrowed down one of the causes that will leave the git.exe hanging. It is happening when checking a diff.

For example, the following steps can be reproduced to cause hung processes:

1. Make a repository.
2. Add some files to it like normal.
3. Make a fork of the repository.
4. Make an edit in the fork.
5. Create a pull request to the main repo from the fork.
6. View the diff page of the pull request. (Files changed page.)

This will trigger a hung process. (Refreshing/reloading the page will make a new hung git.exe each time its loaded as well.)

This appears to be only one of the causes of the hung processes though. There still seems to be others happening not just from diffs.

[uncled1023]

Here are the results of my stack dump after it started being locked up: https://p.teknik.io/LueHh