Use FindFirstFileEx(FIND_FIRST_EX_LARGE_FETCH) instead of FindFirstFile()?

[mehrdadn]

Edit: Apologies, never mind; it seems this doesn't improve the fetch performance of the first entry like I had thought.

[dscho]

It does not improve the fetch performance of the first entry. That was never the intention. It improves performance of subsequent FindNextFile() calls, possibly by a lot e.g. when reading over network (i.e. a UNC path or a mapped network share).

However, it is only available in Windows 7 and later, and Git for Windows still supports Vista (even if we officially dropped support for XP due to lack of active contributors).

And the most benefit would probably come from adding this to the FSCache feature. So what would make most sense is to have a static version check in the beginning of fsentry_create_list() and then continue either with FindFirstFileW() or with FindFirstFileExW() depending on that version test.

Note: this would make for an excellent first-time contribution.

[dscho]

More information on when to use FIND_FIRST_EX_LARGE_FETCH: https://blogs.msdn.microsoft.com/oldnewthing/20131024-00/?p=2843/

[mehrdadn]

Well the reason I posted this originally was that I thought it would reduce the number of syscalls by at least 1 for since currently FindFirstFile only fetches 1 entry (and that entry is ., which is useless) no matter what. It's not as important to me for subsequent fetches since I'm dealing with large trees but not large directories per se. But if it seems useful and I get a chance to implement this I'll let you know! Currently I'm leaning toward calling NtQueryDirectoryFile directly and skipping the Win32 API altogether so I can reduce the number of syscalls, but we'll see.

[dscho]

if it seems useful and I get a chance to implement this I'll let you know!

Great! I still think this has merit, in particular for compat/win32/fscache.c.

[mehrdadn]

Okay so I just did some tests and it seems this wouldn't make a huge difference like I had expected. Depending on the repo the difference for git status was anywhere from completely unobservable (small/normal repo) to 3% (large-ish repo with a few directories that have a few dozen files) to 8% (kotlin) to 18% (artificial repo with 30k files in a single directory). So it doesn't seem to really pay off enough to be worth it. Also note that my test used NtQueryDirectoryFile to get back a large batch on the first call too, so the improvement would be even less for FIND_FIRST_EX_LARGE_FETCH.

[dscho]

I'll take the 8%... :-) thank you for sharing your research!

[mehrdadn]

You know, I reimplemented this last night and it seems it could be more like 10% if you just use NtQueryDirectoryFile's output directly and skip the step where I converted the results into WIN32_FIND_DATA format. Happy to share the code if you'd like to take copyright/ownership of it. :-) Although I didn't quite do the part regarding the reparse-point tags (I think you need to call FSCTL_GET_REPARSE_POINT for files that have reparse points so you can tell if they're symlinks) so that would need to be added too.

[mehrdadn]

Oh apparently Windows uses a hack where it putsthe ReparseTag into the EaSize field, so no need for FSCTL_GET_REPARSE_POINT... go figure.