Under no circumstances should HIE ever stop responding

[KillyMXI]

The title is a quote from #461.

I was looking through opened issues and it really struck me how common this pattern:
On any unexpected input, HIE just shuts in itself.
And it is a worst way to handle the situation when we have a really unstable piece of software with a lot of things "unexpected".
As a user, I:

• can't continue my work at least with the parts that known to work;
• have to go through a lot of trouble to gather debug information.

The issues with the same pattern:

• HIE stops working in VS Code when apply-refact fails #451
• Messaging hangs if the client sends a junk filepath #461
• HIE type on hover dies because of specific file #556
• HIE unresponsive #983
• HIE not working, getting "Loading..." on hover and autocomplete does not work vscode-haskell#124 (I came from there initially)
• maybe more

In no way we can work productively with HIE while you will unhurriedly play whack-a-mole with the issues.
I don't want to learn to avoid certain lines just because you can't handle them.
And restarting HIE to recover it's functionality takes too long.

Messaging itself MUST be more fault-tolerant.

• Failed to handle a request? - report back as useful failure message as you can in given situation and get ready to process following requests. This is not about debug logging - this is to explain to user why his request wasn't handled;
• IDE extensions should be able to tell user what went wrong and what can be done about it;
• User should be able to use other parts of HIE functionality while you're working on fixes for reported issues.

[mpickering]

Seems like this should be one of the higher priority issues to fix.

[Anrock]

Agreed. I've pinned the issue to help with visibility.

[alanz]

I think the first step is to have some reproducable bug reports.

Because the underlying architecture is supposed to make it robust, in that the main loop keeps processing regardless, we have deferred execution to provide a result if the GHC invocation fails, and for things like Hover it should respond immediately based on their being a currently cached result.

[mpickering]

I think actions like "hover" block because the implementation was inefficient.

I also observed that reporting a large number of diagnostics (500) causes something to pause, not sure if that was a HIE or vscode issue.

[alanz]

I am pretty sure that the default setting for number of issues is to limit it to 100. Which are sorted to be the first ones, so perhaps the sorting process is where the delay comes from.

[alanz]

And I think the current delayer is requesting completions, if anyone wants to take a look. It should work like Hover, if there is no currently cached module it returns immediately with an empty result.

[bscottm]

I think this is the appropriate place to post my particular issue, which resembles a closed file descriptor, but seems more closely related to this issue -- HIE stalls after a 3-4 requests. At first I thought the problem was with SublimeText, the LSP package and the Windows implementation of pipes. I tried hie-8.6.4 under Linux on VirtualBox, same problem. It seems to take about three or four JSON RPCs before hie-8.6.4 stalls.

LSP's basic loop calls Python's str.readline() to grab the next line of input looking for the "Content-length" header. That just stalls waiting for input. At first I thought that it might be a closed file descriptor, but I would expect the str.readline() loop to return EOF (an empty string.)

This particular problem showed up about two to three weeks ago, and I've been trying to diagnose it when I can free cycles. I'm on the latest github version with submodules synced. hie-wrapper log is attached, although I anonymized some of my paths. Apologies for the big source file that gets downloaded to the server...

hie-wrapper.log

[mpickering]

@bscottm Can you please open a new issue for your report?

[NickSeagull]

Is there something people eager to contribute can do to help with this? 😃

[fendor]

Sorry for the late response, @NickSeagull!
There is a bunch of stuff you can do:

Improve underlying bios system

Currently the most unstable part of hie seems to be the build system itself. If you are comfortable with stuff like ghc-mod and cabal, then you can try to help out on issue #1126. Ask @mpickering where help may be required.

Plugins that crash

Sometimes plugins crash in hie, taking the whole application with them. Although I am currently not aware of plugins that regurarly crash, each plugin must be validated on its own. Currently, I have checked hoogle, hsimport, package, pragma, floskell and brittany. Almost all of them can still crash on IOExceptions, such as File not found or Permission denied. In practice though, these issues never happened to me and there are no open issues for that. To help with this, you can go through plugins, adding comments and tests.

Build system

Build system of hie has a bunch of problems at the moment. There are missing tests to validate that still everything works as expected. Also, if you are a windows expert, you can try to tackle windows specific issues, such as #1219, #1133, #983, ... the list goes on.

Hlint data-files

Although hie no longer crashes if hlint crashes, it is not optimal that hlint can not be used hlint data-files are missing. The issue #1143 discusses the problem in detail. If you want to tackle this, talk to me or @power-fungus, we have already decided on a plan of attack.

Windows specific problems

It seems we do not have a lot of windows developers at the moment. If you want to tackle that problem, the most obvious problem is that the tests currently do not run on windows. It may be an issue with lsp-test, but currently no one knows for sure.

[jneira]

Nice resume @fendor, i'll try to help with any of windows related ones

[ProofOfKeags]

the underlying crashes may be more difficult to fix, but making sure that it doesn't take HIE down with it seems like a fairly straight forward fix, no? It may be ugly but couldn't you indiscriminately catch all plugin exceptions and then report some null value back, or does LSP have no room in the spec to represent the failure of the request?