JLine 3: tab completion behavior on tab after `.` is broken

[SethTisue]

if you Predef. and then tab, the completions you're offered include .:

scala> Predef.
.

And if you tab again, the cursor moves into the completions matrix but you lose the dot you typed

from one point of view you could call this minor, to me it seems both common enough and annoying enough to be considered a blocker, especially since it might have an easy fix (some off-by-one thing?)

the Dotty REPL does the right thing.

[SethTisue]

hmm, the shell.CompletionResult doesn't include the . entry. (so it also isn't in the result of :completions)

[lrytz]

Similarly, completing : shows : as an option on the first line. Also, maybe unrelated, completing :p doesn't show any results (it should show :power and :paste).

[SethTisue]

I barked up a lot of wrong trees on this, but I learned my way around the forest, at least.

I now believe the problem is that our ScalaParser and ScalaParsedLine don't meet JLine's expectations. When tabbing after Predef., ScalaParsedLine#word returns . ; I think it should be the empty string.

[SethTisue]

The logic in ScalaParser#tokenize isn't sophisticated enough. Predef. is treated as two tokens, but there are really three tokens here: Predef, then ., then an empty token which is the token we want completed.

[SethTisue]

Dotty (in JLineTerminal.scala) has:

          def isCompletable(token: Token) = isIdentifier(token) || isKeyword(token)

and if the current token isn't "completable", then it just returns a dummy result with wordCursor of 0 and wordIndex of -1, and then I guess JLine copes with that okay

[SethTisue]

(Note that I split completion for : commands out as a separate ticket, #708. There are different code paths involved so I don't think it's the "same" bug, as it understandably appeared.)

[SethTisue]

closing on the assumption that the fix I added to scala/scala#8848 will be merged