[WIP] Hi Haddock #906

sjakobi · 2018-08-14T11:31:22Z

(GSoC project description)

What already works

For many packages we can already generate reasonably-looking haddocks. For example see these haddocks for lens.

TODOs

~~Get html-test to run. Currently we can't handle the Bug294 test-case. This requires changes in extractDecl. (Details)~~ Fixed by @harpocrates in 29edbb9.
Scrutinize output changes and update the tests. In particular we currently generate kind signatures for all higher-kinded type variables. (Details)
Fix --hyperlinked-source by integrating Zubin Duggal's GSoC project.
Warnings for ambiguous identfiers are too noisy. (Details)

A few more smaller issues are marked TODO in the source code.

The corresponding changes in GHC are on Phabricator.

Corresponding change in ghc: sjakobi/ghc@19adbf6.

(cherry picked from commit 396961f)

We're stilling missing some info. See FIXMEs.

sjakobi · 2018-10-22T17:13:20Z

@harpocrates I've updated the Phab diff according to your instructions. Thanks a lot for the rebase!

I actually wouldn't mind if you'd comandeer the diff BTW.

sjakobi · 2018-10-26T15:21:07Z

@harpocrates Could you give cad2ed9 a quick review?

The main thing I'm worried about is that in hsDocRenamer, I might recreate the env for each identifier I'm looking up. I'm just not sure what kind of code GHC generates there… :/

haddock/haddock-api/src/Haddock/Interface/LexParseRn.hs

Lines 193 to 196 in cad2ed9

    
           hsDocRenamer :: HsDoc Name -> Renamer 
        
           hsDocRenamer hsDoc s = 
        
               let env = Map.mapKeysMonotonic unpackHDS (hsDocIdEnv hsDoc) 
        
               in Map.lookup s env

Summary: This extends my work from D4758 and enables haddock to generate docs from information enclosed in the `.hi` interface files. * `HaddockLex` contains an Alex-based lexer for docstrings that detects identifiers delimited with `'` or backticks. * `HsDoc.Docs` comprises the `DeclDocMap` and `ArcDocMap` structures added in D4758 and other data needed by haddock. * haddock now uses `GhcMake.load'` to load `.hi`-files, hooking into GHC's recompilation avoidance. In order to avoid recompilation when a `.hi`-file already contains documentation we use a new option `Opt_SkipIfaceVersionCheck`. * [ ] Work out a specification for a deterministic module structure (`docs_structure`) and update the existing code to implement it. The `HaddockIssue849` test case is currently flaky because the ordering of constructors in the `Avail` for `Maybe` fluctuates. * [ ] Possibly fuse `mi_exports` and `docs_structure`. Currently both serialize the same `Avails`. * [ ] Get the recompilation avoidance right: While some of the checks performed in `checkVersions` are irrelevant for haddock, we may want to keep some of them. We probably also want `-haddock` to cause recompilation if an existing `.hi`-file doesn't contain the `Docs`. * [ ] Fix the tests. `./validate` currently reports failures for `TEST="HaddockIssue849 ImpSafeOnly01 ImpSafeOnly02 ImpSafeOnly03 ImpSafeOnly04 ImpSafeOnly05 ImpSafeOnly06 ImpSafeOnly07 ImpSafeOnly08 ImpSafeOnly09 ImpSafeOnly10 Splices T13168 T14304 T4029 T9881 break006 break011 break013 break024 cabal01 dynamicToo002 dynamicToo005 haddock.base haddock.compiler hist001 hist002 plugins09 plugins11 print026 safePkg01"` Several more (minor) issues are marked `TODO` in the source code. --- The corresponding changes in haddock are [here](haskell/haddock#906). * Add an option to configure the identifier delimiters the lexer looks for. * Add an option to make the lexer exclude birdtracks. * Make the `:doc` command also pretty-print arg docs. Reviewers: bgamari, alanz, harpocrates, alexbiehl, hvr Subscribers: Tritlo, Fuuzetsu, thomie, rwbarton, mpickering, carter, wz1000, harpocrates, gershomb, hvr, alexbiehl GHC Trac Issues: #15786 Differential Revision: https://phabricator.haskell.org/D5067

haddock-api/src/Haddock/Convert.hs

Summary: This extends my work from D4758 and enables haddock to generate docs from information enclosed in the `.hi` interface files. * `HaddockLex` contains an Alex-based lexer for docstrings that detects identifiers delimited with `'` or backticks. * `HsDoc.Docs` comprises the `DeclDocMap` and `ArcDocMap` structures added in D4758 and other data needed by haddock. * haddock now uses `GhcMake.load'` to load `.hi`-files, hooking into GHC's recompilation avoidance. In order to avoid recompilation when a `.hi`-file already contains documentation we use a new option `Opt_SkipIfaceVersionCheck`. * [ ] Work out a specification for a deterministic module structure (`docs_structure`) and update the existing code to implement it. The `HaddockIssue849` test case is currently flaky because the ordering of constructors in the `Avail` for `Maybe` fluctuates. * [ ] Possibly fuse `mi_exports` and `docs_structure`. Currently both serialize the same `Avails`. * [ ] Get the recompilation avoidance right: While some of the checks performed in `checkVersions` are irrelevant for haddock, we may want to keep some of them. We probably also want `-haddock` to cause recompilation if an existing `.hi`-file doesn't contain the `Docs`. * [ ] Fix the tests. `./validate` currently reports failures for `TEST="HaddockIssue849 ImpSafeOnly01 ImpSafeOnly02 ImpSafeOnly03 ImpSafeOnly04 ImpSafeOnly05 ImpSafeOnly06 ImpSafeOnly07 ImpSafeOnly08 ImpSafeOnly09 ImpSafeOnly10 Splices T13168 T14304 T4029 T9881 break006 break011 break013 break024 cabal01 dynamicToo002 dynamicToo005 haddock.base haddock.compiler hist001 hist002 plugins09 plugins11 print026 safePkg01"` Several more (minor) issues are marked `TODO` in the source code. --- The corresponding changes in haddock are [here](haskell/haddock#906). * Add an option to configure the identifier delimiters the lexer looks for. * Add an option to make the lexer exclude birdtracks. * Make the `:doc` command also pretty-print arg docs. Reviewers: bgamari, alanz, harpocrates, alexbiehl, hvr Subscribers: Tritlo, Fuuzetsu, thomie, rwbarton, mpickering, carter, wz1000, harpocrates, gershomb, hvr, alexbiehl GHC Trac Issues: #15786 Differential Revision: https://phabricator.haskell.org/D5067

haddock-api/src/Haddock/Convert.hs

Summary: This extends my work from D4758 and enables haddock to generate docs from information enclosed in the `.hi` interface files. ## Main changes * `HaddockLex` contains an Alex-based lexer for docstrings that detects identifiers delimited with `'` or backticks. * `HsDoc.Docs` comprises the `DeclDocMap` and `ArcDocMap` structures added in D4758 and other data needed by haddock. * haddock now uses `GhcMake.load'` to load `.hi`-files, hooking into GHC's recompilation avoidance. In order to avoid recompilation when a `.hi`-file already contains documentation we use a new option `Opt_SkipIfaceVersionCheck`. ## TODOs and requests for comment * [ ] Work out a specification for a deterministic module structure (`docs_structure`) and update the existing code to implement it. The `HaddockIssue849` test case is currently flaky because the ordering of constructors in the `Avail` for `Maybe` fluctuates. See [haddock#956](haskell/haddock#956). * [ ] Possibly fuse `mi_exports` and `docs_structure`. Currently both serialize the same `Avails`. Also discussed at [haddock#956](haskell/haddock#956). * [ ] Get the recompilation avoidance right: While some of the checks performed in `checkVersions` are irrelevant for haddock, we may want to keep some of them. We probably also want `-haddock` to cause recompilation if an existing `.hi`-file doesn't contain the `Docs`. * [ ] Fix the tests. `./validate` currently reports failures for `TEST="HaddockIssue849 ImpSafeOnly01 ImpSafeOnly02 ImpSafeOnly03 ImpSafeOnly04 ImpSafeOnly05 ImpSafeOnly06 ImpSafeOnly07 ImpSafeOnly08 ImpSafeOnly09 ImpSafeOnly10 Splices T13168 T14304 T4029 T9881 break006 break011 break013 break024 cabal01 dynamicToo002 dynamicToo005 haddock.base haddock.compiler hist001 hist002 plugins09 plugins11 print026 safePkg01"` * [ ] Serialize instance locations. Discussed at [haddock#921](haskell/haddock#921). Several more (minor) issues are marked `TODO` in the source code. --- The corresponding changes in haddock are [here](haskell/haddock#906). ## Possible further work * Add an option to configure the identifier delimiters the lexer looks for. * Add an option to make the lexer exclude birdtracks. * Make the `:doc` command also pretty-print arg docs. Reviewers: bgamari, alanz, harpocrates, alexbiehl, hvr Subscribers: Tritlo, Fuuzetsu, thomie, rwbarton, mpickering, carter, wz1000, harpocrates, gershomb, hvr, alexbiehl GHC Trac Issues: #15786 Differential Revision: https://phabricator.haskell.org/D5067

Summary: This extends my work from D4758 and enables haddock to generate docs from information enclosed in the `.hi` interface files. * `HaddockLex` contains an Alex-based lexer for docstrings that detects identifiers delimited with `'` or backticks. * `HsDoc.Docs` comprises the `DeclDocMap` and `ArcDocMap` structures added in D4758 and other data needed by haddock. * haddock now uses `GhcMake.load'` to load `.hi`-files, hooking into GHC's recompilation avoidance. In order to avoid recompilation when a `.hi`-file already contains documentation we use a new option `Opt_SkipIfaceVersionCheck`. * [ ] Work out a specification for a deterministic module structure (`docs_structure`) and update the existing code to implement it. The `HaddockIssue849` test case is currently flaky because the ordering of constructors in the `Avail` for `Maybe` fluctuates. See [haddock#956](haskell/haddock#956). * [ ] Possibly fuse `mi_exports` and `docs_structure`. Currently both serialize the same `Avails`. Also discussed at [haddock#956](haskell/haddock#956). * [ ] Get the recompilation avoidance right: While some of the checks performed in `checkVersions` are irrelevant for haddock, we may want to keep some of them. We probably also want `-haddock` to cause recompilation if an existing `.hi`-file doesn't contain the `Docs`. * [ ] Fix the tests. `./validate` currently reports failures for `TEST="HaddockIssue849 ImpSafeOnly01 ImpSafeOnly02 ImpSafeOnly03 ImpSafeOnly04 ImpSafeOnly05 ImpSafeOnly06 ImpSafeOnly07 ImpSafeOnly08 ImpSafeOnly09 ImpSafeOnly10 Splices T13168 T14304 T4029 T9881 break006 break011 break013 break024 cabal01 dynamicToo002 dynamicToo005 haddock.base haddock.compiler hist001 hist002 plugins09 plugins11 print026 safePkg01"` * [ ] Serialize instance locations. Discussed at [haddock#921](haskell/haddock#921). Several more (minor) issues are marked `TODO` in the source code. --- The corresponding changes in haddock are [here](haskell/haddock#906). * Add an option to configure the identifier delimiters the lexer looks for. * Add an option to make the lexer exclude birdtracks. * Make the `:doc` command also pretty-print arg docs. Reviewers: bgamari, alanz, harpocrates, alexbiehl, hvr Subscribers: Tritlo, Fuuzetsu, thomie, rwbarton, mpickering, carter, wz1000, harpocrates, gershomb, hvr, alexbiehl GHC Trac Issues: #15786 Differential Revision: https://phabricator.haskell.org/D5067

Merge ghc-head into wip/hi-haddock

bgamari · 2019-09-12T20:15:36Z

What is the status of this?

sjakobi · 2019-09-13T16:12:34Z

It's good that you're asking, @bgamari. !1648 reminded me of some pretty cool work by @harpocrates among the other Hi Haddock stuff that resulted in proper haddocks for [] and (->).

I had got stuck on #921 last winter, and had little time for this project. In March I ended up asking @harpocrates to take over the remaining work.

Back then I was under the impression that there were still quite a few tricky (tricky for me) issues to resolve. AFAIK the latest TODO list is this one. What made everything more difficult was the need to sync (and rebase) things between haddock and ghc.

I'm wondering whether there could be a way to merge parts of the work incrementally, reducing the amount of changes outside of master that have to be kept in sync.

@harpocrates what's your perspective on all this these days?

harpocrates · 2019-09-13T18:58:00Z

Sorry, I'm definitely at fault for dragging my feet on this. I thought I'd negotiated 2 weeks of paid time for this with my employer last April, but that ended up not panning out. Lately I have a bit more time - perhaps even as early as this month...

Location information of instances is lost if there is aleady a '.hi' #921 isn't of immediate concern. I think initially Hi Haddock should unconditionally re-generate .hi files with docstrings (the same way it currently unconditionally re-generates .hie files for hyperlinking). When we get to the optimization where Haddock potentially re-uses .hi/hie files, we'll have to deal with Location information of instances is lost if there is aleady a '.hi' #921. Besides, that line of progress will require some work on Cabal too.
I've already cherry-picked a bunch of the improvements that initially started in the hi-haddock branch of Haddock in an effort to trim this PR down (More synify-ing improvements #931, Default signatures #938, Keep forall on H98 existential data constructors #1003, Fix #1004 with a pinch of dropForAlls #1005, Fix #1015 with dataConUserTyVars #1022, Miscellaneous improvements to Convert #1020, Small fixes #1045, and probably some more). Post-rebase, I expect the Haddock diff to be relatively small.
The Haddock-side is done, except for rebasing. The remaining work is really the TODO list (thank @sjakobi for the link): https://gitlab.haskell.org/ghc/ghc/merge_requests/608/#note_189219.

Kleidukos · 2021-05-15T16:27:48Z

What is our current status? Is there a way to better split this PR?

Kleidukos · 2021-05-15T16:28:07Z

@harpocrates @sjakobi

Kleidukos · 2021-12-17T20:13:52Z

Hi! As of today, patches for the ghc-head branch must be done on the Haddock mirror on GitLab, at https://gitlab.haskell.org/ghc/haddock.

sjakobi added 30 commits August 14, 2018 03:16

First steps adjusting to HsDoc

9b55d12

Add hack that creates a HsDoc from a warning

ea274d3

Fix renameLDocHsSyn

0b1d4e9

Adjustments for new WarningTxt type

a29151b

Corresponding change in ghc: sjakobi/ghc@19adbf6.

First progress

4ce9b69

(cherry picked from commit 396961f)

Document how to populate Interface fields

369c813

Resolve type errors with undefineds

6414442

Use haddock opts from ModIface

0d532db

Spread ErrMsgGhc type, add Renamer type

ce0fe21

Add TODO for mkWarningMap

d836aa0

rename: Use HasDynFlags ErrMsgGhc instance

8d47a60

Use getDynFlags instead of the dflags from ModSummary

14659ba

rename: Use a Renamer instead of a GlobalRdrEnv

2078fdf

Adjust processModuleHeader

e9b07a1

Use docs_language and docs_extensions

0dce2f0

Get module warning

2b34210

Fill in ifaceDocMap

995c5b6

Fill in ifaceArgMap

674bcec

Fix mkWarningMap, use it

27d6f66

ifaceOrigFilename: Note idea

f17fd01

ifaceOrigFilename: Won't fix

b692803

Add notes on ifaceOrigFilename

6448107

Adjust mkFixMap

9d70a90

ifaceModuleAliases: Add comment

3550672

Add TODOS

4ed73c9

Get class and family instances via ModDetails

fdde4f5

Fill in ifaceDeclMap

58410ce

mkExportItems': Initial scaffolding

b982aa5

Construct export items

6c630ea

We're stilling missing some info. See FIXMEs.

Comment on ifaceVisibleEXports

2ff77de

sjakobi added 2 commits October 23, 2018 10:59

Silence a warning

dd553ed

Adjust renaming to new Docs format

cad2ed9

sjakobi added 4 commits October 26, 2018 17:22

Stop using NonDetKeyMap

3bd504b

hsDocRenamer: Clarify intended usage

1713905

Remove some dead code

9d8554d

Adjust to changed definition of docs_extensions

dea4275

sjakobi commented Jan 5, 2019

View reviewed changes

haddock-api/src/Haddock/Convert.hs Outdated Show resolved Hide resolved

Merge remote-tracking branch 'origin/ghc-head' into HEAD

17d9720

sjakobi commented Jan 5, 2019

View reviewed changes

haddock-api/src/Haddock/Convert.hs Outdated Show resolved Hide resolved

synifyType: Properly adjust to visual kind annotation changes

cff4abf

harpocrates added 2 commits February 4, 2019 07:00

Merge branch 'ghc-head' into merged-convert-fixes

282f174

Merge pull request #1021 from harpocrates/merged-convert-fixes

1ad8a10

Merge ghc-head into wip/hi-haddock

wz1000 mentioned this pull request Jun 10, 2020

Use functions exported from GHC.HsToCore.Docs #1204

Merged

bgamari mentioned this pull request Dec 15, 2020

Enable two warnings (#1245) #1268

Merged

Kleidukos closed this Dec 17, 2021

Kleidukos deleted the wip/hi-haddock branch December 27, 2021 14:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[WIP] Hi Haddock #906

[WIP] Hi Haddock #906

Uh oh!

sjakobi commented Aug 14, 2018 •

edited by harpocrates

Loading

Uh oh!

sjakobi commented Oct 22, 2018

Uh oh!

sjakobi commented Oct 26, 2018

Uh oh!

Uh oh!

Uh oh!

bgamari commented Sep 12, 2019

Uh oh!

sjakobi commented Sep 13, 2019

Uh oh!

harpocrates commented Sep 13, 2019

Uh oh!

Kleidukos commented May 15, 2021

Uh oh!

Kleidukos commented May 15, 2021

Uh oh!

Kleidukos commented Dec 17, 2021

Uh oh!

Uh oh!

[WIP] Hi Haddock #906

[WIP] Hi Haddock #906

Uh oh!

Conversation

sjakobi commented Aug 14, 2018 • edited by harpocrates Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What already works

TODOs

Uh oh!

sjakobi commented Oct 22, 2018

Uh oh!

sjakobi commented Oct 26, 2018

Uh oh!

Uh oh!

Uh oh!

bgamari commented Sep 12, 2019

Uh oh!

sjakobi commented Sep 13, 2019

Uh oh!

harpocrates commented Sep 13, 2019

Uh oh!

Kleidukos commented May 15, 2021

Uh oh!

Kleidukos commented May 15, 2021

Uh oh!

Kleidukos commented Dec 17, 2021

Uh oh!

Uh oh!

sjakobi commented Aug 14, 2018 •

edited by harpocrates

Loading