Support single files and parent (non-Terraform) folders

[radeksimko]

Single files

A restriction is currently in place that prevents the Language Server from communicating with an IDE (client) that opened a single file. As a result it enforces the user to open whole folders.

This is done by failing initialization with the following error:

Editing a single file is not yet supported. Please open a directory.

Why

The restriction prevents the LS from having to deal with the potentially increased complexity in any logic that concerns the relationship between a file and its plugins (providers).

The LS has responsibility for:

• finding compatible Terraform binary (happens during initialize via $PATH)
• finding and storing Terraform version (happens during initialize)
• retrieving and caching schema for inited providers (happens during initialize)
• invalidating the cache and retrieving it again when providers change (happens any time via watching the lock file in a plugin folder)

The initialize method is called only once for every opened folder, which is mainly where the lower complexity comes from.

Parent (non-Terraform) folders

This also affects users who wish to open a folder higher up the filesystem hierarchy, e.g. a folder with all Terraform workspaces, as opposed to opening workspaces individually. We do not support such case today either, but don't actually raise any error in this case (yet?).

The same reasoning and complexity scope applies here too though and it's likely that both use cases have the same (or very similar) solution.

Local Modules

Users with locally stored modules (where module's source is path in a local filesystem) may fall into this category as such modules tend to be not inited within their own folder (i.e. such modules often don't have their own .terraform folder). Supporting them will rely heavily on how their provider inheritance is set up - i.e. we need to understand where can we get the provider schemas from for every module.

Future

If we support single files, we need to consider the above happening at different times (probably during didOpen) and significantly more often. Hence we need to ensure this scales well in terms of resource consumption - e.g. by ensuring that we never watch/refresh the same plugin directory twice within the context of a running server process. We may also need to decouple the Terraform binary discovery and version handling, to ensure we only do it once per workspace.

Related

This limitation also helps us avoid bugs similar to juliosueiras/terraform-lsp#58

Proposal

• walker: introduce more limited walking mode, such that it doesn't descend into lower directories (to prevent unexpected outcomes if user opens ~ home dir)
• initialize handler: avoid indexing if rootUri is empty (single file was open) and store a flag to say "LS is in single file mode"
• didOpen handler: index on-demand if "LS is in single file mode" - basically just call

  terraform-ls/internal/langserver/handlers/did_change_workspace_folders.go

  Lines 62 to 77 in 99e30e1

  	modPath, err := uri.PathFromURI(added.URI)
  	if err != nil {
  	jrpc2.ServerFromContext(ctx).Notify(ctx, "window/showMessage", &lsp.ShowMessageParams{
  	Type: lsp.Warning,
  	Message: fmt.Sprintf("Ignoring new workspace folder %s: %s."+
  	" This is most likely bug, please report it.", added.URI, err),
  	})
  	continue
  	}
  	err = watcher.AddModule(modPath)
  	if err != nil {
  	svc.logger.Printf("failed to add module to watcher: %s", err)
  	continue
  	}
  	walker.EnqueuePath(modPath)

[amasover]

Users with locally stored modules (where module's source is path in a local filesystem) may fall into this category as such modules tend to be not inited within their own folder

Potentially of note: terragrunt users usually have the .terraform folder stored outside the module folder, and don't directly init on modules. But I can't think of a clean way to not require a direct init on the module in that case.

I have probably a minority use case, but I actually use pre init hooks in terragrunt to set up my backend configs. The way I have things set up, terraform init fails when run directly against a module unless I set the backend to "local" temporarily. I wonder if #56 would help me. Or perhaps I should consider changing my modules in a way that init-ing against them directly runs successfully (without breaking my pre init hooks).

[radeksimko]

Thanks for sharing that use case @amasover - that's a useful insight!

I am not that familiar with Terragrunt and this kind of workflow where .terraform folder is outside of the directory. Would the 1-to-1 mapping between .terraform and a directory still apply in this case, or would Terragrunt share a single .terraform with multiple directories?

If it's 1-to-N, would this functionally be the same as setting a custom plugin cache?

In theory we could traverse up a few levels to lookup parent .terraform, but in practice I worry that such design would be flawed in that we cannot be sure whether the directory genuinely belongs to that .terraform, i.e. there's a lot of room for error, so I'd rather avoid doing that.

That said we could initially allow setting a static plugin cache path per #24 and potentially make it project-specific settings via some kind of standardised configs?

FWIW the only reason the LS needs access to .terraform is for schema for now, but I can imagine us also needing access to downloaded modules eventually, so we can complete references to outputs from modules for example in module.servers.<HERE>, but that's one for #93

[amasover]

The workflow is something more like this:

terraform-live repository:

staging
├── terragrunt.hcl
├── frontend-app
│   ├── .terragrunt-cache/..../module-for-frontend/.terraform
│   ├── main.tf
│   └── terragrunt.hcl
└── mysql
    ├── .terragrunt-cache/..../module-for-mysql/.terraform
    ├── main.tf
    └── terragrunt.hcl
production
├── terragrunt.hcl
├── frontend-app
│   ├── .terragrunt-cache/..../module-for-frontend/.terraform
│   ├── main.tf
│   └── terragrunt.hcl
└── mysql
    ├── .terragrunt-cache/..../module-for-mysql/.terraform
    ├── main.tf
    └── terragrunt.hcl

terraform-modules repository:

module-for-frontend
└── main.tf
module-for-mysql
└── main.tf

(No .terraform dirs in the terraform-modules repository)

The terragrunt.hcl files contain references to modules like this:

terraform {
  source = "git::git@mygithost/terraform-modules//module-for-frontend"
}

Terragrunt then pulls down the module and runs the terraform command that you want against it.

The best example of how this works is Terragrunt's example repos:
https://github.com/gruntwork-io/terragrunt-infrastructure-live-example
https://github.com/gruntwork-io/terragrunt-infrastructure-modules-example

In local development usually you reference a local source. So a typical loop would be, make a change in a module, and try it out from a terraform-live environment via terragrunt apply --terragrunt-source /home/$USER/code/terraform-modules//module-for-frontend.

I think the problem here is that it's more like an N-to-1 mapping where multiple copies of modules can exist, for different environments in terraform-live. For example, you might have a staging environment and a production environment, both referencing module-for-frontend, and each environment-specific copy of the module-for-frontend has its own .terraform dir. That's kind of why I'm saying I don't see a clean way for terraform-ls to solve this problem.

Ultimately though, when you edit a module, you're doing it in the terraform-modules directory, not in the terraform-live directory. So your editor should see the project root being somewhere in terraform-modules, and look there for the .terraform dir. So I'd argue it probably isn't too much of a burden to ask that people run terraform init directly against modules. My case is a little special due to my custom pre-init hooks, but (and I could be wrong), I think usually people who use terragrunt still retain the ability to directly init modules.

I think static plugin cache with standardized configs would also be great in this situation -- then you could check the configs into source control, and it should work for everyone who needs to edit modules. My understanding is, if the LS supports that, then we wouldn't need to directly init in the terraform-modules directory, because the LS can get schema info directly from the plugin cache? I tried to set up juliosueiras's terraform-lsp in this way, but I was only ever able to get language server features to work after running terraform-init directly from modules.

[paultyng]

Different scenarios (will start to populate these with examples):

Repo / Directory Scenarios

Single root module / repo

No subdirectory modules, all modules referenced from external sources (registry, etc). Repo root is Terraform root.

Nested submodule

Repo root is terraform root, subdirectories are modules sourced from the root.

Example:

Our internal terraform-repositories repo for GitHub management. I'm trying to find some good public examples.

Nested root module

Repo root is module, sub directories can contain multiple root Terraform modules.

Example:

Open at repo root: https://github.com/terraform-aws-modules/terraform-aws-security-group
The root of the repo is actually a module, not a terraform root module.
The terraform init and schema cache would be invoked from a nested example:
https://github.com/terraform-aws-modules/terraform-aws-security-group/blob/master/examples/computed/main.tf#L20-L35

Multiple Root Terraform

Repo root is not a Terraform module, subfolders are root Terraform modules.

This is probably easier solved on the editor side, but opening each subfolder as an additional directory in a VS Code "workspace" (or Sublime "project"), etc. and running multiple language servers for each one.

Submodule only / No Terraform root

If someone is working on a submodule in a repository without any test root Terraform modules, there may be no way to acquire schema information / initialize. Mostly listing this for completeness, we could potentially try to guess at the providers/schema. Its possible you could walk up the tree and see if there was an initialized TF directory somewhere above the repo?

Single File Scenarios

I think for the most part we should skip all but the single file in a repo root scenario in the near term.

Single file in Terraform root

This is relatively straightforward, probably follows the same path as repo root being Terraform root.

Single file not in Terraform root

Similar to the Submodule only scenario above, you could potentially walk up the tree, but otherwise not sure what else you could do.

[magne]

I'm using the Main Module Approach with CI, QA, and Prod environments. The CI environments includes the main module as a local module (`source = "../../main"), while the QA and Prod environments use git module source. This allows me to pin QA and Prod to a specific commit or tag.

It would be helpful to be able to select the directory containing the terraform root.