The dep workflow: default constraints, minimal manifest?

[nathany]

This is a meta-issue after some discussion with @sdboyer on #vendor to better understand the dep tool and to both explain my understanding and present what I think the workflow could be like based on what I currently know. Currently I'm only tackling the simple case of building a new app.

The dep tool is unique compared to its contemporaries because it is aware of the import graph contained in source code. That means the import statements in your source code are the source of truth. Bundler, Hex, NPM, etc. look to a manifest file for a list dependencies you manually maintain. This impacts what the so-named (#168) "manifest" file is actually for.

So to use dep, you can start by just writing code. At some point you need to dep init to create the manifest and lock file. The lock file is a full transitive list of all imported code with commit SHAs that make reproducible builds possible. The "manifest" need not list all the dependencies and constraints, in fact it could be essentially empty†. Let me explain.

Say you import "foo/bar" and run dep ensure to ensure that everything imported is correctly vendored and tracked.

The dep tool can be smart enough to fetch the latest stable version v1.3.4 of "foo/bar" rather than pulling master (as with go get) and rather than pulling v2.0.0.beta.1. Pre-releases are always opt-in, as the solver gives preference to stable versions.

At this point there is no need for anything to exist in the manifest, but the lock file will contain a specific commit SHA for "foo/bar" which you commit along with your code (and optionally the vendored copy of "foo/bar" for the hermetic view of the world).

Now lets say you import "baz/qux". Even if "foo/bar" v1.3.5 is available, running dep ensure should vendor "baz/qux" but not touch "foo/bar" at all.

A separate command should exist to dep update foo/bar, which would upgrade to v1.3.5. If no constraints are manually specified, dep update x always gets you the latest stable version, so once v2.0.0 is out, dep update foo/bar would give you that.

IMO, this should be fine for most cases. Upgrading versions is always done in a controlled manner and builds are reproducible thanks to the lock file. Additionally, you can use dep status to see what's going to happen before running dep update x and simply choose not to update to new major versions immediately.

The "manifest" comes into play when the default behaviour isn't what you want.

• To opt-in to a pre-release version of a dependency ^v2.0.0.beta.1
• When two downstream dependencies have conflicting constraints, the manifest can specify an override to determine which one wins.
• Or generally to override the preferences of a downstream dependency if you know better.
• To lock to an older version due to an incompatibility, license change, or other consideration.
• To use a temporary fork in place of a given dependency.

These are all exceptions to the general rule of latest stable version. In all these cases, it is helpful to include a comment in the manifest indicating why the constraint is specified.

[dependencies]
"foo/bar" = "^v1.3.5" # v2 is incompatible with baz/qux

Next time you run dep ensure, the constraints in the manifest will override the default behaviour (latest stable version) when solving. However, if v2.0.0 is already vendored, you may see an error, and need to run dep update to update to the older specified version.

How does this sound so far?

If this makes sense, I'd suggest a few changes that could help direct some open issues:

• The "manifest" should be a human-editable file that is created on init and never modified by the tool. Comments and organization of the file are up to the humans maintaining it. The initial file should include a wall of comments as documentation on usage. To support comments and for readability, the format preferably wouldn't be JSON (Move to TOML for manifest and lock #119), but the lock file could remain JSON.
• The dep ensure foo/bar variant should go away in favour of a separate update command. Neither command should alter the manifest. To add a dependency, write an import statement in your Go code, and run dep ensure to vendor the latest stable version.
• The "manifest" probably shouldn't be called "manifest" (Discuss renaming lock.json and manifest.json #168). It provides a way to override default behaviours. It's not a full list of everything (the lock file is).
• The manifest is for specifying constraints and overrides. It is an error if the manifest specifies a package that isn't ever imported (much like unused variable/import in Go).
• If supported, dep update foo/bar v1.3.4 could override any constraints specified in the manifest too, vendoring v1.3.4. But assuming this doesn't update the manifest (not worth the trouble), maybe it's not worth supporting constraints on the CLI.

The end result of all this is a manifest designed for humans but that rarely needs editing. For the common case, just write your code, run dep ensure, and commit your work as usual. That's as simple as it gets.

† Yet to confirm that the solver can cope with a large number of unspecified constraints (empty manifest) with a preference for the latest stable version.

[sdboyer]

First, this is an awesome writeup. Thanks for doing this!

I still have some unspecified dread about it being recommended - or at least, standard operating procedure - that constraints are omitted from that manifest. I can't point to a concrete reason for it, but this is a new (to my mind) way of thinking about how we could operate given the unique properties of the tool we have. And in a complex space like package management, one with significant and permanent consequences, I view new things with some trepidation :)

That said, I do want to like this direction, and I think we should consider it heavily over the coming months of experimentation. With dep/gps, we/I made a very deliberate choice to still have the import graph be queen, rather than replicating it into the manifest. That's what makes an approach like this possible at all. It's also the single property that makes this tooling very idiomatically Go, as it arises naturally from the way that import paths allow for extensible, universal deduction of upstream source location.

As for the elephant in the room:

† Yet to confirm that the solver can cope with a large number of unspecified constraints (empty manifest) with a preference for the latest stable version.

Yeah, so, this is a big deal. The problem here is constraint solving, which is an NP-hard search (as Russ demonstrated). We have to be cognizant of how the design of the tool, and the recommendations we make about its use, could create an ecosystem with pathological solving cases.

I can say some simple things about this:

• If everyone uses empty manifests, then that's a very fast solve - there's no possibility of (version constraint) conflicts, so the first solution that the solver visits is likely to work. And that's a great solution...if all the versions of code selected actually work with one another.

• It's possible - and if it turns out to be computationally feasible, I would like to do this - that we may introduce additional satisfiability checks in gps' solver in the future. For example, we may do an intransitive type compatibility check (Implement package type/api compatibility checking sdboyer/gps#67). I think this would be fairly magical, especially if not specifying constraints is a norm - the solver could just figure out that two versions won't work together, absent any constraint info, and move on to the next one. But, version constraint checks are generally in the 10s of nanoseconds range, so we do them first, as these other checks are much more expensive. Turns out, magic ain't free.

• The real pathological solving cases are ones where the solver has to work through a large number of variables in order to try one solution, discover than solution doesn't work, backtrack and work through another large set to find that one doesn't work. One way this might happen is if there's a shared/diamond dependency deep in the graph - with both conflicting parents at least two levels away from the root - but the interstitial dependencies have a large number of versions and no constraints on them. That could result in the dreaded exponential search path, as the solver explores all the interstitial versions but keeps getting bitten by the further-removed constraint.

  It's possible we could make this situation more likely to occur if the norm is to have unconstrained imports (the interstitials), only adding constraints when we need them. Of course, it's also possible that some changes in solver ordering logic might obviate this as a risk, too :)

• I am somewhat concerned that if the user starts by working in the carefree wonderland of an empty manifest, and is then suddenly hit with a solve error, then they'd be setting constraints in anger, and may be more inclined to ratchet down the constraint more than they really need to. Ratcheting down constraints like that is locally optimal, but globally harmful - if everyone does it, the ecosystem can seize up, nobody can rely on anyone else's constraints anymore, and everyone has to use overrides (aka the "take my toys and go home" lever).

I've opened sdboyer/gps#165 as one way of trying to get out ahead of this problem a bit - if we can generate random graphs that help identify patterns that gps' solver has problems with, then it could give us a better empirical sense of where the risks may lie.

Beyond that, a couple things, mostly nits:

However, if v2.0.0 is already vendored, you may see an error, and need to run dep update to update to the older specified version.

First, a clarification - dep doesn't care what's vendored when it comes to making decisions about versions to select. It treats vendor as volatile - nothing more than a concrete instantiation of the transitively complete requirements specified in the lock.

So, assuming you meant "if v2.0.0 is what's in the lock": there's no error here, at least not from the solver. There's no flag that prevents incidental downgrades - as long as the version is still admissible according to constraints/other criteria, it's fair game. So If there's an error for this case, it's something dep would have to tack on, or that we'd have to add a new rule for within the solver. (Neither is impossible - I'm just noting the current state of affairs)

At this point there is no need for anything to exist in the manifest, but the lock file will contain a specific commit SHA for "foo/bar" which you commit along with your code

Anything the solver picks on its own is going to be a rev with a corresponding tag or branch (there's one possible exception to this, but not relevant here) - it will record both the tag or branch AND the rev.

Now lets say you import "baz/qux". Even if "foo/bar" v1.3.5 is available, running dep ensure should vendor "baz/qux" but not touch "foo/bar" at all.

In general, this is true. ensure tries to minimize the amount of not-explicitly-requested change that happens. But it doesn't fail/error if that's not possible - it'll search for other solutions. So, if there's a dependency shared between "foo/bar" and "baz/qux" and there's some conflict on it with v1.3.4 of "foo/bar", it is possible that "foo/bar"'s version might end up at v1.3.5 afterwards.

[nathany]

Thanks. I'll pretend that I understand half of that, but I think I get the gist, and I appreciate the experimentation in this direction.

The reason behind this line of though for me is the confusion I see when there is a CLI command to add dependencies to the manifest. That suggests a different workflow that is competing with what the import graph brings. Unions of the manifest and import graph, locking and vendoring things that aren't imported yet, I can feel it getting messy. I think dep can and should avoid that in one way or another. This isn't the only way...

Making the manifest minimal puts more emphasis on just writing code and adding imports.

The type compatibility check would be a pretty awesome way to enhance the solver by making the default behaviour smarter, which may not be possible if the everything is persisted to a manifest instead of allowing recalculation.

If the manifest file is only machine-modified, through some series of CLI commands, outputting everything to the file is okay because it can be updated with every run, but then why have a separate manifest and lock file?

It's a different story if the manifest file is intended to be a primary part of the dep UI with just a few simple CLI commands to enforce it.

In the magical world where it's both human & machine editable with comments and order preserved, would it be a too magical for the tool to muck with that manifest file's constraints on behalf of the user? Perhaps not.

But if so, making the manifest constraints optional seems like the best option.

Here's a simple case:

One of my projects uses import "golang.org/x/net/http2". This is one of several repositories without version tags. There is no latest stable version, so the fallback is branch: master. What could be written to a manifest file in this case? In the current version (e93d9b5) of dep:

"golang.org/x/net": {
    "branch": "master"
}

After dep becomes wildly successful and integrated into the go toolchain, everybody starts using semver so fast forward to the future where there is a v1.0.0 tag on golang.org/x/net.

What does dep update golang.org/x/net do? I sure want it to start using v1.0.0, but does that mean rewriting the manifest file? Do I need to go around changing constraints before running the update, and will I do the right thing tag: ^v1.0.0?

If we can get that magical world of a human/machine editable manifest that doesn't feel surprising in use, that may be best because it provides a visual of all the immediate dependencies without having to hand write it. Otherwise, I quite like the idea of specifying the minimum necessary, and using the CLI to provide visuals of the graph (status).

As far as update vs. ensure and reporting errors, what I'm trying to get at is providing a means to add a dependency in an isolated way. This isn't always possible because of shared dependencies, but I don't want to upset the lock file and vendor folder more than necessary. Gradual, controlled additions and upgrades are more my preference (keep it green!). It sounds like this is already the case 👍

[nathany]

Apologies for the wall of text. I'm thinking "out loud" and had another thought.

Consider this not-so-uncommon diamond scenario where two packages have the same dependency. My project depends on food and power which both depend on sun v1.0.0. After the release of sun v2.0.0 the power library adopts the new version immediately, but food is still happily using the 1st generation sun library.

When I go to upgrade power, I get a conflict that requires some user input.

After inspecting the code, I notice that food will operate unchanged with sun v2.0.0. Whether it never used the modified APIs in the first place, or it wasn't the breaking change the version suggested. So I want to specify an override of some sort:

[overrides]
"sun" = "^v2.0.0"

This isn't here to say that my project depends on sun. It's an override to resolve a downstream conflict.

NOTE: I really like the idea of the type compatibility check. Might later versions of dep hit the conflict and determine to use sun v2 without user intervention?

What happens when things change? Say food begins depending on artificiallight and power begins using nuclear.

What does my sun override do now that sun doesn't exist in the import graph? I think it should become an error similar to imported and not used.

I don't know if I can explain it, but somehow the minimal manifest just feels better to me here. More organized, with less clutter. I'd go so far as to suggest that the manifest doesn't list [dependencies] at all -- instead it could just be a list of [overrides].

The tools in my past (Bundler, Hex) have this a huge list of dependencies, mostly immediate, but some that are hoisted up just to get around conflicts. That happens a lot when early adopting prerelease versions before the rest of the ecosystem catches up. It's a huge jumble, and it's never really clear whether a dependency can be removed.

The import graph promises to clean that up.

What I don't want here is a union of the manifest and import graph. I want to know when I can drop a dependency.

[peterbourgon]

I'm sorry, I'm having trouble following all of the exposition. Do you think you could pare down the "walls of text" to the specific workflow[s] you have in mind?

[nathany]

This has been an exploration of the UX of using dep in one particular direction.

The gist of everything above is to have two inputs into the solver. The import graph and a human-managed "manifest". In the cases above, "manifest" is essentially optional, and can be used to override the default common case (which would be designed to be what people usually want).

This makes the manifest fairly minimal in use (just the exceptions) and the CLI surface minimal too (just enforce the manifest + import graph in a controlled manner).

Alternatives could include:

• A machine-managed manifest modified exclusively by CLI commands (possibly a lot of them), serving essentially the same purpose. In this case, what the manifest contains isn't part of the UX. It should never, ever need to be modified by people and the name (Discuss renaming lock.json and manifest.json #168) and location (Location of manifest and lock file (document) #207) should reflect that. The UX discussion would focus around the CLI.
• A magic rainbows human/machine-managed manifest where the machine doesn't stomp over things humans care about or make surprising changes. It is more difficult to get right (implementation), but it would allow a more complete human-editable manifest, instead of just the overrides.
• Absolute and utter chaos where it's not clear if the user should edit the manifest or use a CLI command, CLI commands with unexpected behaviours (why didn't it vendor?), bizarre special cases like temporarily locking and vendoring things that haven't been used "yet", etc., etc. As in, poor UX that is the result of implementing a lot of cool stuff without giving enough attention to the experience of using it (which I assume is one of the driving reasons why this pre-alpha is available to try out).

Happy to collapse my mutterings and reframe them in terms of the projects direction if desired, preferably after I have a better understanding of what the team's direction with the UX actually is. Thanks!

[sdboyer]

The gist of everything above is to have two inputs into the solver. The import graph and a human-managed "manifest". In the cases above, "manifest" is essentially optional, and can be used to override the default common case (which would be designed to be what people usually want).

Just to be clear, this is how it actually works now. The current project's import graph is represented in SolveParameters.RootPackageTree, and manifest in SolveParameters.Manifest.

[nathany]

@sdboyer I'm concerned with how it feels to use and what expectations users are having when using the tool. How does the UX of working with the CLI and manifest (if applicable) convey how it actually works?

When users see dep ensure github.com/pkg/errors@^0.8.0 in the README and then wonder why running said command didn't lock & vendor pkg/errors, there is a disconnect between the UX as presented and what's actually going on.

When I run dep init and see a bunch of stuff in manifest.json, I don't know just-by-looking whether this manifest is getting overwritten all the time, whether I should be editing it or not, etc.

It took some explaining, which can be turned into documentation, that would be good. It would be great if the documentation wasn't necessary.

What dep is actually doing, thanks to the import graph, is wonderful stuff. I really ❤️ it. How can the tool's CLI best reflect it?

[freeformz]

I was the squeeky wheel here so I want to explain my reasons for championing having dep ensure foo.com/bar/baz@~1.2.3 be a thing. The main motivation was that of convenience. The thought being: Why have to edit the file, manually type in the constraints and then run a command to apply the changes when I can have the tool do the work for me? I've seen that this has caused a lot of concern because of the confusing signaling it's caused.

Moving forward we're going to pursue only having dep create an initial manifest during dep init, but otherwise not modify the manifest, leaving it to the domain of developers to hand modify as necessary. We believe that init provides useful introspection of dependencies in the existing Go ecosystem and as such should continue to create an initial manifest.

I think once these changes are implemented dep will move more towards the system you describe above.