Use spago for managing the server package set

[hdgarrood]

This gives us an easy way of depending on the entire package set at
once, so that any package in the set can be used in Try PS.

[thomashoneyman]

It’s great news that the entire set is available! Just to check: what would actually be compiled on Try PureScript when a user makes changes to their code?

[hdgarrood]

The whole package set would be compiled first, before the server starts listening for requests, so only one module would ever be compiled per request. On my laptop it takes around 34s to compile the whole set (or 5m of CPU time).

[natefaubion]

Do we want the whole set available? Do we know what modules require FFI shims?

[hdgarrood]

I think having the whole set available makes sense; I'm sure a lot of libraries will require shims, but in the case of libraries which need shims which aren't there, the UX isn't really any worse, you just get a slightly different error. It's not as if users have any good way of knowing what modules they can expect to be available on Try PS at the moment.

[natefaubion]

Is there any memory concern for the server compiling against the whole set?

[hdgarrood]

Oh right, yes, that's a good point. Compiling the whole set to start with is fine -- the CI for the package-sets repo manages that -- but I think the problem arises because tryps keeps all of the externs files in memory, and it also keeps a prebuilt JS bundle of the entire package set in memory (which is exposed via the /bundle route). On my laptop, running tryps with the whole package set consumes about 3.3GB of memory; removing the /bundle route brings that down to about 2.1GB. I think running tryps on a server with sufficient memory to handle this is doable if we raise a little more via opencollective, but the fact that the package set will probably get larger over time additionally complicates things. I think if we want to provide the whole package set it may be worth loading externs files from disk, or working out some kind of caching mechanism so that, say, up to 500MB of the most frequently used externs files get to stay in memory, or something like that.

[natefaubion]

The bundle.js should be unnecessary now. I think optimizations to extern size and sharing should help, but I agree that some sort of smarter strategy is necessary.

[hdgarrood]

I am halfway through preparing a PR which removes bundle.js 😄