RSA public keys cannot authenticate against internal SSH if the client has a recent ssh version (which disables ssh-rsa algorithm)

[zeripath]

Gitea Version

1.15.x/1.16.x till the latest

Operating System

• Any SSH client with ssh-rsa algorithm disabled (ex: OpenSSH >= 8.8)

How are you running Gitea?

Builtin/Internal SSH (ex: docker-rootless or START_SSH_SERVER=true in app.ini)

Can you reproduce the bug on the Gitea demo site?

Yes

Description

The ssh-rsa key signing algorithm is disabled on recent ssh versions. Although #17281 means that Gitea will offer rsa-sha2-256 and rsa-sha2-512 for the host key verification - it does not fix the issue of advertising and handling the public key signing algorithm.

This failure then results in ssh authentication failing.

How to reproduce

Use a newer version of SSH where this is disabled and attempt to authenticate using an RSA key.

On older versions of SSH:

ssh -o PubkeyAcceptedKeyTypes=-ssh-rsa,+rsa-sha2-256 ...

Unfortunately OpenSSH have changed the name of this option across versions and in newer versions, including the affected version, this option is called: PubkeyAcceptedAlgorithms

So if the above fails with an error try:

ssh -o PubkeyAcceptedAlgorithms=-ssh-rsa ...

Workaround

• On affected clients, add -o PubkeyAcceptedAlgorithms=+ssh-rsa to the ssh command line (In the unlikely event you have an older version which has deliberately disabled the ssh-rsa algorithm you will need to add
  -o PubkeyAcceptedKeyTypes=+ssh-rsa)
• Use other supported SSH key formats
• Use OpenSSH server instead of builtin SSH server (ex: use Docker instead of Docker-rootless)

History of this issue

For speed I'm going to collect things here:

• key exhange negotiation failed though client and server share some protocols #17175 reported that there was a key negotiation failure with newer versions of SSH and Gitea.
• Through investigation this was revealed to be a HostKey algorithm failure due to the key type being used to determine the signing algorithms available for the key.
• Fortunately the HostSigners are exposed on the underlying ssh server so although there is a fundamental flaw in the way signers (their type is not reportable separate from the key) Offer rsa-sha2-512 and rsa-sha2-256 algorithms in internal SSH #17281 was able to work around this by wrapping those keys around something to pretend to be signers.
• This then fixes the hostkey issue - however, I was unaware of the next issue...
• @andypost then commented on Configurable SSL cipher suite #9691 (comment) (Which is unfortunately an unrelated feature request issue relating to HTTPS not SSH.)
• My comment here Configurable SSL cipher suite #9691 (comment) describes that the issue is actually relating to the next step of authentication - the pubkeyalgorithm
• I then discovered that the problem is relating to the pubkey signing algorithms offered to the client Configurable SSL cipher suite #9691 (comment)
• There is an upstream issue: x/crypto/ssh: support for server-sig-algs extension (RFC8308) golang/go#49269 relating to supporting the required extension.
• And there is a PR here: ssh: add support for extension negotiation (rfc 8308) golang/crypto#197 that implements this but I'm not completely convinced its totally correct.
• In the meantime go have merged: golang/crypto@b4de73f
• This does a similar bodge to the one I did in Offer rsa-sha2-512 and rsa-sha2-256 algorithms in internal SSH #17281, however, whereas Offer rsa-sha2-512 and rsa-sha2-256 algorithms in internal SSH #17281 leaves the possibility of disabling certain host key signing algorithms in the future, if we support ssh-rsa we have to support rsa-sha2-256 and rsa-sha2-512, and no way is provided to support only rsa-sha2-256 and rsa-sha2-512. (The above commit does provide certificate support which is something that Offer rsa-sha2-512 and rsa-sha2-256 algorithms in internal SSH #17281 missed.)
• Now alpine linux are having to build gitea with a patch that adds the above workaround to fix issues with testing and Fix tests hostkeys #17786 proposes upstreaming that patch - but our testing version of OpenSSH is not affected by this problem as it is too old and this PR fails because it uses the new option which our version does not (!).

What is actually needed

The fundamental issue is not resolvable within Gitea without forking and otherwise reimplementing the ssh server. We will need to address this problem upstream.

There are two major structural issues:

• Signers need to be separately configurable and advertisable apart from the key types that they sign. The commit mentioned above may have degraded the ability of users of the upstream ssh package to disable the ssh-rsa signing algorithm in future and this needs to be addressed.
• The Handshake needs to support extension negotiation https://datatracker.ietf.org/doc/html/rfc8308 and in particular the SSH_MSG_EXT_INFO extension described therein.

[suchAdemon]

Another workaround, very ugly, is to modify ssh key files which are used by gitea it self.
By replacing the file content with e.g. ed25519 keys, just make sure that the file name stay the same (gitea.rsa.pub and gitea.rsa), gitea will accept the files and offer the new keys to the clients (but all clients have then to accept the new ones which can be a pain, depending on the amount of users)
Still looking forward to the patch

[zeripath]

@suchAdemon I think you're talking about the HostKey signing algorithm - not the client public key. Host Keys have been fixed by #17281 and its backport #17376 which is in 1.15.5.

This issue is specifically talking about the client's public key and the signing algorithm chosen during its authentication.

Belive I just hit this issue on gitea.com just now. Ended up switching to ed25519 from RSA and the issue was resolved.

With RSA what I saw trying to clone via SSH:

git@gitea.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Debugged with:

ssh -vT git@gitea.com

Everything looked fine. But then I remembered ed25519 was an option looking at the SSH key input placeholder text in the Gitea key input field, switched as I prefer it anyway, and was able to use SSH again.

[trading-peter]

A easy and transparent workaround:

Set "HostkeyAlgorithms" and "PubkeyAcceptedKeyTypes" in ~/.ssh/config

Host <Your Git Name>
  HostkeyAlgorithms +ssh-rsa
  PubkeyAcceptedKeyTypes +ssh-rsa

[6543]

#17786 :D