Wouldn't the private key stay in your phone and you'd be exchanging a challenge and a response?

Not necessarily. I can’t imagine they’d want you to login to your iCloud or Google account on a public computer. It will probably work how Microsoft “Authenticator” works or how when you try logging in to iCloud or your Google account when you have 2FA turned on:

1. Type in your username and click submit on the library computer
2. The service on the computer tells you to look at your phone
3. In the background, the service sent an encrypted challenge to your iCloud account
4. All your devices receives a notification asking if that’s you trying to login
5. You pull out your phone, click yes
6. In the background, your phone decrypts the challenge and sends it back to the server
7. The server verifies its you who is trying to login and logs you in on the library computer

No sharing of keys necessary

Edit: that was just a guess and there are likely a few ways logging in can be achieved on a public computer without needing the private key on that computer. My knowledge on passkeys is surface level, I haven’t really taken the time to look deeply into them yet