authorized_keys files appear to need to be readable by the user logging in.