Is there a way to pull from an encrypted remote?

Use case:

1. Have annex in an encrypted public rsync remote
2. Have USB stick with PGP keys (but not the annex repository)
3. Get to a new computer
4. Set up a new annex using the PGP keys I have.

1-3 work fine :) However, 4'th is the issue:

How would I do `git pull <remote>` for an encrypted remote? Is it possible?