How to change the ssh key while doing a "git pull" on a server with multiple users?

I am using an ec2 server in which there are two users. Let's call them user1 and user2 with their separate user1key and user2key in the ~/.ssh folder.

My git config file in the ~/.ssh folder looks like this

Host user1    HostName github.com    IdentityFile ~/.ssh/user1keyHost user2    HostName github.com    IdentityFile ~/.ssh/user2key

I want to use user1 to pull and commit changes to the origin. However, when I run git pull, it asks for the passphrase of user2key which I don't have.

I tried using git config user.name "user1" inside the repository and then did a git pull but it again asked for a passphrase for user2key.

I even tried removing the information for user2 from the config file and run git pull again but it still asked for the passphrase for user2key.