Multiple github accounts with multiple SSH keys resulting in multiple contributors

I have 2 different Github accounts. And I've setup 2 different SSH keys. Registered both keys on Github with the correct account and created a git config file that should use the right credentials for working with the remote repository (ie use my personal account for personal work and work account for remote work). This is what my git config file looks like:

Host personal.github.com
HostName github.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_rsa_personal
IdentitiesOnly yes

Host work.github.com
HostName github.com
PreferredAuthentications publickey
User git
IdentityFile ~/.ssh/id_rsa_work
IdentitiesOnly yes

It works, I can clone and push projects.

The problem:

Let's say i create a new remote repo in my work account. When I clone the project on my local machine I use: git clone git@work.github.com:[username]/my-plan.git

Then make some changes, commit them and push it back to the remote repo. It works, but instead of showing my work account credentials under "Contributors" it actually shows my personal account.

If I run git remote -v I get:

origin  git@work.github.com:[username]/my-plan.git (fetch)
origin  git@work.github.com:[username]/my-plan.git (push)

I dont understand why is this happening when the associated SSH key is connected to my work account and not my personal account. What I need to do to make sure that every time i push the project back, its from my work account and not personal one?