Long story short, the passphrase is getting annoying especially for my personal project. So, I'm removing it with the command:

ssh-keygen -p -f /path_to_private_key

Usually the private key is located under %USERPROFILE%/.ssh folder in Windows. In my case, it is %USERPROFILE%/.ssh/id_ed25519, so using powershell:

ssh-keygen -p -f $env:USERPROFILE/.ssh/id_ed25519