X509 certificate passphrase won't work in kleopatra and kmail

I've been assigned an X509 certificate, I exported it from firefox in p12 
format and was able to install it in kleopatra using a workaround as detailed 
here:
https://bugzilla.redhat.com/show_bug.cgi?id=548528
unsetting $GPG_AGENT_INFO and launching kleopatra from the un-set shell.

Now I face another issue.
Kleopatra asks me to use a passphrase to protect my key but no matter if I set 
it to "abc", a complex passphrase or an empty string when I try to access the 
key later, via kmail to sign or via kleopatra to change the passphrase, the 
message I get is "bad passphrase".
I'm positive that I've entered the correct passphrase. I have tried to launch 
kleopatra from an un-set shell and in the regular way. I have tried with 
pinentry-qt4 and pinentry-gtk-2 just in case.

Help is very appreciated, unfortunately this certificate is needed at work :/