Connecting to SFTP using only a private key

I am trying to connect to an external SFTP. They have our public key and I have the private key which is password protected. I have the credentials set to the sftp username and the password to the private key.

If I set the private key only, I get a username/password error connecting. If I set both the public and private key files, I get the error below:

ERROR #7510: SSH Error '-19': SSH Error [80101013]: Callback returned error [80101013] at Session.cpp:418,0

I have this working perfectly using filezilla, however I can't get it to work with EnsLib.FTP.PassthroughService. The SSL Configuration is !SFTP.

Can anyone help me out?

Thanks.