Error caused by BC on mTLS when hardware providers are being used.

[max-m-s]

I have a HTTP client application running on an IBM mainframe, it is set up to use Crypto Hardware in priority over other providers, but there is a Java package that depends on BC to be included. Upon startup, the Providers list is as follows:

  IBMJCECCA
  IBMJSSE2
  IBMJCE
  IBMJCEPlus
  IBMJGSSProvider
  IBMCertPath
  IBMSASL
  IBMXMLCRYPTO
  IBMXMLEnc
  IBMSPNEGO
  SUN
  BC

As you can see, "BC" is quite a way down the list. However, when the application connects out using HttpsURLConnection to a server requiring Mutual Authentication, BC is getting involved and then failing, as it tried to access hidden components of a hardware encrypted key:

java.lang.UnsupportedOperationException: Hardware error, function getModulus has no meaning in hardware
  at com.ibm.crypto.hdwrCCA.provider.RSAPrivateHWKey.getModulus(RSAPrivateHWKey.java:81)
  at org.bouncycastle.jcajce.provider.asymmetric.rsa.RSAUtil.generatePrivateKeyParameter(Unknown Source)
  at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineInitSign(Unknown Source)
  at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineInitSign(Unknown Source)
  at java.security.SignatureSpi.engineInitSign(SignatureSpi.java:172)
  at java.security.Signature£Delegate.tryOperation(Signature.java:1298)
  at java.security.Signature£Delegate.chooseProvider(Signature.java:1241)
  at java.security.Signature£Delegate.engineInitSign(Signature.java:1369)
  at java.security.Signature.initSign(Signature.java:675)
  at java.security.Signature£1.initSign(Signature.java:156)
  at com.ibm.jsse2.a8.a(a8.java:89)
  at com.ibm.jsse2.s.a(s.java:207)
  ...

If I explicitly remove "BC" from the provider list prior to this call then it is successful. But when I do that, another part of the application breaks.

My questions are:

1. How is BC getting involved when it is so far down the list of providers?
2. Is there a way to stop it, without removing the library from the JVM? I couldn't find any variables I could use to control this.

Thanks.

[max-m-s]

After doing some further debugging. The chosen signature algorithm at this point was "RSAPSS". These providers support that algorithm:

Services supporting "RSAPSS":
  com.ibm.crypto.provider.RSAPSSSignature - RSAPSS - IBMJCE version 1.8
  com.ibm.crypto.plus.provider.RSAPSSSignature - RSAPSS - IBMJCEPlus version 1.8
  org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi$PSSwithRSA - RSASSA-PSS - BC version 1.55

The two IBM ones return:

Exception in thread "main" java.security.InvalidKeyException: No installed provider supports this key: com.ibm.crypto.hdwrCCA.provider.RSAPrivateHWKey
        at java.security.Signature$Delegate.chooseProvider(Signature.java:1267)
        at java.security.Signature$Delegate.engineInitSign(Signature.java:1340)
        at java.security.Signature.initSign(Signature.java:627)
        ...

Which is fine for the HTTPS Client without BC installed, as it moves onto "SHA2withRSA". My workaround is to disable "RSAPSS" in java.security for now. Maybe BC should be returning the same "not supported" exception rather than giving it a go. Though as I wrote that, I can see that might be difficult since the key claims to implement interfaces such as java.security.interfaces.RSAPrivateKey.