Having spent few hours debugging handshake between WESB and WAS server in the backend, we found out that the problem was with the root CA certificate key size -it was 4K, while WAS policy files restrict it to 1K.
One of the symptoms was that when trying to import certificate generated by our local Enterprise Microsoft CA Server to Websphere Server we’ve got errors described below. Both importing from WAS console and through ikeyman.bat gave errors: “Database is corrupted” and “Could not open keystore“.
Trying to see the content of this PFX file using keytool
\java\jre\bin>keytool -list -v -keystore test.pfx -storetype pkcs12 -storepass test
Also ended up in error “java.io.IOException: Error in loading the keystore: Private key decryption error: (java.security.InvalidKeyException: Illegal key size)
The issue appeared to boil down to a problem with the Java security policy files and thier ‘restricted’ and ‘unrestricted’ flavours and the size of the keys used by our local CA.
Downloading the unrestricted JCE policy files for IBM JVM 1.4.2+ (US_export_policy.jar and local_policy.jar) and replacing them in WAS JVM worked like a charm.
Resolution steps for WAS 6.1:
- Go to the developerWorks Java Technology Security page at URL: Java Technology Security
- Click on the “J2SE 5.0” link for WebSphere Application Server 6.1.xx
- Scroll down on the resulting page and click on the “IBM SDK Policy files” link.
- Select “Unrestricted JCE Policy files for SDK for all newer versions 1.4.2+” and download them
- Extract the local_policy.jar and US_export_policy.jar files from the unrestricted.zip archive.
- Stop the application server
- Back up the local_policy.jar and US_export_policy.jar files located in the following directory: \java\jre\lib\security\
- Place the new files, previously downloaded to \java\jre\lib\security\
- Start the server
Now you will be able to open it with ikeyman and from admin console