[jruby-user] PKCS_OAEP_PADDING issue

[Derry O' Sullivan]

Hi,

I'm using OAEP padding as part of some decryption/encryption steps -
specifically using: /ECB/OAEPWithSHA1AndMGF1Padding

Testing encryption/decryption via standard ruby 1.9.3 and openssl 0.9.8e
works as expected. We don't explicitly name the above padding - we pass
OpenSSL::PKey::RSA::PKCS1_OAEP_PADDING (padding parameter 4) to openssl's
RSA decryption methods and it works correctly.

When repeating the same test via JRuby 1.7.11 + 1.9.3, we saw padding
failures and after some debugging, linked it to the fact that the default
JRuby padding associated with padding parameter 4 (
https://github.com/jruby/jruby/blob/master/ext/openssl/src/main/java/org/jruby/ext/openssl/PKeyRSA.java#L443
)
is actually the same algorithm but with MD5 vs SHA1 as the digest algorithm.

Modifying that file and changing MD5->SHA1 fixed our issues. Is there a
reason why this is the case? I'm trying to understand why the behaviour is
different between standard ruby and JRuby.

Thanks in advance,

Derry