Covering J2EE Security and WebLogic Topics

Tamper-evident audit logs with AuditHawk

WebLogic 10 Active Directory Authentication Provider Bug

Reader Cobbie Behrend emailed with a bug he noticed in the Active Directory authentication provider in WebLogic 10. He writes:

“The class that handles AD authentication has a small bug in it that causes authentication to fail [at a later time] after someone logs in incorrectly. During authentication the AD provider binds twice using the same LDAP connection, once with the username password being authenticated, and once with the credentials supplied when you configure the LDAP provider. If authentication fails, the second binding doesn’t happen, and the unauthenticated LDAP connection is returned to the internal LDAP connection pool. This poses a problem when later trying to authenticate and the unauthenticated LDAP connection is retrieved from the pool (you get a stack trace from netscape LDAP classes telling you that the connection has not been bound).

Below is the nested stack trace that you get from WebLogic. The really confusing part when you try to figure this one out is that the point of failure changes, as it all depends on when the bogus connection is being used… also if you are using the same AD user for WebLogic configuration of LDAP, and for testing your application (typical bad development behavior), you don’t notice that the connection is bogus when you turn security logging on. So below the failure is at getDNForUser, but I’ve also seen it happen getting the group members of a group (when testing using a different user).”

netscape.ldap.LDAPException: error result (1); 00000000: LdapErr: DSID-0C090627, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, vece
at netscape.ldap.LDAPConnection.checkMsg(Unknown Source)
at netscape.ldap.LDAPConnection.checkSearchMsg(Unknown Source)
at netscape.ldap.LDAPConnection.search(Unknown Source)
at weblogic.security.providers.authentication.LDAPAtnDelegate.getDNForUser(LDAPAtnDelegate.java:3310)
at weblogic.security.providers.authentication.LDAPAtnDelegate.authenticate(LDAPAtnDelegate.java:3180)
at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:200)
at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:769)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:579)
at com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
at $Proxy11.login(Unknown Source)
at weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(Unknown Source)
at com.bea.common.security.internal.service.JAASAuthenticationServiceImpl.authenticate(JAASAuthenticationServiceImpl.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
at $Proxy32.authenticate(Unknown Source)
at weblogic.security.service.PrincipalAuthenticator.authenticate(Unknown Source)
at weblogic.servlet.security.internal.SecurityModule.checkAuthenticate(SecurityModule.java:256)
at weblogic.servlet.security.internal.SecurityModule.checkAuthenticate(SecurityModule.java:205)
at weblogic.servlet.security.internal.FormSecurityModule.processJSecurityCheck(FormSecurityModule.java:245)
at weblogic.servlet.security.internal.FormSecurityModule.checkUserPerm(FormSecurityModule.java:200)
at weblogic.servlet.security.internal.FormSecurityModule.checkAccess(FormSecurityModule.java:91)
at weblogic.servlet.security.internal.ServletSecurityManager.checkAccess(ServletSecurityManager.java:82)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2076)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2046)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1366)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:200)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:172)

Cobbie points out that you don’t even see the stack trace above unless you enable authentication debugging. He says, “In the console you must set environment -> servers -> AdminServer -> Logging -> Advanced -> Severity Level to debug. To turn on security logging you have to change environment -> servers -> AdminServer -> Debug -> WebLogic -> security -> atn -> DebugSecurityAtn to enabled.”

Cobbie contacted Oracle and received a patch but it doesn’t seem to be a “normal” one in that it was not given a unique name like most patches. Instead, Oracle sent him a new cssWlSecurityProviders.jar file via email.

Cobbie notes:

“The patch works, but only if you have “Use Retrieved User Name as Principal” set to false. If a user sets “Use Retrieved User Name as Principal” property to true the connection should be returned to the pool for the retrieved user (something that was also fixed by this patch).

This works great if a correct password is passed in. However if incorrect credentials are passed in and you have “Use Retrieved User Name as Principal” set to true, you have the same issue as before. The unauthenticated connection gets added to the LDAP connection pool, and future attempts to use it fail.

It appears that there still is no check to confirm that the connection is authenticated before the connection is returned to the pool (or from the pool).”

Hopefully, this post will spare others the troubleshooting effort when they encounter this subtle bug.

Thanks Cobbie!

5 Comments

  1. I have a question on weblogic security. I have a custom authentication provider (wls816). In the login module I need to get the datasource and connect the db. I used new Initia lContext();When I start the server, I got error:
    javax.naming.NoInitialContextException: Need to specify class name in environmen
    t or system property, or as an applet parameter, or in an application resource f
    ile: java.naming.factory.initial
    Can’t the security provider access the weblogic context? Thanks

    Comment by sharon — November 8, 2008 @ 8:52 pm

  2. Sharon,

    As I understand it, the security providers are started before JNDI is available. That makes sense since JNDI can have security policies against it but it makes it harder to do what you want to do.

    I suppose you could get around it by catching that exception and trying again until you can get the InitialContext.

    One thing to watch out for is infinite loops since BEA/ORACLE recommends not accessing resources that can have security constraints.

    Good Luck!

    Comment by Mike Fleming — November 9, 2008 @ 8:40 pm

  3. Hi Mike,

    Not sure if this is related to the bug described in the blog – but we are getting prod issues and the server got buried because of stuck threads. I am attaching the stack trace. Any help would be appreciated. Thank you.

    Thread-1533 “[STUCK] ExecuteThread: ‘17′ for queue: ‘weblogic.kernel.Default (self-tuning)’” {
    — Waiting for notification on: netscape.ldap.LDAPResponseListener@ffffffff80fe87c1[fat lock]
    java.lang.Object.wait(Object.java:474)
    netscape.ldap.LDAPMessageQueue.waitForMessage(Unknown Source)
    netscape.ldap.LDAPMessageQueue.waitFirstMessage(Unknown Source)
    netscape.ldap.LDAPConnection.sendRequest(Unknown Source)
    ^– Holding lock: netscape.ldap.LDAPConnection@ffffffff80fe7aad[thin lock]
    netscape.ldap.LDAPConnection.simpleBind(Unknown Source)
    netscape.ldap.LDAPConnection.authenticate(Unknown Source)
    netscape.ldap.LDAPConnection.authenticate(Unknown Source)
    netscape.ldap.LDAPConnection.bind(Unknown Source)
    weblogic.security.providers.authentication.LDAPAtnDelegate$LDAPFactory.newInstance(LDAPAtnDelegate.java:3687)
    weblogic.security.utils.Pool.newInstance(Pool.java:37)
    weblogic.security.utils.Pool.getInstance(Pool.java:29)
    weblogic.security.providers.authentication.LDAPAtnDelegate.getConnection(LDAPAtnDelegate.java:3142)
    weblogic.security.providers.authentication.LDAPAtnDelegate.userExists(LDAPAtnDelegate.java:2260)
    weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:136)
    com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
    com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:101)
    sun.reflect.GeneratedMethodAccessor9101.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:570)
    javax.security.auth.login.LoginContext.invoke(LoginContext.java:720)
    javax.security.auth.login.LoginContext.access$000(LoginContext.java:186)
    javax.security.auth.login.LoginContext$4.run(LoginContext.java:683)
    javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    javax.security.auth.login.LoginContext.login(LoginContext.java:566)
    com.bea.common.security.internal.service.JAASLoginServiceImpl.login(JAASLoginServiceImpl.java:71)
    sun.reflect.GeneratedMethodAccessor1066.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:570)
    com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
    $Proxy11.login(Unknown Source)
    weblogic.security.service.internal.WLSJAASLoginServiceImpl$ServiceImpl.login(Unknown Source)
    com.bea.common.security.internal.service.IdentityAssertionCallbackServiceImpl.assertIdentity(IdentityAssertionCallbackServiceImpl.java:96)
    sun.reflect.GeneratedMethodAccessor702.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:570)
    com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
    $Proxy14.assertIdentity(Unknown Source)
    com.bea.common.security.internal.service.IdentityAssertionServiceImpl.assertIdentity(IdentityAssertionServiceImpl.java:78)
    sun.reflect.GeneratedMethodAccessor699.invoke(Unknown Source)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:570)
    com.bea.common.security.internal.utils.Delegator$ProxyInvocationHandler.invoke(Delegator.java:57)
    $Proxy35.assertIdentity(Unknown Source)
    weblogic.security.service.PrincipalAuthenticator.assertIdentity(Unknown Source)
    weblogic.servlet.security.internal.CertSecurityModule.assertIdentity(CertSecurityModule.java:103)
    weblogic.servlet.security.internal.CertSecurityModule.checkUserPerm(CertSecurityModule.java:65)
    weblogic.servlet.security.internal.SecurityModule.checkAccess(SecurityModule.java:107)
    weblogic.servlet.security.internal.ServletSecurityManager.checkAccess(ServletSecurityManager.java:57)
    weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2076)
    weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:1998)
    weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1331)
    weblogic.work.ExecuteThread.execute(ExecuteThread.java:197)
    weblogic.work.ExecuteThread.run(ExecuteThread.java:164)

    Comment by Kris — March 18, 2009 @ 9:37 am

  4. Kris,

    Sorry for the response delay — I just got back from vacation. Unfortunately, I don’t have a solution for the problem you’re seeing. Sorry!

    Comment by Mike Fleming — March 25, 2009 @ 7:39 pm

  5. Hi Kris, Mike,

    I am facing the same problem. (Waiting for notification on: netscape.ldap.LDAPSearchListener@2899caa[fat lock])

    It will be really helpful if you could give any inputs on solving this.

    Many Thanks,
    Sen

    Comment by Sen — March 4, 2010 @ 5:41 am

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

(NOTE: Comments are moderated and won't show up right away.)

 

Bookmark this page on del.icio.us