LDAP login fails, resulting in incorrect user permissions until the next sync

[rconaway]

We are using LDAP for authentication. When a user logs in, an error is logged and the user seems to be logged in correctly, but his permissions are not correct until the next LDAP sync.

We've debugged the code to line 523 in class LDAPAuthProvider. When we take the LDAP arguments (base, filter, attributes) from that point and manually make the LDAP call using Softerra LDAP browser, the manual call works fine.

The production instance in Linux, but I can reproduce on a machine running Windows 7, GitBlit go version 1.6.2.

Our next step will be to try to build from source and reproduce. We'd appreciate any suggestions about what might be wrong or how best to pinpoint the problem.

Rob Conaway

---

The following is the (redacted) log message:

2015-09-18 09:26:09 [INFO ] Running on Windows 7 (6.1)
2015-09-18 09:26:09 [INFO ] Logging initialized @461ms
2015-09-18 09:26:09 [INFO ] Using JCE Standard Encryption Policy files, encryption key lengths will be limited
2015-09-18 09:26:09 [INFO ] Setting up HTTPS transport on port 443
2015-09-18 09:26:09 [INFO ]    certificate alias = localhost
2015-09-18 09:26:09 [INFO ]    keyStorePath   = C:\REDACTED\gitblit-1.6.2\data\serverKeyStore.jks
2015-09-18 09:26:09 [INFO ]    trustStorePath = C:\REDACTED\gitblit-1.6.2\data\serverTrustStore.jks
2015-09-18 09:26:09 [INFO ]    crlPath        = C:\REDACTED\gitblit-1.6.2\data\certs\caRevocationList.crl
2015-09-18 09:26:09 [INFO ] Setting up HTTP transport on port 80
2015-09-18 09:26:09 [INFO ] Shutdown Monitor listening on port 8081
2015-09-18 09:26:09 [INFO ] jetty-9.2.3.v20140905
2015-09-18 09:26:12 [INFO ] NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
2015-09-18 09:26:12 [INFO ] 
  . . .
2015-09-18 09:26:23 [ERROR] Problem Searching LDAP
LDAPSearchException(resultCode=32 (no such object), numEntries=0, numReferences=0, errorMessage='0000208D: NameErr: DSID-0315258B, problem 2001 (NO_OBJECT), data 0, best match of:
  'O=REDACTED,C=US'
', matchedDN='O=REDACTED,C=US')
  at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3091)
  at com.gitblit.auth.LdapAuthProvider.doSearch(LdapAuthProvider.java:523)
  at com.gitblit.auth.LdapAuthProvider.getTeamsFromLdap(LdapAuthProvider.java:456)
  at com.gitblit.auth.LdapAuthProvider.authenticate(LdapAuthProvider.java:340)
  at com.gitblit.manager.AuthenticationManager.authenticate(AuthenticationManager.java:380)
  at com.gitblit.wicket.pages.RootPage$LoginForm$1.onSubmit(RootPage.java:567)
  at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1595)
  at org.apache.wicket.markup.html.form.Form.process(Form.java:960)
  at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:922)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.lang.reflect.Method.invoke(Unknown Source)
  at org.apache.wicket.RequestListenerInterface.invoke(RequestListenerInterface.java:182)
  . . .

---

Here is the LDAP portion of the gitblit.properties

realm.authenticationProviders = ldap
realm.ldap.server = ldaps://REDACTED.com:636
realm.ldap.username = REDACTED=2031441, ou=Employee, ou=People, o=REDACTED, c=US
realm.ldap.password = gv3$0819
realm.ldap.maintainTeams = true
realm.ldap.accountBase = ou=Employee, ou=People, o=REDACTED, c=US
realm.ldap.accountPattern = (&(objectClass=person)(uid=rconawa2)(|(memberOf=CN=REDACTED-DEVELOPERS,OU=Groups,O=REDACTED,C=US)(memberOf=CN=REDACTED-DEVELOPERS,OU=Groups,O=REDACTED,C=US)))
realm.ldap.groupBase = OU=Groups,O=REDACTED,C=US
realm.ldap.groupMemberPattern = (&(objectClass=group)(member=REDACTEDGID=1988870,OU=Employee,OU=People,O=REDACTED,C=US))
realm.ldap.groupEmptyMemberPattern = (&(objectClass=group)(!(member=*)))
realm.ldap.admins = (CN=REDACTED-GIT-ADMINISTRATORS,OU=Groups,O=REDACTED,C=US)
realm.ldap.displayName = REDACTEDDisplayName 
realm.ldap.email = mail
realm.ldap.synchronize = true
realm.ldap.syncPeriod = 5 MINUTES
realm.ldap.removeDeletedUsers = true

[gitblit]

Hi Rob,

Thanks for the report. If you can build from source then I would say that directly debugging the authentication code is the way to go.

Another perhaps useful option would be to enable UnboundID (LDAP library) logging. If you startup Gitblit with some -D VM args then you should be able to enable this:

-Dcom.unboundid.ldap.sdk.debug.enabled=true
-Dcom.unboundid.ldap.sdk.debug.level=INFO

You'll have to play with the log levels.
"ALL", "SEVERE", "WARNING", "INFO", "CONFIG", "FINE", "FINER", "FINEST", or "OFF".

[rconaway]

After several days of off and on debugging, we've discovered that we can
"fix" the problem by using fresh LDAP connections when searching for the
teams during authentication. Specifically, we changed
LdapAuthProvider.java as follows:

private void getTeamsFromLdap(LDAPConnection ldapConnection, String
simpleUsername, SearchResultEntry loggingInUser, UserModel user) {
ldapConnection = getLdapConnection();
if (ldapConnection != null) {
try {
String loggingInUserDN = loggingInUser.getDN();
...
}
} finally {
ldapConnection.close();
}
}
}

Before this change, sync() worked fine, but authenticate() failed. After
the change, they both seem to work fine.

Obviously, there is something going on with the state of the LDAP
connection that I cannot see.

Any insights? Is there still some way to work around this without a code
change?

BTW, we've beat our heads on this problem for a while because GitBlit
really solves a problem for us and we'd love to make it work. Thanks for
the time you've put into it.

Rob

On Fri, Sep 18, 2015, 12:54 PM James Moger notifications@github.com wrote:

Hi Rob,

Thanks for the report. If you can build from source then I would say that
directly debugging the authentication code is the way to go.

Another perhaps useful option would be to enable UnboundID (LDAP library)
logging. If you startup Gitblit with some -D VM args then you should be
able to enable this:

-Dcom.unboundid.ldap.sdk.debug.enabled=true
-Dcom.unboundid.ldap.sdk.debug.level=INFO

You'll have to play with the log levels.
"ALL", "SEVERE", "WARNING", "INFO", "CONFIG", "FINE", "FINER", "FINEST",
or "OFF".

—
Reply to this email directly or view it on GitHub
#920 (comment).

[gitblit]

Sorry Rob, I'm not an LDAP expert. It does seem suspicious that reusing the connection fails. When you get a fresh connection, the connection is either bound to fixed credentials specified in the config or is not bound at all - depends on your config. When you re-use the connection from authenticate, the connection is bound to the logging-in user, I think. I'm sure you already know that but I'm wondering if that detail is significant here.

[rconaway]

Your comment about the rebinding was very helpful. There is a powerful
account initially connecting and doing the authenticate, but the logging-in
user does not have the permissions to list his teams. It seems like this
would be a typical situation, but maybe we're using LDAP differently.

At any rate, I don't see any way to work around without a code change.
We're prepared to make a change in our installation if absolutely
necessary, but are you open to a change in the code base?

I see two ways of approaching it:

1. Save and revert to the original binding at the end of isAuthenticate.
   This seems to be the "correct" behavior, but could break anyone depending
   on the previous behavior.

2. Do the change, but make it conditional on a property that defaults to
   false. This would maintain backward compatibility.

If you agree, is this something I can change then make a pull request?

Rob

On Thu, Sep 24, 2015 at 10:08 PM, James Moger notifications@github.com
wrote:

Sorry Rob, I'm not an LDAP expert. It does seem suspicious that reusing
the connection fails. When you get a fresh connection, the connection is
either bound to fixed credentials specified in the config or is not bound
at all - depends on your config. When you re-use the connection from
authenticate, the connection is bound to the logging-in user, I think. I'm
sure you already know that but I'm wondering if that detail is significant
here.

—
Reply to this email directly or view it on GitHub
#920 (comment).

[paulsputer]

Hi Rob, I'm not that familiar with LDAP but have successfully configured GitBlit via an LDAP service with 20k+ users. Whilst setting that up I found the Active Directory Explorer tool very useful for debugging (https://technet.microsoft.com/en-us/library/bb963907.aspx).

A few things I notice with your config though I'm not sure how much is due to different LDAP structures:

• Your username config defines an ad path, I didn't have any success with this style of config but did have when I define it as ad\\{service-user-account-name}
• Your accountBase config is specifying down to the OU level, I ran into difficulties when doing this so just specified to DC level. Apparently this may slow searches but I haven't noticed any difference.
• Your accountPattern contains an actual username, I use %{username} also I did have fun identifying the id to use, I found that uid wasn't always correct, so use sAMAccountName instead.
• Your groupBase and accountBase paths differ, I found I had to have the higher level parts of the path matching.
• Your groupMemberPattern contains an actual username for member, I had to use ${dn}
• Also, you may need to check what your realm.ldap.uid is set to and if it matches above

As I don't have admin access to LDAP to manage teams I simply use OU's which thankfully are granular enough for my requirements. It does sound a bit odd for a user not to be able to list the teams it is a member of, could it be perhaps the user doesn't have access to the particular AD path defined such as groupBase. If you log in using the user (not service) credentials to AD explorer maybe you will find a different path to use which works for all accounts.

HTH

Paul

[RainerW]

Hi, i think this is the same issue as #247

[gitblit]

@RainerW Yes, I think it is the same (or at least very similar) issue.

@rconaway Did Paul's comments move you further along?

[rconaway]

There was a pull request for #247 that would have solved this problem. The pull request was refused then #247 was closed. Is there anything else going on? Should I make a pull request?

Rob

[gitblit]

Hi Rob, a PR would be nice. No one is actively working on this issue.

[mstaffeld]

@rconaway 👍

We are experiencing the issue at a large enterprise client where the LDAP user does not have the permission to view LDAP groups. Go figure. Yes I am serious...

The LDAP connection would need to be an elevated user that can query groups.

[kellerkindt]

We are also experiencing the same issue. The normal LDAP users are not allowed to list all LDAP groups while the user from the configuration file is allowed to do so.

Any chance for a fix? Building gitblit on our own just for that does not seem to be a good solution at all.

@gitblit Would you accept a new pull request of #247 for the develop branch?

[gitblit]

@kellerkindt My recollection is the proposed fix solved one problem while creating a new one. The new problem was either backwards-incompatibility for config or unexpected behavior changes or both - can't recall. If you can propose a fix that doesn't break others then I'm all for it.

[flaix]

I, too, have that same problem now. From what I see this has come up a few times now.
I would consider the current behavior as wrong and thus wouldn't object to a change.
Since there is activity on this issue but not so much on the thread in the discussion group, I'm providing a link there so that the conversation can be merged.