GitLab stuck MR


MRs (Merge Requests) in GitLab are similar to PRs (Pull Requests) in GitHub, although the process and language around them are slightly different. The name specifically refers the the request to merge into another branch from a branch (or repository) that isn't the same. Simple enough.

Most of the time since moving to GitLab I've been extremely happy with both our productivity and the stability of GitLab, especially considering that we're running it from a docker container in SmartOS. All told, it's been a really good experience.

With that said, it hasn't come without bumps. Mostly these are related to the operating environment, and result in relatively straightforward failures, such as an inability to see memory consumption correctly.

Today, though, I got a bonus problem. This one took some real time and resulted in not only digging through gitlab.com's issues, but dropping into the rails console for gitlab to resolve it.

I'm not sure what actually caused this problem, and it may well be that I did or am doing something problematic. But, the result was that I had a Merge Request (MR) that was in the "merging" state for nearly an hour. Considering most other merges have taken seconds, even with my large codebase for Cartographica, it seemed like something was wrong.

I dug around on gitlab.com looking for some answers and ran across a couple of aging examples of a similar "stuck" MR.

If you find yourself in this position, you may want to look at:

Issue 18048 which details a few different diagnostics and work-arounds.

I used the gitlab console by logging in to the server container and

gitlab-rails console

to get the console running and then:

proj = Project.find_by_full_path('namespace/myproject')
mr = proj.merge_requests.find_by(iid: 123)
mr.state

This prints out the status of the merge request, which in my case was locked.

I did the following:

  1. Verified that the record looked OK by using mr.valid?
  2. Unlocked the MR using mr.unlock_mr, which resulted in true

After that, the state returned to open and I was able to go to the UI and merge it.

Further down there were recommendations for going all the way to the database console using:

# login to prostgres instance
sudo gitlab-rails dbconsole

-- check the list of locked merge requests
select id, iid as merge_id, source_branch, target_branch, locked_at, merge_error, merge_commit_sha, in_progress_merge_commit_sha, merge_status, state from merge_requests where state='locked' order by id desc;

But, I didn't find that necessary.

I'll put here (lest I forget it later) that I also made use of the log tailing mechanism (gitlab-ctl tail) which can also be directed a specific log by adding an argument.