所以我最近遇到了一个奇怪的问题,我的 jenkins 构建每次构建都会无缘无故地失败。构建很简单,它运行 ansible playbook,它会依次将代码部署到远程机器、清除缓存、运行维护等。
构建基本上运行这个:
ansible-playbook -i ansible/hosts ansible/backend.yml
然后,当构建号为奇数时,此步骤将失败(以 force=yes 运行,因此本地修改应该没有问题):
TASK [backend : Clone backend repo] ********************************************
task path: /opt/provision/ansible/roles/backend/tasks/main.yml:5
Using module file /usr/lib/python2.6/site-packages/ansible/modules/core/source_control/git.py
<server_ip_address> ESTABLISH SSH CONNECTION FOR USER: username
<server_ip_address> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r server_ip_address '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499 `" && echo ansible-tmp-1485853855.83-181127458563499="` echo $HOME/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499 `" ) && sleep 0'"'"''
<server_ip_address> PUT /tmp/tmpdgsmCK TO /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/git.py
<server_ip_address> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r '[server_ip_address]'
<server_ip_address> ESTABLISH SSH CONNECTION FOR USER: username
<server_ip_address> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r server_ip_address '/bin/sh -c '"'"'chmod u+x /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/ /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/git.py && sleep 0'"'"''
<server_ip_address> ESTABLISH SSH CONNECTION FOR USER: username
<server_ip_address> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o 'IdentityFile="/var/lib/jenkins/.ssh/jenkins"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=username -o ConnectTimeout=10 -o ConnectionAttempts=20 -o ControlPath=/var/lib/jenkins/.ansible/cp/ansible-ssh-%h-%p-%r -tt server_ip_address '/bin/sh -c '"'"'/usr/bin/python /home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/git.py; rm -rf "/home/username/.ansible/tmp/ansible-tmp-1485853855.83-181127458563499/" > /dev/null 2>&1 && sleep 0'"'"''
fatal: [backend]: FAILED! => {
"changed": false,
"failed": true,
"invocation": {
"module_name": "git"
},
"module_stderr": "OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 56: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 23456\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to server_ip_address closed.\r\n",
"module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 1040, in <module>\r\n main()\r\n File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 994, in main\r\n result.update(changed=True, after=remote_head, msg='Local modifications exist')\r\nUnboundLocalError: local variable 'remote_head' referenced before assignment\r\n",
"msg": "MODULE FAILURE"
}
to retry, use: --limit @/opt/provision/ansible/backend.retry
PLAY RECAP *********************************************************************
backend : ok=2 changed=1 unreachable=0 failed=1
回溯格式:
Traceback (most recent call last):
File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 1040, in <module>
main()
File \"/tmp/ansible_lVW4UB/ansible_module_git.py\", line 994, in main
result.update(changed=True, after=remote_head, msg='Local modifications exist')
UnboundLocalError: local variable 'remote_head' referenced before assignment
失败的步骤:
- name: Clone backend
git: >
repo="{{ backend_repo }}"
dest="{{ backend_path }}"
accept_hostkey=yes
key_file="{{ backend_key }}"
clone=yes
force=yes
update=yes
version={{ backend_branch }}
如果在失败后立即运行构建,它会顺利构建。而且每次我从命令行运行它时它都会构建。