cPanel update hanging

I have received below notification from cpanel server “/usr/local/cpanel/scripts/upcp was running as pid ‘11348’ for longer than 6 hours. cPanel will kill this process and run a new upcp in its place”. Then i have just login to the server and saw the cpanel update log. The cpanel update log is located at “/var/cpanel/updatelogs”. Here you can see last 10 updated log files.

1) How to check the updated logs

Login to server via ssh and move to updatelog location, see the output.

[email protected] [/var/cpanel/updatelogs]# ll -h
total 2.0M
drwx------  2 root root 4.0K Dec 27 02:44 ./
drwxr-xr-x 81 root root 4.0K Dec 27 05:26 ../
lrwxrwxrwx  1 root root   44 Dec 27 02:44 last -> /var/cpanel/updatelogs/update.1388112241.log
-rw-r--r--  1 root root  729 Dec 27 02:44 summary.log
-rw-r--r--  1 root root 259K Dec 17 02:50 update.1387248241.log
-rw-r--r--  1 root root 127K Dec 18 02:44 update.1387334642.log
-rw-r--r--  1 root root 437K Dec 19 02:53 update.1387421041.log
-rw-r--r--  1 root root 201K Dec 20 02:45 update.1387507441.log
-rw-r--r--  1 root root 127K Dec 21 08:32 update.1387593841.log
-rw-r--r--  1 root root 195K Dec 23 02:44 update.1387680241.log
-rw-r--r--  1 root root 156K Dec 24 02:44 update.1387766641.log
-rw-r--r--  1 root root  80K Dec 25 02:44 update.1387853041.log
-rw-r--r--  1 root root 156K Dec 26 02:44 update.1387939441.log
-rw-r--r--  1 root root 157K Dec 27 02:44 update.1388025841.log
-rw-r--r--  1 root root 9.0K Dec 27 02:58 update.1388112241.log

The above output is showing last 10 updated log details, Now i’m going to check the last updated log file whether the upcp got completed or not. The last updated log file is “update.1388112241.log”

2) How to open the upcp log

Use your favourite editor to open the last updated log files. I’m going to open the last updated log file is “update.1388112241.log”

[email protected] [/var/cpanel/updatelogs]# nano update.1388112241.log
  GNU nano 2.0.9                                       File: update.1388112241.log

----------------------------------------------------------------------------------------------------
=> Log opened from /usr/local/cpanel/scripts/upcp (1301) at Fri Dec 27 02:44:01 2013
[20131227.024401] W previous PID (11348) has been running more than 6 hours. Killing processes.
[20131227.024403]   Detected cron=1 (cron mode set from command line)
[20131227.024403]   1% complete
[20131227.024403]   Running Standardized hooks
[20131227.024403]   2% complete
[20131227.024403]   mtime on upcp is 1387421308 (Thu Dec 19 02:48:28 2013)
----------------------------------------------------------------------------------------------------
=> Log opened from /usr/local/cpanel/scripts/updatenow (1328) at Fri Dec 27 02:44:03 2013
[20131227.024403]   Detected version '11.40.1.8' from version file.
.
.
.
[20131227.024417]   Processing: Validating cPanel system users
[20131227.024417]    - Processing command `/usr/local/cpanel/scripts/checkusers`
[20131227.024417]      [2802] Homedir: /var/cpanel/userhomes/cpanel
[20131227.024417]   32% complete
[20131227.024417]   Processing: Making sure /var/cpanel/sysinfo.conf is present
[20131227.024417]    - Processing command `/usr/local/cpanel/scripts/gensysinfo`
[20131227.024417]   Processing: Checking and repairing nameserver config
[20131227.024417]    - Processing command `/usr/local/cpanel/scripts/fixrndc`
[20131227.025747]      [2807] Named could not be restarted, any obvious config errors should show up below this line.
[20131227.025747]      [2807] No critical problems found, will attempt to regenerate keys regardless.
[20131227.025850]      [2807] Creating rndc.conf
[20131227.025850]      [2807] Creating /etc/rndc.key
[20131227.025850]      [2807] Restarting named

----------------------------------------------------------------------------------------------------
=> Log opened from cPanel Update (upcp) - Slave (11348) at Fri Dec 27 02:44:01 2013
[20131227.024401] E User hit ^C or killed the process ( pid file '/var/run/upcp.pid' removed ).
=> Log closed Fri Dec 27 02:44:01 2013

The above output is showing the upcp is not completed and got disconnected due to stalling on the “Restarting named step”.

3) What action taken to identified the issue ?

In tailing the update log, I found that it was stalling on the “Restarting named step”. So the final option is need to be run the upcp manually. While running upcp manually, I have monitor the below things to identify the issue.

[email protected] [~]# ps aux | grep upcp
root 1301 0.0 0.0 135632 5476 ? Ss 02:44 0:01 cPanel Update (upcp) - Master
root 1326 0.0 0.0 135632 4572 ? Ss 02:44 0:00 cPanel Update (upcp) - Slave
root 7960 0.0 0.0 103252 828 pts/2 S+ 07:22 0:00 grep upcp

I also took note that the process tree for the currently running upcp was stuck on on an rndc call

[email protected] [~]# pstree -A 1301
cPanel\040Update\040(---cPanel\040Update\040(---perl---/usr/local/cpan---rndc---2*[{rndc}]

When I ran /scripts/fixrndc or any other DNS related command, I found that this also stalled. Then i have tried to ping the loopback address 127.0.0.1 and got disconnected, See the output.

[email protected] [~]# ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
^C
--- 127.0.0.1 ping statistics ---
20 packets transmitted, 0 received, 100% packet loss, time 19170ms

I’m never expect lo interface was down but unfortunately lo interface is down which is caused the issue. Now, I’m going to up the lo interface.

[email protected] [~]# ifconfig lo up

Now the loopback interface is up and restored local connectivity to the server.

[email protected] [~]# ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.029 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.028 ms

Here i’m going to run the upcp again and its completed within 10 mins see the log file.

[email protected] [~]# /scripts/upcp –force

4) UPCP completed log details.

See the output, upcp has been completed without any issues.

[email protected] [~]# nano /var/cpanel/updatelogs/update.1388134182.log
[20131227.090102]   Target version set to '11.40.1.8'
[20131227.090102]   Up to date (11.40.1.8)
=> Log closed Fri Dec 27 09:01:02 2013
----------------------------------------------------------------------------------------------------
=> Log opened from cPanel Update (upcp) - Slave (11422) at Fri Dec 27 09:01:02 2013
[20131227.090102]   95% complete
[20131227.090102]   Running Standardized hooks
[20131227.090102]   100% complete
[20131227.090102]
[20131227.090102]       cPanel update completed
[20131227.090102]   A log of this update is available at /var/cpanel/updatelogs/update.1388134182.log
[20131227.090102]   Removing upcp pidfile
[20131227.090102]
[20131227.090102] Completed all updates
=> Log closed Fri Dec 27 09:01:02 2013

Magesh Maruthamuthu

Love to play with all Linux distribution

You may also like...