Wind River Support Network

Description

Problem Description
======================
Caught 'connection to localhost.localdomain:54321 refused'  or 'Failed to initialize storage' while create gluster volume.

# vdsClient -s 0 glusterVolumeCreate volumeName=gv0 bricks=128.224.165.233:/export/brick1,128.224.165.233:/export/brick2


Expected Behavior
======================
create gluster volume success every time.

Observed Behavior
======================
None

Logs
======================
vdsm.log
--------------------- 
MainThread::WARNING::2013-08-01 01:43:08,145::fileUtils::185::fileUtils::(createdir) Dir /windriver/data-center/mnt already exists
MainThread::ERROR::2013-08-01 01:43:08,182::clientIF::248::vds::(_initIRS) Error initializing IRS
Traceback (most recent call last):
  File "/usr/share/vdsm/clientIF.py", line 246, in _initIRS
    self.irs = Dispatcher(HSM())
  File "/usr/share/vdsm/storage/hsm.py", line 348, in __init__
    if not multipath.isEnabled():
  File "/usr/share/vdsm/storage/multipath.py", line 87, in isEnabled
    mpathconf = svdsm.readMultipathConf()
  File "/usr/share/vdsm/supervdsm.py", line 77, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 67, in <lambda>
    getattr(self._supervdsmProxy._svdsm, self._funcName)(*args,
AttributeError: 'ProxyCaller' object has no attribute 'readMultipathConf'
MainThread::INFO::2013-08-01 01:43:08,220::momIF::36::MOM::(__init__) Starting up MOM
MainThread::INFO::2013-08-01 01:43:08,223::vmChannels::140::vds::(settimeout) Setting channels' timeout to 30 seconds.
VM Channels Listener::INFO::2013-08-01 01:43:08,266::vmChannels::128::vds::(run) Starting VM channels listener thread.
MainThread::ERROR::2013-08-01 01:43:08,265::vdsm::90::vds::(run) Exception raised
Traceback (most recent call last):
  File "/usr/share/vdsm/vdsm", line 88, in run
    serve_clients(log)
  File "/usr/share/vdsm/vdsm", line 55, in serve_clients
    cif = clientIF.getInstance(log)
  File "/usr/share/vdsm/clientIF.py", line 126, in getInstance
    cls._instance = clientIF(log)
  File "/usr/share/vdsm/clientIF.py", line 116, in __init__
    self._prepareBindings()
  File "/usr/share/vdsm/clientIF.py", line 166, in _prepareBindings
    self._loadBindingXMLRPC()
  File "/usr/share/vdsm/clientIF.py", line 151, in _loadBindingXMLRPC
    default_bridge)
  File "/usr/share/vdsm/BindingXMLRPC.py", line 55, in __init__
    self.server = self._createXMLRPCServer()
  File "/usr/share/vdsm/BindingXMLRPC.py", line 146, in _createXMLRPCServer
    requestHandler=LoggingHandler)
  File "/usr/lib64/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 137, in __init__
    self.server_bind()
  File "/usr/lib64/python2.7/SocketServer.py", line 419, in server_bind
    self.socket.bind(self.server_address)
  File "/usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 83, in bind
    self.socket.bind(addr)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 98] Address already in use
MainThread::INFO::2013-08-01 01:43:08,306::vdsm::93::vds::(run) VDSM main thread ended. Waiting for 17 other threads...

Workaround

Use 'service vdsmd stop; sleep 10; service vdsmd start' rather than 'service vdsmd restart'.  Or instead of sleeping, use 'lsof -i TCP:54321' to wait for the listening socket to become available after 'service vdsmd stop'.

Steps to Reproduce

1) /data/fast_prod/wrlinuxovp-501/lv20_13sp/wrlinux-5/wrlinux/configure --enable-jobs=32 --enable-parallel-pkgbuilds=16 --enable-kernel=preempt-rt --enable-addons=wr-ovp --enable-rootfs=ovp-ovirt-node --enable-board=intel_xeon_core --with-rcpl-version=0005
2) Deploy images
3) retart vdsmd service
    retart glusterd service
    # vdsClient -s 0 glusterVolumeCreate volumeName=gv0 bricks=128.224.165.233:/export/brick1,128.224.165.233:/export/brick2,128.224.165.233:/export/brick3,128.224.165.233:/export/brick4

Not every time this issue reproduced, high rate 60% to reproduce it.

Misc Info
======================
1) ssl is enabled.
2) oVirt-engine with ovirt-host-deploy-v2.patch