When increasing the buffer size up to VIR_NET_MESSAGE_MAX, we
currently quadruple it each time.  This unfortunately means that we
cannot allow certain buffer sizes -- for example the current
VIR_NET_MESSAGE_MAX == 33554432 can never be "hit" since ‘newlen’
jumps from 16MB to 64MB.

Instead of quadrupling, double it each time.

Thanks: Daniel Berrange.
Signed-off-by: NRichard W.M. Jones <rjones@redhat.com>

This removes the opencoded payload freeing in the client, to use
the shared virNetMessageClearPayload call. Two changes:

- ClearPayload sets nfds=0, which fixes a potential crash if
  an error path called virNetMessageFree/Clear on the message
  after fds was free'd
- We drop the inner loop VIR_FORCE_CLOSE... this may mean fds are
  kept open a little bit longer if the call is blocking but in
  practice I don't think it will have any effect

Set all counters to 0. This doesn't impact current users, but
future users will want this

Handles freeing the buffer and fds, but not the message details.
Use it to drop some duplicate code.

Any source file which calls the logging APIs now needs
to have a VIR_LOG_INIT("source.name") declaration at
the start of the file. This provides a static variable
of the virLogSource type.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

With Mac OS X 10.9, xdrproc_t is no longer defined as:

typedef bool_t (*xdrproc_t)(XDR *, ...);

but instead as:

typdef bool_t (*xdrproc_t)(XDR *, void *, unsigned int);

For reference, Linux systems typically define it as:

typedef bool_t (*xdrproc_t)(XDR *, void *, ...);

The rationale explained in the header is that using a vararg is
incorrect and has a potential to change the ABI slightly do to compiler
optimizations taken and the undefined behavior. They decided
to specify the exact number of parameters and for compatibility with old
code decided to make the signature require 3 arguments. The third
argument is ignored for cases that its not used and its recommended to
supply a 0.

<...>
/* Size of message length field. Not counted in VIR_NET_MESSAGE_MAX
 * and VIR_NET_MESSAGE_INITIAL.
 */
const VIR_NET_MESSAGE_LEN_MAX = 4;
</...>

However, msg->bufferLength includes the length word. The wrong checking
was introduced by commit e914dcfd.

* src/rpc/virnetmessage.c:
  - Correct the checking in virNetMessageEncodePayloadRaw
  - Use a new variable to track the new payload length in
    virNetMessageEncodePayloadRaw

Previously, we were freeing verr fields instead of rerr which we've
allocated just a line above.

We have seen an issue on s390x platform where domain XMLs larger than 1MB
were used. The define command was finished successfully. The dumpxml command
was not successful (i.e. could not encode message payload).

Enlarged message related sizes (e.g. maximum string size, message size, etc.)
to handle larger system configurations used on s390x platform.

To improve handling of the RPC message size the allocation during encode process
is changed to a dynamic one (i.e. starting with 64kB initial size and increasing
that size in steps up to 16MB if the payload data is larger).
Signed-off-by: NDaniel Hansel <daniel.hansel@linux.vnet.ibm.com>
Signed-off-by: NViktor Mihajlovski <mihajlov@linux.vnet.ibm.com>

A number of bugs handling file descriptors received from the
server caused the FDs to be lost and leaked.
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

On OOM, xdr_destroy got called even though it wasn't created yet.

Found by coverity:
Error: UNINIT (CWE-457):
    libvirt-0.10.2/src/rpc/virnetmessage.c:214: var_decl: Declaring
    variable "xdr" without initializer.
    libvirt-0.10.2/src/rpc/virnetmessage.c:219: cond_true: Condition
    "virReallocN(&msg->buffer, 1UL /* sizeof (*msg->buffer) */,
    msg->bufferLength) < 0", taking true branch
    libvirt-0.10.2/src/rpc/virnetmessage.c:221: goto: Jumping to label
    "cleanup"
    libvirt-0.10.2/src/rpc/virnetmessage.c:257: label: Reached label
    "cleanup"
    libvirt-0.10.2/src/rpc/virnetmessage.c:258: uninit_use: Using
    uninitialized value "xdr.x_ops".

https://www.gnu.org/licenses/gpl-howto.html recommends that
the 'If not, see <url>.' phrase be a separate sentence.

* tests/securityselinuxhelper.c: Remove doubled line.
* tests/securityselinuxtest.c: Likewise.
* globally: s/;  If/.  If/

Per the FSF address could be changed from time to time, and GNU
recommends the following now: (http://www.gnu.org/licenses/gpl-howto.html)

  You should have received a copy of the GNU General Public License
  along with Foobar.  If not, see <http://www.gnu.org/licenses/>.

This patch removes the explicit FSF address, and uses above instead
(of course, with inserting 'Lesser' before 'General').

Except a bunch of files for security driver, all others are changed
automatically, the copyright for securify files are not complete,
that's why to do it manually:

  src/security/security_selinux.h
  src/security/security_driver.h
  src/security/security_selinux.c
  src/security/security_apparmor.h
  src/security/security_apparmor.c
  src/security/security_driver.c

This rmoves all the per-file error reporting macros
from the code in src/rpc/
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

My latest patch for RPC rework (a2c304f6) introduced a memory leak.
virNetMessageEncodeHeader() is calling VIR_ALLOC_N(msg->buffer, ...)
despite fact, that msg->buffer isn't VIR_FREE()'d on all paths calling
the function. Therefore, rather than injecting free statement switch to
VIR_REALLOC_N().

Currently, we are allocating buffer for RPC messages statically.
This is not such pain when RPC limits are small. However, if we want
ever to increase those limits, we need to allocate buffer dynamically,
based on RPC message len (= the first 4 bytes). Therefore we will
decrease our mem usage in most cases and still be flexible enough in
corner cases.

Our HACKING discourages use of malloc and free, for at least
a couple of years now.  But we weren't enforcing it, until now :)

For now, I've exempted python and tests, and will clean those up
in subsequent patches.  Examples should be permanently exempt,
since anyone copying our examples won't have use of our
internal-only memory.h via libvirt_util.la.

* cfg.mk (sc_prohibit_raw_allocation): New rule.
(exclude_file_name_regexp--sc_prohibit_raw_allocation): and
exemptions.
* src/cpu/cpu.c (cpuDataFree): Avoid false positive.
* src/conf/network_conf.c (virNetworkDNSSrvDefParseXML): Fix
offenders.
* src/libxl/libxl_conf.c (libxlMakeDomBuildInfo, libxlMakeVfb)
(libxlMakeDeviceModelInfo): Likewise.
* src/rpc/virnetmessage.c (virNetMessageSaveError): Likewise.
* tools/virsh.c (_vshMalloc, _vshCalloc): Likewise.

Define two new RPC message types VIR_NET_CALL_WITH_FDS and
VIR_NET_REPLY_WITH_FDS. These message types are equivalent
to VIR_NET_CALL and VIR_NET_REPLY, except that between the
message header, and payload there is a 32-bit integer field
specifying how many file descriptors have been passed.

The actual file descriptors are sent/recv'd out of band.

* src/rpc/virnetmessage.c, src/rpc/virnetmessage.h,
  src/libvirt_private.syms: Add support for handling
  passed file descriptors
* src/rpc/virnetprotocol.x: Extend protocol for FD
  passing

Commit 2c85644b attempted to
fix a problem with tracking RPC messages from streams by doing

-            if (msg->header.type == VIR_NET_REPLY) {
+            if (msg->header.type == VIR_NET_REPLY ||
+                (msg->header.type == VIR_NET_STREAM &&
+                 msg->header.status != VIR_NET_CONTINUE)) {
                 client->nrequests--;

In other words any stream packet, with status NET_OK or NET_ERROR
would cause nrequests to be decremented. This is great if the
packet from from a synchronous virStreamFinish or virStreamAbort
API call, but wildly wrong if from a server initiated abort.
The latter resulted in 'nrequests' being decremented below zero.
This then causes all I/O for that client to be stopped.

Instead of trying to infer whether we need to decrement the
nrequests field, from the message type/status, introduce an
explicit 'bool tracked' field to mark whether the virNetMessagePtr
object is subject to tracking.

Also add a virNetMessageClear function to allow a message
contents to be cleared out, without adversely impacting the
'tracked' field as a naive memset() would do

* src/rpc/virnetmessage.c, src/rpc/virnetmessage.h: Add
  a 'bool tracked' field and virNetMessageClear() API
* daemon/remote.c, daemon/stream.c, src/rpc/virnetclientprogram.c,
  src/rpc/virnetclientstream.c, src/rpc/virnetserverclient.c,
  src/rpc/virnetserverprogram.c: Switch over to use
  virNetMessageClear() and pass in the 'bool tracked' value
  when creating messages.

Improve log messages issued when encountering a bogus
message length to include the actual length and the
limit violated

* src/rpc/virnetmessage.c: Improve log messages

On stream completion it is neccessary to send back a
message with an empty payload. The message header was
not being filled out correctly, since we were not writing
any payload. Add a method for encoding an empty payload
which updates the message headers correctly.

* src/rpc/virnetmessage.c, src/rpc/virnetmessage.h: Add
  a virNetMessageEncodePayloadEmpty method
* src/rpc/virnetserverprogram.c: Write empty payload on
  stream completion

This guts the libvirtd daemon, removing all its networking and
RPC handling code. Instead it calls out to the new virServerPtr
APIs for all its RPC & networking work

As a fallout all libvirtd daemon error reporting now takes place
via the normal internal error reporting APIs. There is no need
to call separate error reporting APIs in RPC code, nor should
code use VIR_WARN/VIR_ERROR for reporting fatal problems anymore.

* daemon/qemu_dispatch_*.h, daemon/remote_dispatch_*.h: Remove
  old generated dispatcher code
* daemon/qemu_dispatch.h, daemon/remote_dispatch.h: New dispatch
  code
* daemon/dispatch.c, daemon/dispatch.h: Remove obsoleted code
* daemon/remote.c, daemon/remote.h: Rewrite for new dispatch
  APIs
* daemon/libvirtd.c, daemon/libvirtd.h: Remove all networking
  code
* daemon/stream.c, daemon/stream.h: Update for new APIs
* daemon/Makefile.am: Link to libvirt-net-rpc-server.la

This provides a new struct that contains a buffer for the RPC
message header+payload, as well as a decoded copy of the message
header. There is an API for applying a XDR encoding & decoding
of the message headers and payloads. There are also APIs for
maintaining a simple FIFO queue of message instances.

Expected usage scenarios are:

To send a message

   msg = virNetMessageNew()

   ...fill in msg->header fields..
   virNetMessageEncodeHeader(msg)
   ...loook at msg->header fields to determine payload filter
   virNetMessageEncodePayload(msg, xdrfilter, data)
   ...send msg->bufferLength worth of data from buffer

To receive a message

   msg = virNetMessageNew()
   ...read VIR_NET_MESSAGE_LEN_MAX of data into buffer
   virNetMessageDecodeLength(msg)
   ...read msg->bufferLength-msg->bufferOffset of data into buffer
   virNetMessageDecodeHeader(msg)
   ...look at msg->header fields to determine payload filter
   virNetMessageDecodePayload(msg, xdrfilter, data)
   ...run payload processor

* src/Makefile.am: Add to libvirt-net-rpc.la
* src/rpc/virnetmessage.c, src/rpc/virnetmessage.h: Internal
  message handling API.
* testutils.c, testutils.h: Helper for printing binary differences
* virnetmessagetest.c: Validate all XDR encoding/decoding