This patch extends the NWFilter driver for Linux (ebiptables) to create
rules for each member of a previously introduced list. If for example
an attribute value (internally) looks like this:

IP = [10.0.0.1, 10.0.0.2, 10.0.0.3]

then 3 rules will be generated for a rule accessing the variable 'IP',
one for each member of the list. The effect of this is that this now
allows for filtering for multiple values in one field. This can then be
used to support for filtering/allowing of multiple IP addresses per
interface.

An iterator is introduced that extracts each member of a list and
puts it into a hash table which then is passed to the function creating
a rule. For the above example the iterator would cause 3 loops.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

NWFilters can be provided name-value pairs using the following
XML notation:

      <filterref filter='xyz'>
        <parameter name='PORT' value='80'/>
        <parameter name='VAL' value='abc'/>
      </filterref>

The internal representation currently is so that a name is stored as a
string and the value as well. This patch now addresses the value part of it
and introduces a data structure for storing a value either as a simple
value or as an array for later support of lists.

This patch adjusts all code that was handling the values in hash tables
and makes it use the new data type.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

The previous patch extends the priority of filtering rules into negative
numbers. We now use this possibility to interleave the jumping into
chains with filtering rules to for example create the 'root' table of
an interface with the following sequence of rules:

Bridge chain: libvirt-I-vnet0, entries: 6, policy: ACCEPT
-p IPv4 -j I-vnet0-ipv4
-p ARP -j I-vnet0-arp
-p ARP -j ACCEPT 
-p 0x8035 -j I-vnet0-rarp
-p 0x835 -j ACCEPT 
-j DROP 

The '-p ARP -j ACCEPT' rule now appears between the jumps.
Since the 'arp' chain has been assigned priority -700 and the 'rarp'
chain -600, the above ordering can now be achieved with the following
rule:

  <rule action='accept' direction='out' priority='-650'>
    <mac protocolid='arp'/>
  </rule>

This patch now sorts the commands generating the above shown jumps into
chains and interleaves their execution with those for generating rules.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

So far rules' priorities have only been valid in the range [0,1000].
Now I am extending their priority into the range [-1000, 1000] for subsequently
being able to sort rules and the access of (jumps into) chains following
priorities.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

Use the name of the chain rather than its type index (enum).
This pushes the later enablement of chains with user-given names
into the XML parser. For now we still only allow those names that
are well known ('root', 'arp', 'rarp', 'ipv4' and 'ipv6').
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

Use scripts for the renaming and cleaning up of chains. This allows us to get
rid of some of the code that is only capable of renaming and removing chains
whose names are hardcoded.

A shell function 'collect_chains' is introduced that is given the name
of an ebtables chain and then recursively determines the names of all
chains that are accessed from this chain and its sub-chains using 'jumps'.

The resulting list of chain names is then used to delete all the found
chains by first flushing and then deleting them.

The same function is also used for renaming temporary filters to their final
names.

I tested this with the bash and dash as script interpreters.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

Use the previously introduced chain priorities to sort the chains for access
from an interface's 'root' table and have them created in the proper order.
This gets rid of a lot of code that was previously creating the chains in a 
more hardcoded way.

To determine what protocol a filter is used for evaluation do prefix-
matching, i.e., the filter 'arp' is used to filter for the 'arp' protocol,
'ipv4' for the 'ipv4' protocol and 'arp-xyz' will also be used to filter
for the 'arp' protocol following the prefix 'arp' in its name.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

For better handling of the sorting of chains introduce an internally used
priority. Use a lookup table to store the priorities. For now their actual
values do not matter just that the values cause the chains to be properly
sorted through changes in the following patches. However, the values are
chosen as negative so that once they are sorted along with filtering rules
(whose priority may only be positive for now) they will always be instantiated
before them (lower values cause instantiation before higher values). This
is done to maintain backwards compatibility.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

It's not worth even worrying about a temporary file, unless we
ever expect the script to exceed maximum command-line argument
length limits.

* src/nwfilter/nwfilter_ebiptables_driver.c (ebiptablesExecCLI):
Run the commands as an argument to /bin/sh, rather than worrying
about a temporary file.
(ebiptablesWriteToTempFile): Delete unused function.

If /tmp is mounted with the noexec flag (common on security-conscious
systems), then nwfilter will fail to initialize, because we cannot
run any temporary script via virRun("/tmp/script"); but we _can_
use "/bin/sh /tmp/script".  For that matter, using /tmp risks collisions
with other unrelated programs; we already have /var/run/libvirt as a
dedicated temporary directory for use by libvirt.

* src/nwfilter/nwfilter_ebiptables_driver.c
(ebiptablesWriteToTempFile): Use internal directory, not /tmp;
drop attempts to make script executable; and detect close error.
(ebiptablesExecCLI): Switch to virCommand, and invoke the shell to
read the script, rather than requiring an executable script.

The socket address APIs in src/util/network.h either take the
form  virSocketAddrXXX, virSocketXXX or virSocketXXXAddr.

Sanitize this so everything is virSocketAddrXXXX, and ensure
that the virSocketAddr parameter is always the first one.

* src/util/network.c, src/util/network.h: Santize socket
  address API naming
* src/conf/domain_conf.c, src/conf/network_conf.c,
  src/conf/nwfilter_conf.c, src/network/bridge_driver.c,
  src/nwfilter/nwfilter_ebiptables_driver.c,
  src/nwfilter/nwfilter_learnipaddr.c,
  src/qemu/qemu_command.c, src/rpc/virnetsocket.c,
  src/util/dnsmasq.c, src/util/iptables.c,
  src/util/virnetdev.c, src/vbox/vbox_tmpl.c: Update for
  API renaming

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

Seems reasonable to have all command wrappers in the same place

v2:
    Dont move SetInherit

v3:
    Comment spelling fix
    Adjust WARN0 comment
    Remove spurious #include movement
    Don't include sys/types.h
    Combine virExec enums
Signed-off-by: NCole Robinson <crobinso@redhat.com>

This patch enables filtering of gratuitous ARP packets using the following XML:

<rule action='accept' direction='in' priority='425'>
<arp gratuitous='true'/>
</rule>

These VIR_XXXX0 APIs make us confused, use the non-0-suffix APIs instead.

How do these coversions works? The magic is using the gcc extension of ##.
When __VA_ARGS__ is empty, "##" will swallow the "," in "fmt," to
avoid compile error.

example: origin				after CPP
	high_level_api("%d", a_int)	low_level_api("%d", a_int)
	high_level_api("a  string")	low_level_api("a  string")

About 400 conversions.

8 special conversions:
VIR_XXXX0("") -> VIR_XXXX("msg") (avoid empty format) 2 conversions
VIR_XXXX0(string_literal_with_%) -> VIR_XXXX(%->%%) 0 conversions
VIR_XXXX0(non_string_literal) -> VIR_XXXX("%s", non_string_literal)
  (for security) 6 conversions
Signed-off-by: NLai Jiangshan <laijs@cn.fujitsu.com>

We already have virAsprintf, so picking a similar name helps for
seeing a similar purpose.  Furthermore, the prefix V before printf
generally implies 'va_list', even though this variant was '...', and
the old name got in the way of adding a new va_list version.

global rename performed with:

$ git grep -l virBufferVSprintf \
  | xargs -L1 sed -i 's/virBufferVSprintf/virBufferAsprintf/g'

then revert the changes in ChangeLog-old.

Ebtables filtering doesn't work on macvtap device. Remove support for direct type of interface.

gcc 4.6 warns when a variable is initialized but isn't used afterwards:

vmware/vmware_driver.c:449:18: warning: variable 'vmxPath' set but not used [-Wunused-but-set-variable]

This patch fixes these warnings. There are still 2 offending files:

- vbox_tmpl.c: the variable is used inside an #ifdef and is assigned several
  times outside of #ifdef. Fixing the warning would have required wrapping
  all the assignment inside #ifdef which hurts readability.

vbox/vbox_tmpl.c: In function 'vboxAttachDrives':
vbox/vbox_tmpl.c:3918:22: warning: variable 'accessMode' set but not used [-Wunused-but-set-variable]

- esx_vi_types.generated.c: the name implies it's generated code and I
  didn't want to dive into the code generator

esx/esx_vi_types.generated.c: In function 'esxVI_FileQueryFlags_Free':
esx/esx_vi_types.generated.c:1203:3: warning: variable 'item' set but not used [-Wunused-but-set-variable]

This patch adds support for the evaluation of TCP flags in nwfilters.

It adds documentation to the web page and extends the tests as well.
Also, the nwfilter schema is extended.

The following are some example for rules using the tcp flags:

<rule action='accept' direction='in'>
    <tcp state='NONE' flags='SYN/ALL' dsptportstart='80'/>
</rule>
<rule action='drop' direction='in'>
    <tcp state='NONE' flags='SYN/ALL'/>
</rule>

Child processes don't always reach _exit(); if they die from a
signal, then any messages should still be accurate.  Most users
either expect a 0 status (thankfully, if status==0, then
WIFEXITED(status) is true and WEXITSTATUS(status)==0 for all
known platforms) or were filtering on WIFEXITED before printing
a status, but a few were missing this check.  Additionally,
nwfilter_ebiptables_driver was making an assumption that works
on Linux (where WEXITSTATUS shifts and WTERMSIG just masks)
but fails on other platforms (where WEXITSTATUS just masks and
WTERMSIG shifts).

* src/util/command.h (virCommandTranslateStatus): New helper.
* src/libvirt_private.syms (command.h): Export it.
* src/util/command.c (virCommandTranslateStatus): New function.
(virCommandWait): Use it to also diagnose status from signals.
* src/security/security_apparmor.c (load_profile): Likewise.
* src/storage/storage_backend.c
(virStorageBackendQEMUImgBackingFormat): Likewise.
* src/util/util.c (virExecDaemonize, virRunWithHook)
(virFileOperation, virDirCreate): Likewise.
* daemon/remote.c (remoteDispatchAuthPolkit): Likewise.
* src/nwfilter/nwfilter_ebiptables_driver.c (ebiptablesExecCLI):
Likewise.

This patch adds the possibility to not just drop packets, but to also have them rejected where iptables at least sends an ICMP msg back to the originator. On ebtables this again maps into dropping packets since rejecting is not supported.

I am adding 'since 0.8.9' to the docs assuming this will be the next version of libvirt.

This patch reorders the connlimit and comment match extensions relative to the state match (-m state); connlimit being most useful if found after a -m state --state NEW and not before it.

When run non-root the nwfilter driver logs error messages about
being unable to find iptables/ebtables commands (they are in
/sbin which isn't in $PATH). The nwfilter driver can't ever work
as non-root, so simply skip it entirely thus avoiding the error
messages

* src/conf/nwfilter_conf.h, src/nwfilter/nwfilter_driver.c,
  src/nwfilter/nwfilter_gentech_driver.c,
  src/nwfilter/nwfilter_gentech_driver.h: Pass 'bool privileged'
  flag down to final driver impl
* src/nwfilter/nwfilter_ebiptables_driver.c: Skip initialization
  if not privileged

Pure cosmetic change.

VIR_ERR_INVALID_* is meant for invalid pointers only.

Rather than only cleaning any remaining ebtables rules, also clean those applied to iptables and ip6tables when detecting the IP address of an interface. Previous applied iptables rules may hinder DHCP packets.

* src/nwfilter/nwfilter_ebiptables_driver.c (CMD_EXEC): Fix syntax
error in previous patch.
Reported by Stefan Berger.

* src/nwfilter/nwfilter_ebiptables_driver.c
(ebiptablesWriteToTempFile): Use /bin/sh.
(bash_cmd_path): Delete.
(ebiptablesDriverInit, ebiptablesDriverShutdown): No need to
search for bash.
(CMD_EXEC): Prefer $() over ``, since we can assume POSIX.
(iptablesSetupVirtInPost): Use portable 'test' syntax.
(iptablesLinkIPTablesBaseChain): Use POSIX $(()) syntax.

Using automated replacement with sed and editing I have now replaced all
occurrences of close() with VIR_(FORCE_)CLOSE() except for one, of
course. Some replacements were straight forward, others I needed to pay
attention. I hope I payed attention in all the right places... Please
have a look. This should have at least solved one more double-close
error.

The getnameinfo() function is more flexible than inet_ntop()
avoiding the need to if/else the code based on socket family.
Also make it support UNIX socket addrs and allow inclusion
of a port (service) address. Finally do proper error reporting
via normal APIs.

* src/conf/domain_conf.c, src/nwfilter/nwfilter_ebiptables_driver.c,
  src/qemu/qemu_conf.c: Fix error handling with virSocketFormat
* src/util/network.c: Rewrite virSocketFormat to use getnameinfo
  and cope with UNIX socket addrs.

The nwIPAddress was simply a wrapper about virSocketAddr.
Just use the latter directly, removing all the extra field
de-references from code & helper APIs for parsing/formatting.

Also remove all the redundant casts from strong types to
void * and then immediately back to strong types.

* src/conf/nwfilter_conf.h: Remove nwIPAddress
* src/conf/nwfilter_conf.c, src/nwfilter/nwfilter_ebiptables_driver.c:
  Update to use virSocketAddr and remove void * casts.

 Avoid the enforcement of direction if
- icmp rules specify the type/code information
- the 'skipMatch' variable is set to 'true'

In the table built for traffic coming from the VM going to the host make the following changes:

- don't ACCEPT the packets but do a 'RETURN' and let the host-specific firewall rules in subsequent rules evaluate whether the traffic is allowed to enter

- use the '-m state' in the rules as everywhere else

The following filter transition from a filter allowing incoming TCP connections

  <rule action='accept' direction='in' priority='401'>
    <tcp/>
  </rule>
  <rule action='accept' direction='out' priority='500'>
    <tcp/>
  </rule>

to one that does not allow them

  <rule action='drop' direction='in' priority='401'>
    <tcp/>
  </rule>
  <rule action='accept' direction='out' priority='500'>
    <tcp/>
  </rule>

did previously not cut off existing (ssh) connections but only prevented newly initiated ones. The attached patch allows to cut off existing connections as well, thus enforcing what the filter is showing.

I had only tested with a configuration where the physical interface is connected to the bridge where the filters are applied. This patch now also solves a filtering problem where the physical interface is not connected to the bridge, but the bridge is given an IP address and the host routes between bridge and physical interface. Here the filters drop non-allowed traffic on the outgoing side on the host.

to avoid long timeouts waiting for DNS servers

In this patch I am extending the rule instantiator to create the state
match according to the state attribute in the XML. Only one iptables
rule in the incoming or outgoing direction will be created for a rule
in direction 'in' or 'out' respectively. A rule in direction 'inout' does
get iptables rules in both directions.

Fixing memory leak shown by valgrind and freeing buffer in two more places.

In this patch I am extending the rule instantiator to create the comment
node where supported, which is the case for iptables and ip6tables.

Since commands are written in the format

cmd='iptables ...-m comment --comment \"\" '

certain characters ('`) in the comment need to be escaped to
prevent comments from becoming commands themselves or cause other
forms of (bash) substitutions. I have tested this with various input and in
my tests the input made it straight into the comment. A test case for TCK
will be provided separately that tests this.

The patch below reports a warning in the log if the generated ip(6)tables rules would not be effective due to the proc filesystem entries

    /proc/sys/net/bridge/bridge-nf-call-iptables
    /proc/sys/net/bridge/bridge-nf-call-ip6tables

containing a '0'. The warning tells the user what to do. I am rate-limiting the warning message to appear only every 10 seconds.

While testing the SIGHUP handling and reloading of the nwfilter driver, I found that when the filters are rebuilt and mutlipe threads handled the individual interfaces, concurrently running multiple external bash scripts causes strange failures even though the executed ebtables commands are working on different tables for different interfaces. I cannot say for sure where the concurrency problems are caused, but introducing this lock definitely helps.