Update the network filter driver to use virReportError instead
of the virNWFilterReportError custom macro
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

This removes all the per-file error reporting macros
from the code in src/conf/
Signed-off-by: NDaniel P. Berrange <berrange@redhat.com>

Introduce new members in the virMacAddr 'class'
- virMacAddrSet: set virMacAddr from a virMacAddr
- virMacAddrSetRaw: setting virMacAddr from raw 6 byte MAC address buffer
- virMacAddrGetRaw: writing virMacAddr into raw 6 byte MAC address buffer
- virMacAddrCmp: comparing two virMacAddr
- virMacAddrCmpRaw: comparing a virMacAddr with a raw 6 byte MAC address buffer

then replace raw MAC addresses by replacing

- 'unsigned char *' with virMacAddrPtr
- 'unsigned char ... [VIR_MAC_BUFLEN]' with virMacAddr

and introduce usage of above functions where necessary.

This patch adds support for the recent ipset iptables extension
to libvirt's nwfilter subsystem. Ipset allows to maintain 'sets'
of IP addresses, ports and other packet parameters and allows for
faster lookup (in the order of O(1) vs. O(n)) and rule evaluation
to achieve higher throughput than what can be achieved with
individual iptables rules.

On the command line iptables supports ipset using

iptables ... -m set --match-set <ipset name> <flags> -j ...

where 'ipset name' is the name of a previously created ipset and
flags is a comma-separated list of up to 6 flags. Flags use 'src' and 'dst'
for selecting IP addresses, ports etc. from the source or
destination part of a packet. So a concrete example may look like this:

iptables -A INPUT -m set --match-set test src,src -j ACCEPT

Since ipset management is quite complex, the idea was to leave ipset 
management outside of libvirt but still allow users to reference an ipset.
The user would have to make sure the ipset is available once the VM is
started so that the iptables rule(s) referencing the ipset can be created.

Using XML to describe an ipset in an nwfilter rule would then look as
follows:

  <rule action='accept' direction='in'>
    <all ipset='test' ipsetflags='src,src'/>
  </rule>

The two parameters on the command line are also the two distinct XML attributes
'ipset' and 'ipsetflags'.

FYI: Here is the man page for ipset:

https://ipset.netfilter.org/ipset.man.html

Regards,
    Stefan

This patch addresses the following coverity findings:

/libvirt/src/conf/nwfilter_params.c:157:
deref_parm: Directly dereferencing parameter "val".

/libvirt/src/conf/nwfilter_params.c:473:
negative_returns: Using variable "iterIndex" as an index to array "res->iter".

/libvirt/src/nwfilter/nwfilter_ebiptables_driver.c:2891:
unchecked_value: No check of the return value of "virAsprintf(&protostr, "-d 01:80:c2:00:00:00 ")".

/libvirt/src/nwfilter/nwfilter_ebiptables_driver.c:2894:
unchecked_value: No check of the return value of "virAsprintf(&protostr, "-p 0x%04x ", l3_protocols[protoidx].attr)".

/libvirt/src/nwfilter/nwfilter_ebiptables_driver.c:3590:
var_deref_op: Dereferencing null variable "inst".

Fix the support for trusted DHCP server in the ebtables code's
hard-coded function applying DHCP only filtering rules:
Rather than using a char * use the more flexible
virNWFilterVarValuePtr that contains the trusted DHCP server(s)
IP address. Process all entries.

Since all callers so far provided NULL as parameter, no changes
are necessary in any other code.

Return statements with parameter enclosed in parentheses were modified
and parentheses were removed. The whole change was scripted, here is how:

List of files was obtained using this command:
git grep -l -e '\<return\s*([^()]*\(([^()]*)[^()]*\)*)\s*;' |             \
grep -e '\.[ch]$' -e '\.py$'

Found files were modified with this command:
sed -i -e                                                                 \
's_^\(.*\<return\)\s*(\(\([^()]*([^()]*)[^()]*\)*\))\s*\(;.*$\)_\1 \2\4_' \
-e 's_^\(.*\<return\)\s*(\([^()]*\))\s*\(;.*$\)_\1 \2\3_'

Then checked for nonsense.

The whole command looks like this:
git grep -l -e '\<return\s*([^()]*\(([^()]*)[^()]*\)*)\s*;' |             \
grep -e '\.[ch]$' -e '\.py$' | xargs sed -i -e                            \
's_^\(.*\<return\)\s*(\(\([^()]*([^()]*)[^()]*\)*\))\s*\(;.*$\)_\1 \2\4_' \
-e 's_^\(.*\<return\)\s*(\([^()]*\))\s*\(;.*$\)_\1 \2\3_'

Improve the logging during driver initialization when testing the command
line tools.

Rename virFormatMacAddr, virGenerateMacAddr and virParseMacAddr
to virMacAddrFormat, virMacAddrGenerate and virMacAddrParse
respectively

This patch introduces the capability to use a different iterator per
variable.

The currently supported notation of variables in a filtering rule like

  <rule action='accept' direction='out'>
     <tcp  srcipaddr='$A' srcportstart='$B'/>
  </rule>

processes the two lists 'A' and 'B' in parallel. This means that A and B
must have the same number of 'N' elements and that 'N' rules will be 
instantiated (assuming all tuples from A and B are unique).

In this patch we now introduce the assignment of variables to different
iterators. Therefore a rule like

  <rule action='accept' direction='out'>
     <tcp  srcipaddr='$A[@1]' srcportstart='$B[@2]'/>
  </rule>

will now create every combination of elements in A with elements in B since
A has been assigned to an iterator with Id '1' and B has been assigned to an
iterator with Id '2', thus processing their value independently.

The first rule has an equivalent notation of

  <rule action='accept' direction='out'>
     <tcp  srcipaddr='$A[@0]' srcportstart='$B[@0]'/>
  </rule>

Remove the requirement that DHCP messages have to be broadcasted.
DHCP requests are most often sent via broadcast but can be directed
towards a specific DHCP server. For example 'dhclient' takes '-s <server>'
as a command line parameter thus allowing DHCP requests to be sent to a
specific DHCP server.

If only iptables rules are created then two unnecessary ebtables chains
are also created. This patch fixes this and prevents these chains from
being created. They have been cleaned up properly, though.

This patch cleans up return codes in the nwfilter subsystem.

Some functions in nwfilter_conf.c (validators and formatters) are
keeping their bool return for now and I am converting their return
code to true/false.

All other functions now have failure return codes of -1 and success
of 0.

[I searched for all occurences of ' 1;' and checked all 'if ' and
adapted where needed. After that I did a grep for 'NWFilter' in the source
tree.]

Remove the virConnectPtr from the nwfilter's internal API calls as
far as possible.

In preparation for the DHCP Snooping code:
Pass an additional parameter into the applyDHCPOnly function
of the 'techdriver'.

Introduce a shell variable 'IBT' to invoke the ip(6)tables command.

Tested with libvirt-tck.

Introduce a shell variable 'EBT' to invoke the ebtables command.
Hard-code the used ebtables table to '-t nat'.

Tested with libvirt-tck.

Display the executed command and failure message if a command failed to
execute.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

This patch adds support for filtering of STP (spanning tree protocol) traffic
to the parser and makes us of the ebtables support for STP filtering. This code
now enables the filtering of traffic in chains with prefix 'stp'.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

With hunks borrowed from one of David Steven's previous patches, we now
add the capability of having a 'mac' chain which is useful to filter
for multiple valid MAC addresses.
Signed-off-by: NDavid L Stevens <dlstevens@us.ibm.com>
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

This patch adds support for filtering of VLAN (802.1Q) traffic to the
parser and makes us of the ebtables support for VLAN filtering. This code
now enables the filtering of traffic in chains with prefix 'vlan'.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

This patch extends the NWFilter driver for Linux (ebiptables) to create
rules for each member of a previously introduced list. If for example
an attribute value (internally) looks like this:

IP = [10.0.0.1, 10.0.0.2, 10.0.0.3]

then 3 rules will be generated for a rule accessing the variable 'IP',
one for each member of the list. The effect of this is that this now
allows for filtering for multiple values in one field. This can then be
used to support for filtering/allowing of multiple IP addresses per
interface.

An iterator is introduced that extracts each member of a list and
puts it into a hash table which then is passed to the function creating
a rule. For the above example the iterator would cause 3 loops.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

NWFilters can be provided name-value pairs using the following
XML notation:

      <filterref filter='xyz'>
        <parameter name='PORT' value='80'/>
        <parameter name='VAL' value='abc'/>
      </filterref>

The internal representation currently is so that a name is stored as a
string and the value as well. This patch now addresses the value part of it
and introduces a data structure for storing a value either as a simple
value or as an array for later support of lists.

This patch adjusts all code that was handling the values in hash tables
and makes it use the new data type.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

The previous patch extends the priority of filtering rules into negative
numbers. We now use this possibility to interleave the jumping into
chains with filtering rules to for example create the 'root' table of
an interface with the following sequence of rules:

Bridge chain: libvirt-I-vnet0, entries: 6, policy: ACCEPT
-p IPv4 -j I-vnet0-ipv4
-p ARP -j I-vnet0-arp
-p ARP -j ACCEPT 
-p 0x8035 -j I-vnet0-rarp
-p 0x835 -j ACCEPT 
-j DROP 

The '-p ARP -j ACCEPT' rule now appears between the jumps.
Since the 'arp' chain has been assigned priority -700 and the 'rarp'
chain -600, the above ordering can now be achieved with the following
rule:

  <rule action='accept' direction='out' priority='-650'>
    <mac protocolid='arp'/>
  </rule>

This patch now sorts the commands generating the above shown jumps into
chains and interleaves their execution with those for generating rules.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

So far rules' priorities have only been valid in the range [0,1000].
Now I am extending their priority into the range [-1000, 1000] for subsequently
being able to sort rules and the access of (jumps into) chains following
priorities.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

Use the name of the chain rather than its type index (enum).
This pushes the later enablement of chains with user-given names
into the XML parser. For now we still only allow those names that
are well known ('root', 'arp', 'rarp', 'ipv4' and 'ipv6').
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

Use scripts for the renaming and cleaning up of chains. This allows us to get
rid of some of the code that is only capable of renaming and removing chains
whose names are hardcoded.

A shell function 'collect_chains' is introduced that is given the name
of an ebtables chain and then recursively determines the names of all
chains that are accessed from this chain and its sub-chains using 'jumps'.

The resulting list of chain names is then used to delete all the found
chains by first flushing and then deleting them.

The same function is also used for renaming temporary filters to their final
names.

I tested this with the bash and dash as script interpreters.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

Use the previously introduced chain priorities to sort the chains for access
from an interface's 'root' table and have them created in the proper order.
This gets rid of a lot of code that was previously creating the chains in a 
more hardcoded way.

To determine what protocol a filter is used for evaluation do prefix-
matching, i.e., the filter 'arp' is used to filter for the 'arp' protocol,
'ipv4' for the 'ipv4' protocol and 'arp-xyz' will also be used to filter
for the 'arp' protocol following the prefix 'arp' in its name.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

For better handling of the sorting of chains introduce an internally used
priority. Use a lookup table to store the priorities. For now their actual
values do not matter just that the values cause the chains to be properly
sorted through changes in the following patches. However, the values are
chosen as negative so that once they are sorted along with filtering rules
(whose priority may only be positive for now) they will always be instantiated
before them (lower values cause instantiation before higher values). This
is done to maintain backwards compatibility.
Signed-off-by: NStefan Berger <stefanb@linux.vnet.ibm.com>

It's not worth even worrying about a temporary file, unless we
ever expect the script to exceed maximum command-line argument
length limits.

* src/nwfilter/nwfilter_ebiptables_driver.c (ebiptablesExecCLI):
Run the commands as an argument to /bin/sh, rather than worrying
about a temporary file.
(ebiptablesWriteToTempFile): Delete unused function.

If /tmp is mounted with the noexec flag (common on security-conscious
systems), then nwfilter will fail to initialize, because we cannot
run any temporary script via virRun("/tmp/script"); but we _can_
use "/bin/sh /tmp/script".  For that matter, using /tmp risks collisions
with other unrelated programs; we already have /var/run/libvirt as a
dedicated temporary directory for use by libvirt.

* src/nwfilter/nwfilter_ebiptables_driver.c
(ebiptablesWriteToTempFile): Use internal directory, not /tmp;
drop attempts to make script executable; and detect close error.
(ebiptablesExecCLI): Switch to virCommand, and invoke the shell to
read the script, rather than requiring an executable script.

The socket address APIs in src/util/network.h either take the
form  virSocketAddrXXX, virSocketXXX or virSocketXXXAddr.

Sanitize this so everything is virSocketAddrXXXX, and ensure
that the virSocketAddr parameter is always the first one.

* src/util/network.c, src/util/network.h: Santize socket
  address API naming
* src/conf/domain_conf.c, src/conf/network_conf.c,
  src/conf/nwfilter_conf.c, src/network/bridge_driver.c,
  src/nwfilter/nwfilter_ebiptables_driver.c,
  src/nwfilter/nwfilter_learnipaddr.c,
  src/qemu/qemu_command.c, src/rpc/virnetsocket.c,
  src/util/dnsmasq.c, src/util/iptables.c,
  src/util/virnetdev.c, src/vbox/vbox_tmpl.c: Update for
  API renaming

In preparation for a future patch adding new virFile APIs.

* src/util/files.h, src/util/files.c: Move...
* src/util/virfile.h, src/util/virfile.c: ...here, and rename
functions to virFile prefix.  Macro names are intentionally
left alone.
* *.c: All '#include "files.h"' uses changed.
* src/Makefile.am (UTIL_SOURCES): Reflect rename.
* cfg.mk (exclude_file_name_regexp--sc_prohibit_close): Likewise.
* src/libvirt_private.syms: Likewise.
* docs/hacking.html.in: Likewise.
* HACKING: Regenerate.

Seems reasonable to have all command wrappers in the same place

v2:
    Dont move SetInherit

v3:
    Comment spelling fix
    Adjust WARN0 comment
    Remove spurious #include movement
    Don't include sys/types.h
    Combine virExec enums
Signed-off-by: NCole Robinson <crobinso@redhat.com>

This patch enables filtering of gratuitous ARP packets using the following XML:

<rule action='accept' direction='in' priority='425'>
<arp gratuitous='true'/>
</rule>

These VIR_XXXX0 APIs make us confused, use the non-0-suffix APIs instead.

How do these coversions works? The magic is using the gcc extension of ##.
When __VA_ARGS__ is empty, "##" will swallow the "," in "fmt," to
avoid compile error.

example: origin				after CPP
	high_level_api("%d", a_int)	low_level_api("%d", a_int)
	high_level_api("a  string")	low_level_api("a  string")

About 400 conversions.

8 special conversions:
VIR_XXXX0("") -> VIR_XXXX("msg") (avoid empty format) 2 conversions
VIR_XXXX0(string_literal_with_%) -> VIR_XXXX(%->%%) 0 conversions
VIR_XXXX0(non_string_literal) -> VIR_XXXX("%s", non_string_literal)
  (for security) 6 conversions
Signed-off-by: NLai Jiangshan <laijs@cn.fujitsu.com>

We already have virAsprintf, so picking a similar name helps for
seeing a similar purpose.  Furthermore, the prefix V before printf
generally implies 'va_list', even though this variant was '...', and
the old name got in the way of adding a new va_list version.

global rename performed with:

$ git grep -l virBufferVSprintf \
  | xargs -L1 sed -i 's/virBufferVSprintf/virBufferAsprintf/g'

then revert the changes in ChangeLog-old.

Ebtables filtering doesn't work on macvtap device. Remove support for direct type of interface.

gcc 4.6 warns when a variable is initialized but isn't used afterwards:

vmware/vmware_driver.c:449:18: warning: variable 'vmxPath' set but not used [-Wunused-but-set-variable]

This patch fixes these warnings. There are still 2 offending files:

- vbox_tmpl.c: the variable is used inside an #ifdef and is assigned several
  times outside of #ifdef. Fixing the warning would have required wrapping
  all the assignment inside #ifdef which hurts readability.

vbox/vbox_tmpl.c: In function 'vboxAttachDrives':
vbox/vbox_tmpl.c:3918:22: warning: variable 'accessMode' set but not used [-Wunused-but-set-variable]

- esx_vi_types.generated.c: the name implies it's generated code and I
  didn't want to dive into the code generator

esx/esx_vi_types.generated.c: In function 'esxVI_FileQueryFlags_Free':
esx/esx_vi_types.generated.c:1203:3: warning: variable 'item' set but not used [-Wunused-but-set-variable]

This patch adds support for the evaluation of TCP flags in nwfilters.

It adds documentation to the web page and extends the tests as well.
Also, the nwfilter schema is extended.

The following are some example for rules using the tcp flags:

<rule action='accept' direction='in'>
    <tcp state='NONE' flags='SYN/ALL' dsptportstart='80'/>
</rule>
<rule action='drop' direction='in'>
    <tcp state='NONE' flags='SYN/ALL'/>
</rule>