1. 01 2月, 2008 1 次提交
  2. 29 1月, 2008 3 次提交
    • E
      [NET]: prot_inuse cleanups and optimizations · 65f76517
      Eric Dumazet 提交于
      1) Cleanups (all functions are prefixed by sock_prot_inuse)
      
      sock_prot_inc_use(prot) -> sock_prot_inuse_add(prot,-1)
      sock_prot_dec_use(prot) -> sock_prot_inuse_add(prot,-1)
      sock_prot_inuse()       -> sock_prot_inuse_get()
      
      New functions :
      
      sock_prot_inuse_init() and sock_prot_inuse_free() to abstract pcounter use.
      
      2) if CONFIG_PROC_FS=n, we can zap 'inuse' member from "struct proto",
      since nobody wants to read the inuse value.
      
      This saves 1372 bytes on i386/SMP and some cpu cycles.
      Signed-off-by: NEric Dumazet <dada1@cosmosbay.com>
      Signed-off-by: NDavid S. Miller <davem@davemloft.net>
      65f76517
    • P
      [INET]: Uninline the __inet_lookup_established function. · 77a5ba55
      Pavel Emelyanov 提交于
      This is -700 bytes from the net/ipv4/built-in.o
      
      add/remove: 1/0 grow/shrink: 1/3 up/down: 340/-1040 (-700)
      function                                     old     new   delta
      __inet_lookup_established                      -     339    +339
      tcp_sacktag_write_queue                     2254    2255      +1
      tcp_v4_err                                  1304     973    -331
      tcp_v4_rcv                                  2089    1744    -345
      tcp_v4_do_rcv                                826     462    -364
      
      Exporting is for dccp module (used via e.g. inet_lookup).
      Signed-off-by: NPavel Emelyanov <xemul@openvz.org>
      Signed-off-by: NDavid S. Miller <davem@davemloft.net>
      77a5ba55
    • P
      [INET]: Uninline the __inet_hash function. · 152da81d
      Pavel Emelyanov 提交于
      This one is used in quite many places in the networking code and
      seems to big to be inline.
      
      After the patch net/ipv4/build-in.o loses ~650 bytes:
      add/remove: 2/0 grow/shrink: 0/5 up/down: 461/-1114 (-653)
      function                                     old     new   delta
      __inet_hash_nolisten                           -     282    +282
      __inet_hash                                    -     179    +179
      tcp_sacktag_write_queue                     2255    2254      -1
      __inet_lookup_listener                       284     274     -10
      tcp_v4_syn_recv_sock                         755     493    -262
      tcp_v4_hash                                  389      35    -354
      inet_hash_connect                           1086     599    -487
      
      This version addresses the issue pointed by Eric, that
      while being inline this function was optimized by gcc
      in respect to the 'listen_possible' argument.
      Signed-off-by: NPavel Emelyanov <xemul@openvz.org>
      Signed-off-by: NDavid S. Miller <davem@davemloft.net>
      152da81d
  3. 26 11月, 2007 1 次提交
    • P
      [IPV4]: Fix memory leak in inet_hashtables.h when NUMA is on · 218ad12f
      Pavel Emelyanov 提交于
      The inet_ehash_locks_alloc() looks like this:
      
      #ifdef CONFIG_NUMA
      	if (size > PAGE_SIZE)
      		x = vmalloc(...);
      	else
      #endif
      		x = kmalloc(...);
      
      Unlike it, the inet_ehash_locks_alloc() looks like this:
      
      #ifdef CONFIG_NUMA
      	if (size > PAGE_SIZE)
      		vfree(x);
      	else
      #else
      		kfree(x);
      #endif
      
      The error is obvious - if the NUMA is on and the size
      is less than the PAGE_SIZE we leak the pointer (kfree is
      inside the #else branch).
      
      Compiler doesn't warn us because after the kfree(x) there's
      a "x = NULL" assignment, so here's another (minor?) bug: we 
      don't set x to NULL under certain circumstances.
      
      Boring explanation, I know... Patch explains it better.
      Signed-off-by: NPavel Emelyanov <xemul@openvz.org>
      Signed-off-by: NHerbert Xu <herbert@gondor.apana.org.au>
      218ad12f
  4. 11 11月, 2007 1 次提交
  5. 07 11月, 2007 1 次提交
    • E
      [INET]: Remove per bucket rwlock in tcp/dccp ehash table. · 230140cf
      Eric Dumazet 提交于
      As done two years ago on IP route cache table (commit
      22c047cc) , we can avoid using one
      lock per hash bucket for the huge TCP/DCCP hash tables.
      
      On a typical x86_64 platform, this saves about 2MB or 4MB of ram, for
      litle performance differences. (we hit a different cache line for the
      rwlock, but then the bucket cache line have a better sharing factor
      among cpus, since we dirty it less often). For netstat or ss commands
      that want a full scan of hash table, we perform fewer memory accesses.
      
      Using a 'small' table of hashed rwlocks should be more than enough to
      provide correct SMP concurrency between different buckets, without
      using too much memory. Sizing of this table depends on
      num_possible_cpus() and various CONFIG settings.
      
      This patch provides some locking abstraction that may ease a future
      work using a different model for TCP/DCCP table.
      Signed-off-by: NEric Dumazet <dada1@cosmosbay.com>
      Acked-by: NArnaldo Carvalho de Melo <acme@redhat.com>
      Signed-off-by: NDavid S. Miller <davem@davemloft.net>
      230140cf
  6. 26 10月, 2007 1 次提交
  7. 11 10月, 2007 1 次提交
  8. 09 2月, 2007 1 次提交
    • E
      [NET]: change layout of ehash table · dbca9b27
      Eric Dumazet 提交于
      ehash table layout is currently this one :
      
      First half of this table is used by sockets not in TIME_WAIT state
      Second half of it is used by sockets in TIME_WAIT state.
      
      This is non optimal because of for a given hash or socket, the two chain heads 
      are located in separate cache lines.
      Moreover the locks of the second half are never used.
      
      If instead of this halving, we use two list heads in inet_ehash_bucket instead 
      of only one, we probably can avoid one cache miss, and reduce ram usage, 
      particularly if sizeof(rwlock_t) is big (various CONFIG_DEBUG_SPINLOCK, 
      CONFIG_DEBUG_LOCK_ALLOC settings). So we still halves the table but we keep 
      together related chains to speedup lookups and socket state change.
      
      In this patch I did not try to align struct inet_ehash_bucket, but a future 
      patch could try to make this structure have a convenient size (a power of two 
      or a multiple of L1_CACHE_SIZE).
      I guess rwlock will just vanish as soon as RCU is plugged into ehash :) , so 
      maybe we dont need to scratch our heads to align the bucket...
      
      Note : In case struct inet_ehash_bucket is not a power of two, we could 
      probably change alloc_large_system_hash() (in case it use __get_free_pages()) 
      to free the unused space. It currently allocates a big zone, but the last 
      quarter of it could be freed. Again, this should be a temporary 'problem'.
      
      Patch tested on ipv4 tcp only, but should be OK for IPV6 and DCCP.
      Signed-off-by: NEric Dumazet <dada1@cosmosbay.com>
      Signed-off-by: NDavid S. Miller <davem@davemloft.net>
      dbca9b27
  9. 08 12月, 2006 1 次提交
  10. 29 9月, 2006 2 次提交
  11. 23 9月, 2006 2 次提交
  12. 26 4月, 2006 1 次提交
  13. 04 1月, 2006 2 次提交
  14. 06 11月, 2005 1 次提交
  15. 05 10月, 2005 1 次提交
  16. 04 10月, 2005 1 次提交
    • E
      [INET]: speedup inet (tcp/dccp) lookups · 81c3d547
      Eric Dumazet 提交于
      Arnaldo and I agreed it could be applied now, because I have other
      pending patches depending on this one (Thank you Arnaldo)
      
      (The other important patch moves skc_refcnt in a separate cache line,
      so that the SMP/NUMA performance doesnt suffer from cache line ping pongs)
      
      1) First some performance data :
      --------------------------------
      
      tcp_v4_rcv() wastes a *lot* of time in __inet_lookup_established()
      
      The most time critical code is :
      
      sk_for_each(sk, node, &head->chain) {
           if (INET_MATCH(sk, acookie, saddr, daddr, ports, dif))
               goto hit; /* You sunk my battleship! */
      }
      
      The sk_for_each() does use prefetch() hints but only the begining of
      "struct sock" is prefetched.
      
      As INET_MATCH first comparison uses inet_sk(__sk)->daddr, wich is far
      away from the begining of "struct sock", it has to bring into CPU
      cache cold cache line. Each iteration has to use at least 2 cache
      lines.
      
      This can be problematic if some chains are very long.
      
      2) The goal
      -----------
      
      The idea I had is to change things so that INET_MATCH() may return
      FALSE in 99% of cases only using the data already in the CPU cache,
      using one cache line per iteration.
      
      3) Description of the patch
      ---------------------------
      
      Adds a new 'unsigned int skc_hash' field in 'struct sock_common',
      filling a 32 bits hole on 64 bits platform.
      
      struct sock_common {
      	unsigned short		skc_family;
      	volatile unsigned char	skc_state;
      	unsigned char		skc_reuse;
      	int			skc_bound_dev_if;
      	struct hlist_node	skc_node;
      	struct hlist_node	skc_bind_node;
      	atomic_t		skc_refcnt;
      +	unsigned int		skc_hash;
      	struct proto		*skc_prot;
      };
      
      Store in this 32 bits field the full hash, not masked by (ehash_size -
      1) Using this full hash as the first comparison done in INET_MATCH
      permits us immediatly skip the element without touching a second cache
      line in case of a miss.
      
      Suppress the sk_hashent/tw_hashent fields since skc_hash (aliased to
      sk_hash and tw_hash) already contains the slot number if we mask with
      (ehash_size - 1)
      
      File include/net/inet_hashtables.h
      
      64 bits platforms :
      #define INET_MATCH(__sk, __hash, __cookie, __saddr, __daddr, __ports, __dif)\
           (((__sk)->sk_hash == (__hash))
           ((*((__u64 *)&(inet_sk(__sk)->daddr)))== (__cookie))   &&  \
           ((*((__u32 *)&(inet_sk(__sk)->dport))) == (__ports))   &&  \
           (!((__sk)->sk_bound_dev_if) || ((__sk)->sk_bound_dev_if == (__dif))))
      
      32bits platforms:
      #define TCP_IPV4_MATCH(__sk, __hash, __cookie, __saddr, __daddr, __ports, __dif)\
           (((__sk)->sk_hash == (__hash))                 &&  \
           (inet_sk(__sk)->daddr          == (__saddr))   &&  \
           (inet_sk(__sk)->rcv_saddr      == (__daddr))   &&  \
           (!((__sk)->sk_bound_dev_if) || ((__sk)->sk_bound_dev_if == (__dif))))
      
      
      - Adds a prefetch(head->chain.first) in 
      __inet_lookup_established()/__tcp_v4_check_established() and 
      __inet6_lookup_established()/__tcp_v6_check_established() and 
      __dccp_v4_check_established() to bring into cache the first element of the 
      list, before the {read|write}_lock(&head->lock);
      Signed-off-by: NEric Dumazet <dada1@cosmosbay.com>
      Acked-by: NArnaldo Carvalho de Melo <acme@ghostprotocols.net>
      Signed-off-by: NDavid S. Miller <davem@davemloft.net>
      81c3d547
  17. 30 8月, 2005 12 次提交