sync

THANKS

@@ -8,6 +8,7 @@ Anthony Towns		       <ajt@debian.org>
 Antti-Juhani Kaijanaho	       <ajk@debian.org>
 Ben Collins		       <bcollins@debian.org>
 Brendan O'Dea		       <bod@debian.org>
+Brian Russo		       <wolfie@debian.org>
 Chris Leishman		       <masklin@debian.org>
 Daniel Jacobwitz	       <dan@debian.org>
 Drake Diedrich		       <dld@debian.org>

TODO

+TOFIX
+=====
+
 Urgent
 ------
 
   o Claire needs to know about sections on auric.
+  o maintainers file needs overrides.
+  o jenna needs to use order by to keep aj from going nutzo
+  o fix the silly rhona p-u bug
+  o have cron job symlink -dr properly
+  o need to have process_it run a big try: loop so that individual
+    package screws up don't kill the daily cron run.
+  o Add an 'add' ability to melanie
+  o Make experimental work
 
 Less Urgent
 -----------
 
-  o Need to merge non-non-US and non-US DBs.
-  o need a poolifier that will poolify X mb a day.. (catherine)
-  o Need to vacuum daily
-
-  == 
-
-  o Log files for jenna, rhona and katie.
-  o Optimize all the queries by using EXAMINE and building some INDEXs.
-  o enclose all the setting SQL stuff in transactions (mostly done).
-  o jenna needs to use order by to keep aj from going nutzo
-  o remove empty directories in the pool in cron.weekly or so
-
-  == 
+  o [Hard] Need to merge non-non-US and non-US DBs.
+  o [Easy] Need to vacuum daily
+  o [Easy] remove empty directories in the pool in cron.weekly or so
 
-  o ability to rebuild all other tables from dists _or_ pools (in the event of disaster) (?)
-  o check errors on apt_pkg calls so we don't bomb out on daily runs (?)
-  o Need to finish alyson (a way to sync katie.conf and the DB)
-  o dinstall should remove != stable on stable install, not just f + u
-  o jenna needs to validate arguments
-  o jenna's untochable flag doesn't stop adding new packages
-  o db_access' get_files needs to use exceptions not this None, > 0, < 0 return val BS (?)
+  o Fix dinstall to remove != stable on stable install, not just f + u
   o Fix 'mixed' component handling
   o Fix component handling in overrides (aj)
   o Fix lack of entires in source overrides (aj)
-  o Fix potato files to not regen and sort by in jenna
   o Fix rhona to unconditionally read override file on startup
   o Fix dinstall to warn if not in incoming or p-u
+  o db_access' get_files needs to use exceptions not this None, > 0, < 0 return val BS (?)
+  o check errors on apt_pkg calls (?)
+  o The untouchable flag doesn't stop adding new packages
+  o jenna needs to validate arguments (?)
 
 ===================================================================================================
 
-Packaging TODO
---------------
+TODO
+====
+
+Packaging
+---------
 
   o Install python libraries (db_access and utils)
   o Install config file
   o Fix stuff to look in sensible places for libs and config file in debian package (?)
   o man pages and/or documentation
 
-Future Enhancements
--------------------
+Urgent
+------
+
+  o Secure incoming handling
+  o [Hard] dependency checking
+  o dpkg 1.7 Changed-By field support
+  o Overrides for Maintainers
+  o Possibly disallow binary versions >> source except for bin-only NMUs (Branden Robinson, IWJ & others)
+  o Kill orphaned
+
+Less Urgent
+-----------
+
+  o Need a poolifier that will poolify X mb a day.. (catherine)
+  o Need to optimize all the queries by using EXAMINE and building some INDEXs.
+  o Need to enclose all the setting SQL stuff in transactions (mostly done).
+  o Need to finish alyson (a way to sync katie.conf and the DB)
+  o Need log files for jenna, rhona and katie.
+  o Need the ability to rebuild all other tables from dists _or_ pools (in the event of disaster) (?)
+  o Make the --help and --version options do stuff for all scripts
+  o Need to check for .dsc when source is mentioned
 
-  o possibly disallow binary versions >> source except for bin-only NMUs (Branden Robinson)
-  o project/orphaned should be a timed dist so that things only stay
-    in there temporarily (say 3 months) [aj]
-  o make the --help and --version options do stuff for all scripts
-  o check for .dsc when source is mentioneD?
   o fix parse_changes()/build_file_list() to sanity check filenames
   o saftey check and/or rename debs so they match what they should be
+
   o charisma can't handle whitespace-only lines (for the moment, this is feature)
-  o Fix problems with bad .sig .changes having no Maintainer field to REJECT to
-  o Dep checking
   o Should use $EDITOR, not hardcode vi
-  o should reject timestamp fucked debs
-  o dpkg 1.7 Changed-By field support
-  o Secure incoming handling
-  o revamp NEW package handling so we odn't have to read in the override file
-  o make mkmaintainers obey override changes
-  o Report stuff (? needed ?)
-  o handle the case of 1:1.1 which would overwrite 1.1
+  o Should reject packages with bad timestamps
+  o Replicate old dinstall report stuff (? needed ?)
+  o Handle the case of 1:1.1 which would overwrite 1.1 (?)
   o aj's binary-all stuff (foo-doc depending on foo) (?)
   o heidi should report suite name not ID [aj]
 
-  o fubar and warn/error wrappers like in C
-  o generic way of saying isabinary and isadsc.
+  o fubar and warn/error wrappers like in C (?)
+  o generic way of saying isabinary and isadsc. (?)
   o substitution stuff (cf. userdir-ldap) for announce, reject etc.
 
   o s/distribution/suite/g

docs/melanie.1.sgml

@@ -32,7 +32,7 @@
   
   <RefSect1><Title>Description</>
     <para>   
-      <command>melanie</command> is the command line tool to add and remove packages from suites.
+      <command>melanie</command> is the command line tool to add and remove package sets from suites with enforced logging, optional bug closing and override updates.
 
   <RefSect1><Title>Options</>
     
@@ -90,12 +90,13 @@
       <para>There are 3 methods for selecting packages.</para>
       <itemizedlist>
 	<listitem>
-	  <para>Source + Binary. (default)</para>
+	  <para>Source + Binaries. (default)</para>
 	  <para>In this mode <command>melanie</command> will assume each of the package(s) passed as arguments are source packages and will also remove any binary packages built from these source packages.</para>
 	</listitem>
 	<listitem>
 	  <para>Binary only.</para>
 	  <para>Only binary packages are searched; source packages are ignored.  This mode is chosen by use of the <option>-b/--binary</option> switch.</para>
+	  <para>This should <emphasis>only</emphasis> be used for orphaned binary packages (i.e. those no longer built by source packages); otherwise, in any system (e.g. Debian) which has auto-building, pointless (and uninstallable) recompiles will be triggered.</para>
 	</listitem>
 	<listitem>
 	  <para>Source only.</para>
@@ -104,15 +105,71 @@
       </itemizedlist>
     </refsect1>
 
+    <refsect1>
+      <title>Configuration</title>
+      <para><command>melanie</command> uses katie's configuration file. It follows the typical ISC configuration format as seen in ISC tools like bind 8 and dhcpd.  Apart from being able to alter the defaults for command line options, the following configuration items are used:</para>
+      <variablelist>
+	<varlistentry>
+	  <term>Melanie::MyEmailAddress</term>
+	  <listitem>
+	    <para>This is used as the From: line for bug closing mails as per the -d/--done command line switch.  It, obviously, must be a RFC-822 valid email address.</para>
+	  </listitem>
+	</varlistentry>
+	<varlistentry>
+	  <term>Melanie::LogFile</term>
+	  <listitem>
+	    <para>This is the (absolute) file name of the logfile that melanie unconditionally writes too.  This can not be empty or an invalid file.</para>
+	  </listitem>
+	</varlistentry>
+      </variablelist>
+    </refsect1>
+
   <refsect1>
     <title>Examples</title>
-    <para></para>
+      <para>The first example is of a source+binaries package removal.</para>
+      <informalexample>
+	<literallayout>
+troup@auric| ~ $melanie -d 68136 -m "Requested by tbm@; confirmed with maintainer.  Superseded by libgmp2" gmp1   
+Working... done.
+Will remove the following packages from unstable:
+
+      gmp1 |  1.3.2-8.2 | source, alpha, hppa, arm, i386, m68k, powerpc, sparc
+  gmp1-dev |  1.3.2-8.2 | alpha, hppa, arm, i386, m68k, powerpc, sparc
+
+
+------------------- Reason -------------------
+Requested by tbm@; confirmed with maintainer.  Superseded by libgmp2
+----------------------------------------------
+
+Continue (y/N)? y
+ Deleting... done.
+troup@auric| ~ $
+	  </literallayout>
+      </informalexample>
+      <para>The second example is of a binary-only multi-package removal.</para>
+      <informalexample>
+	<literallayout>
+troup@auric| ~ $melanie -d 82562 -m "Requested by paul@; NBS." -b libgtkextra{5,9,10}
+Working... done.
+Will remove the following packages from unstable:
+
+libgtkextra10 |  0.99.10-2 | alpha, i386, m68k, powerpc, sparc
+libgtkextra5 |   0.99.5-1 | alpha, i386, m68k, powerpc, sparc
+libgtkextra9 |   0.99.9-1 | alpha, i386, m68k, powerpc, sparc
+
+Will also close bugs: 82562
+
+------------------- Reason -------------------
+Requested by paul@; NBS.
+----------------------------------------------
+
+Continue (y/N)? y
+ Deleting... done.
+troup@auric| ~ $
+	</literallayout>
+      </informalexample>
   </refsect1>
 
-  <RefSect1><Title>Description</>
-    <para>   
-      <command>melanie</command> is the command line tool to add and remove packages from suites.
-
   <RefSect1><Title>Diagnostics</>
     <para>
       <command>melanie</command> returns zero on normal operation, non-zero on error.

katie

@@ -2,7 +2,7 @@
 
 # Installs Debian packaes
 # Copyright (C) 2000  James Troup <james@nocrew.org>
-# $Id: katie,v 1.18 2001-01-16 21:52:37 troup Exp $
+# $Id: katie,v 1.19 2001-01-18 04:51:10 troup Exp $
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -113,9 +113,9 @@ def check_signature (filename):
 
 # See if a given package is in the override table
 
-def in_override_p (package, component, suite, binary_type):
-    global overrides;
-
+def in_override_p (package, component, suite, binary_type, file):
+    global files;
+    
     if binary_type == "": # must be source
         type = "dsc";
     else:
@@ -136,15 +136,20 @@ def in_override_p (package, component, suite, binary_type):
     if string.lower(component[:7]) == "non-us/":
         component = component[7:];
 
-    q = projectB.query("SELECT package FROM override WHERE package = '%s' AND suite = %s AND component = %s AND type = %s"
+    q = projectB.query("SELECT s.section, p.priority FROM override o, section s, priority p WHERE package = '%s' AND suite = %s AND component = %s AND type = %s AND o.section = s.id AND o.priority = p.id"
                        % (package, suite_id, component_id, type_id));
     result = q.getresult();
     # If checking for a source package fall back on the binary override type
     if type == "dsc" and not result:
         type_id = db_access.get_override_type_id("deb");
-        q = projectB.query("SELECT package FROM override WHERE package = '%s' AND suite = %s AND component = %s AND type = %s"
+        q = projectB.query("SELECT s.section, p.priority FROM override o, section s, priority p WHERE package = '%s' AND suite = %s AND component = %s AND type = %s AND o.section = s.id AND o.priority = p.id"
                            % (package, suite_id, component_id, type_id));
         result = q.getresult();
+
+    # Remember the section and priority so we can check them later if appropriate
+    if result != []:
+        files[file]["override section"] = result[0][0];
+        files[file]["override priority"] = result[0][1];
         
     return result;
 
@@ -366,7 +371,7 @@ def check_files():
                 continue
 
             # See if the package is NEW
-            if not in_override_p(files[file]["package"], files[file]["component"], suite, files[file].get("dbtype","")):
+            if not in_override_p(files[file]["package"], files[file]["component"], suite, files[file].get("dbtype",""), file):
                 files[file]["new"] = 1
                 
             # Find any old binary packages
@@ -576,6 +581,51 @@ def check_md5sums ():
             if apt_pkg.md5sum(file_handle) != files[file]["md5sum"]:
                 reject_message = reject_message + "Rejected: md5sum check failed for %s.\n" % (file);
 
+def check_override ():
+    # Only check section & priority on sourceful uploads
+    if not changes["architecture"].has_key("source"):
+        return;
+
+    summary = ""
+    for file in files.keys():
+        if not files[file].has_key("new") and (files[file]["type"] == "dsc" or files[file]["type"] == "deb"):
+            section = files[file]["section"];
+            override_section = files[file]["override section"];
+            if section != override_section and section != "-":
+                summary = summary + "%s: section is overridden from %s to %s.\n" % (file, section, override_section);
+            if files[file]["type"] == "deb": # don't do priority for source
+                priority = files[file]["priority"];
+                override_priority = files[file]["override priority"];
+                if priority != override_priority and priority != "-":
+                    summary = summary + "%s: priority is overridden from %s to %s.\n" % (file, priority, override_priority);
+
+    if summary == "":
+        return;
+    
+    mail_message = """Return-Path: %s
+From: %s
+To: %s
+Bcc: troup@auric.debian.org
+Subject: %s override disparity
+
+There are disparities between your recently installed upload and the
+override file for the following file(s):
+
+%s
+Either the package or the override file is incorrect.  If you think
+the override is correct and the package wrong please fix the package
+so that this disparity is fixed in the next upload.  If you feel the
+override is incorrect then please reply to this mail and explain why.
+
+--
+Debian distribution maintenance software
+
+(This message was generated automatically; if you believe that there
+is a problem with it please contact the archive administrators by
+mailing ftpmaster@debian.org)
+""" % (Cnf["Dinstall::MyEmailAddress"], Cnf["Dinstall::MyEmailAddress"], changes["maintainer822"], changes["source"], summary);
+    utils.send_mail (mail_message, "")
+
 #####################################################################################################################
 
 def action (changes_filename):
@@ -805,6 +855,7 @@ Installing:
 %s""" % (Cnf["Dinstall::MyEmailAddress"], Cnf["Dinstall::MyEmailAddress"], changes["maintainer822"], os.path.basename(changes_filename), reject_message, summary, installed_footer)
         utils.send_mail (mail_message, "")
         announce (short_summary, 1)
+        check_override ();
 
 #####################################################################################################################
 

katie.conf

@@ -277,10 +277,12 @@ Suite
 	  "hppa" "";
 	  "hurd-i386" "";
 	  "i386" "";
+	  "ia64" "";
 	  "m68k" "";
 	  "mips" "";
 	  "mipsel" "";
 	  "powerpc" "";
+	  "s390" "";
 	  "sh" "";
 	  "sparc" "";
 	};
@@ -323,10 +325,12 @@ Architectures
   hppa "HP PA RISC";
   arm "Arm";
   i386 "Intel ia32";
+  ia64 "Intel ia64";
   m68k "Motorola Mc680x0";
   mips "SGI MIPS";
   mipsel "SGI MIPS (Little Endian)";
   powerpc "PowerPC";
+  s390 "IBM S/390";
   sh "Hitatchi SuperH";
   sparc "Sun SPARC/UltraSPARC";
 

katie.conf-non-US

@@ -276,10 +276,12 @@ Suite
 	  "hppa" "";
 	  "hurd-i386" "";
 	  "i386" "";
+	  "ia64" "";
 	  "m68k" "";
 	  "mips" "";
 	  "mipsel" "";
 	  "powerpc" "";
+	  "s390" "";
 	  "sh" "";
 	  "sparc" "";
 	};
@@ -322,10 +324,12 @@ Architectures
   hppa "HP PA RISC";
   arm "Arm";
   i386 "Intel ia32";
+  ia64 "Intel ia64";
   m68k "Motorola Mc680x0";
   mips "SGI MIPS";
   mipsel "SGI MIPS (Little Endian)";
   powerpc "PowerPC";
+  s390 "IBM S/390";
   sh "Hitatchi SuperH";
   sparc "Sun SPARC/UltraSPARC";