dpkg-source in jessie doesn't know what to do with them.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

Signed-off-by: NJulien Cristau <jcristau@debian.org>

Otherwise foo.orig.tar.gz.asc matches orig_tar before it has a change to
be matched against orig_tar_sig.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

Doing so would violate the database constraints.

Its last use seems to have been removed in
90885e3c.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

Signed-off-by: NJulien Cristau <jcristau@debian.org>

Move hashfile update and by-hash links creation to separate methods.  No
actual code changes.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

Instead of doing two queries per filename and per hash function, store a
set of known hashfiles up front and group the UPDATEs in a single query.

Suggested by Colin Watson.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

When setting suite.byhash to false, the suite's existing hashfiles need
to be marked as unreferenced so they can be cleaned up.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

This way we only touch the filesystem after the db is updated.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

Group knowledge of hash function and corresponding name in our db and in
Release files in a class instead of playing clever but unreadable
tricks.

Suggested by Colin Watson.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

This makes sure the Release files are synced last, after all Packages
indices (and their correspondig by-hash files).

Fixed in python-debian 0.1.11; squeeze had 0.1.18.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

These seem to no longer be used as of c91ff0bd

This is controlled via the database, however
fcfcdcc1 left the setting in
config/debian/dak.conf.

It doesn't seem to be used anywhere.

Add a per-suite boolean to enable by-hash; store the by-hash files in
the db, and record when they stopped being referenced, so that
clean-suites can delete them after the archive's stayofexecution time.

In generate-release, where we have checksums for all the things,
hardlink files to the by-hash dir for each of the suite's configured
hash methods.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

---
changes in v2:
- use archive.stayofexecution as delay before removing files from
  by-hash
- don't assume any particular ordering for suite.checksums

changes in v3:
- rebase on latest master, update115 is now update116
- handle missing files in clean_byhash

changes in v4:
- use hardlinks instead of symlinks
- don't initialize `unreferenced` to its default value
- in clean_byhash, remove useless fetchall, and handle ENOENT from
  unlink instead of checking for existence beforehand

Needed to run the test suite:

    ======================================================================
    ERROR: test_importing_rm (test_imports.ImportTestCase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/srv/daksrc/tests/test_imports.py", line 15, in test_fn
        __import__('dak', fromlist=[cmd])
      File "/srv/daksrc/dak/rm.py", line 54, in <module>
        from daklib.rm import remove
      File "/srv/daksrc/dak/daklib/rm.py", line 52, in <module>
        import debianbts as bts
    ImportError: No module named debianbts
Signed-off-by: NLuke Faraone <lfaraone@debian.org>

Before Debian LTS started to use the security archive, all uploads that
would trigger buildds would be sourceful uploads to a policy queue. So
new packages entering the build queues and pushing the public mirrors
would likely not happen in the same cron.unchecked run.

With Debian LTS, uploads would go to both the build queues and trigger a
public mirror push. If the mirror push takes more than three minutes,
cron.buildd would however not notify w-b of changes.

This patch moves the w-b notification right behind the update of the
build queues and before the mirror push to avoid the problem. Ideally
we would not rely on anything happening within the last three minutes
though.

Let the python interpreter handle it and actually exit with the right
status, otherwise when a sub-command calls sys.exit(1) we end up exiting
0 instead, which is confusing if not dangerous.
Signed-off-by: NJulien Cristau <jcristau@debian.org>

With "wheezy" being used for Debian LTS, the archive needs to be
published not only when policy queue uploads are accepted, but also
when direct uploads to "wheezy" arrive.

The same applies to uploads to "jessie-kfreebsd".

Newer SQLAlchemy versions seem to track the `binary` attribute of
`BinaryMetadata`.  This means the association proxy in `DBBinary`
would first create a `BinaryMetadata` with `binary` set to `None`,
flush this to the database and then set the actual value of `binary`
we want.  However the flush fails, as `NULL` is not allowed for the
`binary_id` column.

The problem can be avoided by not setting `binary` to `None`.  This is
treated as an "undefined value" by SQLAlchemy and changing it later
means the database row never sees `NULL`.

The same applies to the `source` attribute of `SourceMetadata`.

The orig tarball regexes match also the orig tarball detached
signatures because they are not anchored at the end, and the loop
short-circuits on first match. So we have to move them before their
more general counterparts.

Fixes commit 84f230b6.

Calling `f.read()` would read the entire file at once which is
wasteful. It's also not intended given the read loop.

Also replace a use of `readline()` with `read(65536)` as well where we
want to read the entire file and not only single lines.

We should only allow upstream tarballs with detached signatures, but not
native tarballs, «<source>_<version>.orig.asc» or «<source>_<version>.asc»
filenames. We also need to allow this on all relevant regexes. This commit
fixes and refactors the regexes to implement the above, and to make it
easier to modify a single place in the future.

Fixes commit 4911282c.

Closes: #759401

When there's an unknown file in a source package, the function spits
this:

  ,---
  Processing raised an exception: global name 'reject' is not defined.
  Traceback (most recent call last):
    File "/srv/ftp-master.debian.org/dak/dak/daklib/archive.py", line 966, in check
      chk().check(self)
    File "/srv/ftp-master.debian.org/dak/dak/daklib/checks.py", line 497, in check
      rejects = utils.check_dsc_files(dsc_fn, control, source.files.keys())
    File "/srv/ftp-master.debian.org/dak/dak/daklib/utils.py", line 343, in check_dsc_files
      reject("%s: unexpected source file '%s'" % (dsc_filename, f))
  NameError: global name 'reject' is not defined
  `---

Just set the reject message and break from the loop.