This page describes how to make custom Debian packages for HCoop. <> = Overview = The idea is to keep track of each custom HCoop Debian package using three branches, which are as follows. 1. {{{upstream}}}: The source code from the current release of the upstream software. 1. {{{debian}}}: The source code plus the latest Debian packaging that Debian has for the software. 1. {{{master}}}: The source code plus the latest Debian packaging plus any changes that HCoop has made to the source or the packaging. If you are creating a native package (e.g. for configuration files) then you only have a `master` branch. = Developing Packages = Common to all of the types of packages we might develop. == Building a package == Years ago HCoop standardized on Git for VersionControl; as such we're using [[http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html|git-buildpackage]] to maintain our packages. First, make sure you are on the "master" branch by running: {{{ git branch -l }}} If you see an asterisk by "master", you're on the right branch. If we want to build the package with some uncommitted changes, as a sanity check, then do: {{{ git-buildpackage --git-ignore-new }}} When it comes time to test the changes, build the package using: {{{ git-buildpackage }}} The packages will be built and placed in the parent directory. To indicate that we are done making changes to this particular version of the Debian package, tag it with: {{{ git-buildpackage --git-tag }}} This makes the package version show up when you do {{{git tag -l}}}, for easy diffing and viewing. == New Packages == After creating the git-buildpackage repository, push it to the public HCoop debian packages git area: {{{ gbp-create-remote-repo --remote-url-pattern=/afs/hcoop.net/user/h/hc/hcoop/.hcoop-git/debian/'%(pkg)s'.git }}} We may revisit only having one are for Debian packages at a later time. = Creating a Configuration Package = Background: * http://debathena.mit.edu/config-packages/ * http://debathena.mit.edu/packaging/ * http://cdbs-doc.duckcorp.org/en/cdbs-doc.xhtml By storing configuration in Debian repositories and taking advantage of the debathena tools, we are able to preseed installations, and better distribute configuration that needs to be coherent across all machines. Quirks: * Set the Section: `hcoop-config/$section` (where `$section` = section of the package we are configuring ideally) * Note that, just as with forked Debian packages, the repository lives in a subdirectory of the main $package directory (otherwise, the src dir would be riddled with package build results) Our configuration build trees are kept in `/afs/hcoop.net/common/debian/src/hcoop/`. To create a new repository: {{{ cd /afs/hcoop.net/common/debian/src/hcoop/ mkdir -p hcoop-$package-config/hcoop-$package-config cd hcoop-$package-config/hcoop-$package-config git init dh_make --native --indep # clean up *.ex files, update control, rules, etc. using http://debathena.mit.edu/config-packages/ or current packages as a guide # make sure it builds dh_clean git add debian/** #and any other files git commit git-buildpackage --git-tag }}} = Forking a Debian Package = Once upon a time http://backports.debian.org was not part of Debian and we had to maintain our own backports. This might be needed in the future if we want to pull software that isn't backported officially, but consider the following officially discouraged. == Making a new custom package == If you want to make changes to an existing Debian package, and we haven't made our own custom package before, then do the following. {{{ mkdir -p /afs/hcoop.net/common/debian/src/backports// cd /afs/hcoop.net/common/debian/src/backports// # Browse http://packages.debian.org/ and find a link to a dsc file # If you already have the .dsc, .diff.gz, and orig tarball downloaded # to the current directory, then skip this step. dget http://path/to/file.dsc git-import-dsc --debian-branch=debian -.dsc cd }}} These last two steps create a subdirectory named after the package. The subdirectory has the complete source, including the {{{./debian}}} directory. The original tarball (without {{{./debian}}}) is in the "upstream" branch, and the original stuff plus Debian changes would be in the "debian" branch, and a copy of the contents of the "debian" branch is placed in the "master" branch. You will be in the "master" branch now. Make your HCoop-specific changes (preferably in an incremental and atomic fashion) and commit them using git. === hcoopifying the debian package === 1. Open {{{debian/changelog}}} in emacs and invoke {{{M-x debian-changelog-mode}}}. 1. Press {{{C-c C-v}}} to create a new entry in the changelog and set the version so that the portion after the dash is {{{hcoop1}}} (or {{{hcoop2}}}, etc). 1. Change the distribution to $stable-backports (as of 2012, this is `squeeze-backports`) 1. Add a comment 1. Press {{{C-c C-c}}} to close the entry. 1. Save and exit. == New package from Debian == When a new Debian package comes out, and we want to incorporate their changes, the routine will be as follows. * is the name of the package. * is the upstream version of the software. * is the patch level of the package. For example: "1". We always add an "hcoop" suffix to patch levels of packages that we modify. {{{ cd .. dget -x cd git checkout debian cd ../- GIT_DIR=..//.git git add . GIT_DIR=..//.git git add -u . GIT_DIR=..//.git git commit -m "Import Debian package -" cd ../ rm -fr ../- git clean -d && git reset --hard git tag -a -m "Debian release -" debian/- }}} OK, let's stop to take a breather. Here's what the effect of all of this stuff has been. * Retrieved a .dsc file from debian and expanded it into the - directory. * Switched to the debian branch. * Moved into the new directory. * Set GIT_DIR, which specifies the git metadata directory to use. * Added all untracked files. * Updated all tracked files. * Committed the result. Now the git metadata directory knows about this Debian package version. * Removed the - directory. * Reset the working directory to match the metadata, so that we get the latest changes that we just committed. * Tagged this particular state with the version of the package. Now we'll want to switch back to the master branch (where we keep HCoop-specific changes) and merge the latest Debian changes. {{{ git checkout master git merge debian [fix any conflicts, particularly in debian/changelog] git commit }}} Now, make a new debian/changelog entry and list the changes that were kept in our version. When done, commit, build packages, and tag the version of the package as in the '''Building a Package''' section. == New upstream version not yet in Debian == If you want to update an existing custom HCoop Debian package with a new version of the upstream program, and no Debian package yet exists for that version, then you'll need to work with the upstream tarball for the new release directly. Instructions are as follows. * Make a directory for the new version. {{{ cd /afs/hcoop.net/common/debian/ mkdir cd }}} * Download the new upstream tarball to this directory. * Rename it to {{{_.orig.tar.gz}}}. * Move the git repo for the old version over to the new directory. {{{ mv ..// .}}} * Run git-import-orig. {{{ cd git-import-orig ../_.orig.tar.gz}}} * Resolve conflicts and built the new package. When Debian catches up to our blazing pace and makes their own package, perhaps with changes that we want, then we will need to use some trickery to make the packages sync up. * Change directory to {{{/afs/hcoop.net/common/debian//}}}. * Obtain the debian .dsc file and extract the contents to -, as in '''New package from Debian''' section. * Switch to the {{{debian}}} branch. {{{ cd git checkout debian}}} * Check in Debian's changes. {{{ cd ../-ver GIT_DIR=..//.git git add . GIT_DIR=..//.git git add -u . GIT_DIR=..//.git git commit -m "Import Debian package -" cd ../ git add . ; git reset --hard }}} * Do an "ours" merge with the {{{upstream}}} branch. This basically does a merge that is guaranteed not to have conflicts, with the end result being the contents of the current branch. This allows us to more easily merge in the changes that Debian made, later on. {{{ git merge -s ours upstream}}} * For instructive purposes, do a {{{git log}}}. You will see a log entry for the upstream version just below the log entry for the new Debian package. Very nifty. * Now switch back to the {{{master}}} branch which contains our changes and merge from the {{{debian}}} branch. {{{ git checkout master git merge debian }}} * Resolve any conflicts. You shouldn't see conflicts in the upstream source -- only the {{{debian/}}} directory might have conflicts. * Build and tag the package, making a new HCoop version. = Debian Archive = * Using debarchiver on hopper (we want to run as little as possible on fritz) * `/afs/hcoop.net/common/debian/...` * `.../old/` = current contents (obsolete package sources / builds) * `.../src/` * `hcoop/` our custom packages (`hcoop-$foo-config` and `libnss-afs`) * `backport/` manually backported packages (ideally, this contains nothing) * `.../archive/` = debarchiver * `/afs/hcoop.net/debian/archive/` is exported as http://debian.hcoop.net/ * Using `debuild` on fritz for `libnss-afs` and personal machines for config file debs for now (only going to package the `amd64` version of `libnss-afs` (for now) and arch independent config file debs) * TODO: Dedicate a (local LAN only, probably safe to put behind the libvirt nat) KVM to buildd / sbuild or figure out how to make pbuilder use libvirt == Installing Packages to the Archive == `debarchiver` is configured to scan `/afs/hcoop.net/common/debian/archive/incoming/` every five minutes. The easiest way to install a package to the archive is to use `dput` on the `.changes` file. Example `~/.dput.cf`: {{{ [hcoop] fqdn = local method = local incoming = /afs/hcoop.net/common/debian/archive/incoming/ allow_unsigned_uploads = 1 }}} == sources.list == Preseeding should take care of setting up the repository, but: `/etc/apt/sources.list` should contain: {{{ deb http://debian.hcoop.net/ squeeze main hcoop-config deb-src http://debian.hcoop.net/ squeeze main hcoop-config }}}