|
Size: 6295
Comment: Document the handling of new upstream versions.
|
Size: 13735
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 3: | Line 3: |
| [[TableOfContents()]] = Making a new custom package = |
<<TableOfContents>> = Overview = The idea is to keep track of each custom HCoop Debian package using three branches, which are as follows. 1. {{{upstream}}}: The source code from the current release of the upstream software. 1. {{{debian}}}: The source code plus the latest Debian packaging that Debian has for the software. 1. {{{master}}}: The source code plus the latest Debian packaging plus any changes that HCoop has made to the source or the packaging. If you are creating a native package (e.g. for configuration files) then you only have a `master` branch. = Developing Packages = Common to all of the types of packages we might develop. HCoop is standardized on all amd64 packages, aside from architecture independent packages. == Setting Up Environment For Clean Builds == Packages must be signed to be accepted, in `~/.devscripts` make sure your signing key is set: {{{ DEBSIGN_KEYID=YOUR_KEYID_HERE }}} TODO: pbuilder [[https://wiki.debian.org/PbuilderTricks#How_to_build_for_different_distributions|Set up pbuilder for each distribution and architecture]]. We build with backports and the hcoop repository available. Example: {{{ DIST=stretch-backports ARCH=amd64 git-pbuilder create DIST=stretch-backports ARCH=amd64 git-pbuilder login --save-after-login # add hcoop repos to sources.list and save system }}} == Building a package == Years ago HCoop standardized on Git for VersionControl; as such we're using [[http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html|git-buildpackage]] to maintain our packages. First, make sure you are on the "master" branch by running: {{{ git branch -l }}} If you see an asterisk by "master", you're on the right branch. If we want to build the package with some uncommitted changes, as a sanity check, then do: {{{ gbp buildpackage --git-pbuilder --git-ignore-new --git-dist=$dist-backports --git-arch=amd64 }}} When it comes time to test the changes, build the package using: {{{ gbp buildpackage --git-pbuilder --git-dist=$dist-backports --git-arch=amd64 --git-export-dir=$tmpdir }}} The packages will be built and placed in the temporary directory you specify. You have to use a directory not in afs, because pbuilder runs using `sudo` and will not have your tokens. To indicate that we are done making changes to this particular version of the Debian package, tag it with: {{{ gbp dch --release gbp buildpackage --git-tag --git-pbuilder --git-dist=$dist-backports --git-arch=amd64 --git-export-dir=/tmp/pbuild }}} This makes the package version show up when you do {{{git tag -l}}}, for easy diffing and viewing. == New Packages == After creating the git-buildpackage repository, push it to the public HCoop debian packages git area: {{{ gbp-create-remote-repo --remote-url-pattern=/afs/hcoop.net/user/h/hc/hcoop/.hcoop-git/debian/'%(pkg)s'.git }}} We may revisit only having one area for Debian packages at a later time. = Forking a Debian Package = If a package is available in the official backports, use it. If you need to backport something not backported, make a sloppy backport from testing/unstable to stable/oldstable, or must make changes for afs and kerberos support, read on. == Making a new custom package == |
| Line 11: | Line 94: |
| mkdir -p /afs/hcoop.net/common/debian/<pkg>/<upstream-version> cd /afs/hcoop.net/common/debian/<pkg>/<upstream-version> |
mkdir -p /afs/hcoop.net/common/debian/src/{backports,fork}/<pkg> cd /afs/hcoop.net/common/debian/src/{backports,fork}/<pkg> |
| Line 14: | Line 97: |
| dget -x http://path/to/file.dsc # Or if you already have the .dsc, .diff.gz, and orig tarball downloaded, # then do dpkg-source -x path/to/file.dsc instead. git-import-dsc --debian-branch=debian <pkg>-<ver>.dsc }}} This last step creates a subdirectory named after the package. The subdirectory |
# If you already have the .dsc, .diff.gz, and orig tarball downloaded # to the current directory, then skip this step. gbp import-dsc --debian-branch=debian --upstream-branch=upstream http://path/to/file.dsc cd <pkg> }}} These last two steps create a subdirectory named after the package. The subdirectory |
| Line 25: | Line 108: |
| be in the "master" branch now. | be in the "master" branch now. If you are not, create it with `git checkout -b master` |
| Line 28: | Line 111: |
| fashion) and commit them using git. = Building a package = First, make sure you are on the "master" branch by running: {{{ git branch -l }}} If you see an asterisk by "master", you're on the right branch. If we want to build the package with some uncommitted changes, as a sanity check, then do: {{{ git-buildpackage --git-ignore-new }}} When it comes time to test the changes, build the package using: {{{ git-buildpackage }}} The packages will be built and placed in the parent directory. To indicate that we are done making changes to this particular version of the Debian package, tag it with: {{{ git-buildpackage --git-tag }}} This makes the package version show up when you do {{{git tag -l}}}, for easy diffing and viewing. = New package from Debian = |
fashion) and commit them using git. You may want to use quilt and commit the quilt patches instead if the package uses quilt. === hcoopifying the debian package === 1. Open {{{debian/changelog}}} in emacs and invoke {{{M-x debian-changelog-mode}}}. 1. Press {{{C-c C-v}}} to create a new entry in the changelog and append `+hcoopN` (where `N` is the hcoop revision) to the version. E.g. `0.60.0-3` become `0.60.0-3+hcoop1` 1. If it is a backport, change the distribution to $stable-backports (as of 2015, this is `jessie-backports`). The version should also have [[http://backports.debian.org/news/jessie_released_-_backports_related_changes/|~bpo8+hcoopN]] for jessie, `~bpo70+hcoopN` for wheezy (`~bpo7+hcoopN` for a sloppy backport), or `~bpo60+hcoopN` for squeeze appended to conform to standard backports versioning. 1. Add a comment 1. Press {{{C-c C-c}}} to close the entry. 1. Save and exit. Alternatively, you can use `git-dch` for this task if you ensure that your git commits work as debian changelog entries. == New package from Debian == |
| Line 74: | Line 134: |
| cd .. dget -x <URL to dsc file on packages.debian.org> |
|
| Line 77: | Line 135: |
| git checkout debian cd ../<pkgname>-<ver> GIT_DIR=../<pkgname>/.git git add . GIT_DIR=../<pkgname>/.git git add -u . GIT_DIR=../<pkgname>/.git git commit -m "Import Debian package <ver>-<patch>" cd <pkgname> rm -fr ../<pkgname>-<ver> git add . ; git reset --hard git tag -a -m "Debian release <ver>-<patch>" debian/<ver>-<patch> }}} OK, let's stop to take a breather. Here's what the effect of all of this stuff has been. * Retrieved a .dsc file from debian and expanded it into the <pkgname>-<ver> directory. * Switched to the debian branch. * Moved into the new directory. * Set GIT_DIR, which specifies the git metadata directory to use. * Added all untracked files. * Updated all tracked files. * Committed the result. Now the git metadata directory knows about this Debian package version. * Removed the <pkgname>-<ver> directory. * Reset the working directory to match the metadata, so that we get the latest changes that we just committed. * Tagged this particular state with the version of the package. |
gbp import-dsc --debian-branch=debian http://path/to/file.dsc }}} `git-import-dsc` should do the right thing. |
| Line 112: | Line 151: |
| = New upstream version not yet in Debian etch = | == New upstream version not yet in Debian == {{{#!wiki caution This section needs decrufting and may produce unexpected results. It also makes it difficult for the package for sync with Debian again in the future. }}} |
| Line 153: | Line 196: |
= Debian Archive = * Using debarchiver on gibran * Configuration is managed in Puppet class `hcoop::service::debarchiver` * `/afs/hcoop.net/common/debian/...` * `.../old/` = current contents (obsolete package sources / builds) * `.../src/` * `hcoop/` our custom packages (`hcoop-$foo-config` and `libnss-afs`) * `backport/` manually backported packages (ideally, this contains only a few packages) * `fork/` manually forked packages (ideally, this contains nothing) * `.../archive/` = debarchiver * `/afs/hcoop.net/debian/archive/` is exported as http://debian.hcoop.net/ * Packages are built using `git-pbuilder` for all arch/dist combinations hcoop must support at the moment == Debian Archive Signing == Our apt repository requires signed uploads and releases are signed. === Upload Signing Keys === ==== Generating An Upload Signing Key ==== Generate the key on your local machine, where you will be running pbuilder/uploading from, with: {{{ gpg --full-gen-key }}} Keys used by admins to sign uploads should have the following attributes: * Key Type: RSA and RSA * Key Length: 4096 bits * Expiration: 1y * Name: `YOUR NAME` (HCoop Debian Archive Upload Signing Key) * Email: `you`_admin@hcoop.net * Comment: `CURRENT_YEAR` Ensure the keyid is set in `~/.devscripts` so debsign will sign uploads with the correct key: {{{ DEBSIGN_KEYID=YOUR_KEYID_HERE }}} ==== Importing a New Upload Key ==== Export the key that will be used to sign uploads {{{ gpg --armor --export YOUR_KEYID_HERE }}} Copy the exported key to the debarchiver server, and import it: {{{ sudo -u debarchiver gpg --no-default-keyring --keyring uploaders.gpg --import YOUR_KEYFILE_HERE }}} If managed using Puppet, enter the hcoop private data repository for gnupg and run as root: {{{ gpg --no-default-keyring --keyring /path/to/private/puppet/module/files/debarchiver/gnupg/uploaders.gpg --import YOUR_KEYFILE_HERE }}} === Archive Key === The Debian archive is signed, and the signing key should be rotated every year (currently February 2nd). The keyring is managed by Puppet, and is '''not''' committed to git. GPG has weird restrictions on the length of the agent socket filename, so you may need to symlink the directory into /root to work around them. You will also need to reset the default mask and we are using a POSIX ACL, and GPG removes the mask bits during key generation, negating all ACLs. As root: {{{ ln -s /path/to/private/puppet/module/files/debarchiver/gnupg/ /root/debarchiver-gnupg gpg --homedir /root/debarchiver-gnupg --full-gen-key setfacl -m m::rwx -R /root/debarchiver-gnupg/ rm /root/debarchiver-gnupg }}} * Key Type: RSA and RSA * Key Length: 4096 bits * Expiration: 1y * Name: HCoop Debian Archive Signing Key * Email: admins@hcoop.net * Comment: `CURRENT_YEAR` After generating, run `sudo debarchiver gpg --list-secret-keys` and copy the keyid of the private key that was generated to the debarchiver config option `$gpgkey`. == Installing Packages to the Archive == `debarchiver` is configured to scan `/afs/hcoop.net/common/debian/archive/incoming/$dists` every five minutes. The easiest way to install a package to the archive is to use `dput` on the `.changes` file. By uploading to a `distinputdir`, you can leave the distribution as `unstable` in the changelog, and upload a package to multiple releases. The package should be built using pbuilder for each target release, and the source tarballs must match. You can to upload packages for backports into a distinput directory, but you still have to update the version in changelog. Example `~/.dput.cf`: {{{ [hcoop-stretch] fqdn = local method = local incoming = /afs/hcoop.net/common/debian/archive/incoming/stretch/ [hcoop-stretch-backports] fqdn = local method = local incoming = /afs/hcoop.net/common/debian/archive/incoming/stretch_backports/ [hcoop] fqdn = local method = local incoming = /afs/hcoop.net/common/debian/archive/incoming }}} To upload a new package, * sign the changes file: `debsign PACKAGE.changes` * upload with dput: `dput RELEASE PACKAGE.changes`, for example to upload exim 4.89-2+deb9u6~hcoop11 to stretch: `put hcoop-stretch exim4_4.89-2+deb9u6~hcoop11_amd64.changes` == Checking for new versions == Many packages supply [[https://wiki.debian.org/debian/watch/][debian/watch]] files which allow for easy scanning of new upstream versions. Run `uscan /afs/hcoop.net/common/debian/src/` occasionally to scan for new upstream versions. ---- CategorySystemAdministration |
This page describes how to make custom Debian packages for HCoop.
1. Overview
The idea is to keep track of each custom HCoop Debian package using three branches, which are as follows.
upstream: The source code from the current release of the upstream software.
debian: The source code plus the latest Debian packaging that Debian has for the software.
master: The source code plus the latest Debian packaging plus any changes that HCoop has made to the source or the packaging.
If you are creating a native package (e.g. for configuration files) then you only have a master branch.
2. Developing Packages
Common to all of the types of packages we might develop.
HCoop is standardized on all amd64 packages, aside from architecture independent packages.
2.1. Setting Up Environment For Clean Builds
Packages must be signed to be accepted, in ~/.devscripts make sure your signing key is set:
DEBSIGN_KEYID=YOUR_KEYID_HERE
TODO: pbuilder
Set up pbuilder for each distribution and architecture. We build with backports and the hcoop repository available. Example:
DIST=stretch-backports ARCH=amd64 git-pbuilder create DIST=stretch-backports ARCH=amd64 git-pbuilder login --save-after-login # add hcoop repos to sources.list and save system
2.2. Building a package
Years ago HCoop standardized on Git for VersionControl; as such we're using git-buildpackage to maintain our packages.
First, make sure you are on the "master" branch by running:
git branch -l
If you see an asterisk by "master", you're on the right branch.
If we want to build the package with some uncommitted changes, as a sanity check, then do:
gbp buildpackage --git-pbuilder --git-ignore-new --git-dist=$dist-backports --git-arch=amd64
When it comes time to test the changes, build the package using:
gbp buildpackage --git-pbuilder --git-dist=$dist-backports --git-arch=amd64 --git-export-dir=$tmpdir
The packages will be built and placed in the temporary directory you specify. You have to use a directory not in afs, because pbuilder runs using sudo and will not have your tokens. To indicate that we are done making changes to this particular version of the Debian package, tag it with:
gbp dch --release gbp buildpackage --git-tag --git-pbuilder --git-dist=$dist-backports --git-arch=amd64 --git-export-dir=/tmp/pbuild
This makes the package version show up when you do git tag -l, for easy diffing and viewing.
2.3. New Packages
After creating the git-buildpackage repository, push it to the public HCoop debian packages git area:
gbp-create-remote-repo --remote-url-pattern=/afs/hcoop.net/user/h/hc/hcoop/.hcoop-git/debian/'%(pkg)s'.git
We may revisit only having one area for Debian packages at a later time.
3. Forking a Debian Package
If a package is available in the official backports, use it. If you need to backport something not backported, make a sloppy backport from testing/unstable to stable/oldstable, or must make changes for afs and kerberos support, read on.
3.1. Making a new custom package
If you want to make changes to an existing Debian package, and we haven't made our own custom package before, then do the following.
mkdir -p /afs/hcoop.net/common/debian/src/{backports,fork}/<pkg>
cd /afs/hcoop.net/common/debian/src/{backports,fork}/<pkg>
# Browse http://packages.debian.org/<pkg> and find a link to a dsc file
# If you already have the .dsc, .diff.gz, and orig tarball downloaded
# to the current directory, then skip this step.
gbp import-dsc --debian-branch=debian --upstream-branch=upstream http://path/to/file.dsc
cd <pkg>These last two steps create a subdirectory named after the package. The subdirectory has the complete source, including the ./debian directory. The original tarball (without ./debian) is in the "upstream" branch, and the original stuff plus Debian changes would be in the "debian" branch, and a copy of the contents of the "debian" branch is placed in the "master" branch. You will be in the "master" branch now. If you are not, create it with git checkout -b master
Make your HCoop-specific changes (preferably in an incremental and atomic fashion) and commit them using git. You may want to use quilt and commit the quilt patches instead if the package uses quilt.
3.1.1. hcoopifying the debian package
Open debian/changelog in emacs and invoke M-x debian-changelog-mode.
Press C-c C-v to create a new entry in the changelog and append +hcoopN (where N is the hcoop revision) to the version. E.g. 0.60.0-3 become 0.60.0-3+hcoop1
If it is a backport, change the distribution to $stable-backports (as of 2015, this is jessie-backports). The version should also have ~bpo8+hcoopN for jessie, ~bpo70+hcoopN for wheezy (~bpo7+hcoopN for a sloppy backport), or ~bpo60+hcoopN for squeeze appended to conform to standard backports versioning.
- Add a comment
Press C-c C-c to close the entry.
- Save and exit.
Alternatively, you can use git-dch for this task if you ensure that your git commits work as debian changelog entries.
3.2. New package from Debian
When a new Debian package comes out, and we want to incorporate their changes, the routine will be as follows.
<pkgname> is the name of the package.
<ver> is the upstream version of the software.
<patch> is the patch level of the package. For example: "1". We always add an "hcoop" suffix to patch levels of packages that we modify.
cd <pkgname> gbp import-dsc --debian-branch=debian http://path/to/file.dsc
git-import-dsc should do the right thing.
Now we'll want to switch back to the master branch (where we keep HCoop-specific changes) and merge the latest Debian changes.
git checkout master git merge debian [fix any conflicts, particularly in debian/changelog] git commit
Now, make a new debian/changelog entry and list the changes that were kept in our version. When done, commit, build packages, and tag the version of the package as in the Building a Package section.
3.3. New upstream version not yet in Debian
This section needs decrufting and may produce unexpected results. It also makes it difficult for the package for sync with Debian again in the future.
If you want to update an existing custom HCoop Debian package with a new version of the upstream program, and no Debian package yet exists for that version, then you'll need to work with the upstream tarball for the new release directly. Instructions are as follows.
Make a directory for the new version.
cd /afs/hcoop.net/common/debian/<pkgname> mkdir <ver> cd <ver>
- Download the new upstream tarball to this directory.
Rename it to <pkgname>_<ver>.orig.tar.gz.
Move the git repo for the old version over to the new directory.
mv ../<old-ver>/<pkgname> .
Run git-import-orig.
cd <pkgname> git-import-orig ../<pkgname>_<ver>.orig.tar.gz
- Resolve conflicts and built the new package.
When Debian catches up to our blazing pace and makes their own package, perhaps with changes that we want, then we will need to use some trickery to make the packages sync up.
Change directory to /afs/hcoop.net/common/debian/<pkgname>/<ver>.
Obtain the debian .dsc file and extract the contents to <pkgname>-<ver>, as in New package from Debian section.
Switch to the debian branch.
cd <pkgname> git checkout debian
Check in Debian's changes.
cd ../<pkgname>-ver GIT_DIR=../<pkgname>/.git git add . GIT_DIR=../<pkgname>/.git git add -u . GIT_DIR=../<pkgname>/.git git commit -m "Import Debian package <ver>-<patch>" cd ../<pkgname> git add . ; git reset --hard
Do an "ours" merge with the upstream branch. This basically does a merge that is guaranteed not to have conflicts, with the end result being the contents of the current branch. This allows us to more easily merge in the changes that Debian made, later on.
git merge -s ours upstream
For instructive purposes, do a git log. You will see a log entry for the upstream version just below the log entry for the new Debian package. Very nifty.
Now switch back to the master branch which contains our changes and merge from the debian branch.
git checkout master git merge debian
Resolve any conflicts. You shouldn't see conflicts in the upstream source -- only the debian/ directory might have conflicts.
- Build and tag the package, making a new HCoop version.
4. Debian Archive
- Using debarchiver on gibran
Configuration is managed in Puppet class hcoop::service::debarchiver
/afs/hcoop.net/common/debian/...
.../old/ = current contents (obsolete package sources / builds)
.../src/
hcoop/ our custom packages (hcoop-$foo-config and libnss-afs)
backport/ manually backported packages (ideally, this contains only a few packages)
fork/ manually forked packages (ideally, this contains nothing)
.../archive/ = debarchiver
/afs/hcoop.net/debian/archive/ is exported as http://debian.hcoop.net/
Packages are built using git-pbuilder for all arch/dist combinations hcoop must support at the moment
4.1. Debian Archive Signing
Our apt repository requires signed uploads and releases are signed.
4.1.1. Upload Signing Keys
4.1.1.1. Generating An Upload Signing Key
Generate the key on your local machine, where you will be running pbuilder/uploading from, with:
gpg --full-gen-key
Keys used by admins to sign uploads should have the following attributes:
- Key Type: RSA and RSA
- Key Length: 4096 bits
- Expiration: 1y
Name: YOUR NAME (HCoop Debian Archive Upload Signing Key)
Email: you_admin@hcoop.net
Comment: CURRENT_YEAR
Ensure the keyid is set in ~/.devscripts so debsign will sign uploads with the correct key:
DEBSIGN_KEYID=YOUR_KEYID_HERE
4.1.1.2. Importing a New Upload Key
Export the key that will be used to sign uploads
gpg --armor --export YOUR_KEYID_HERE
Copy the exported key to the debarchiver server, and import it:
sudo -u debarchiver gpg --no-default-keyring --keyring uploaders.gpg --import YOUR_KEYFILE_HERE
If managed using Puppet, enter the hcoop private data repository for gnupg and run as root:
gpg --no-default-keyring --keyring /path/to/private/puppet/module/files/debarchiver/gnupg/uploaders.gpg --import YOUR_KEYFILE_HERE
4.1.2. Archive Key
The Debian archive is signed, and the signing key should be rotated every year (currently February 2nd). The keyring is managed by Puppet, and is not committed to git. GPG has weird restrictions on the length of the agent socket filename, so you may need to symlink the directory into /root to work around them. You will also need to reset the default mask and we are using a POSIX ACL, and GPG removes the mask bits during key generation, negating all ACLs. As root:
ln -s /path/to/private/puppet/module/files/debarchiver/gnupg/ /root/debarchiver-gnupg gpg --homedir /root/debarchiver-gnupg --full-gen-key setfacl -m m::rwx -R /root/debarchiver-gnupg/ rm /root/debarchiver-gnupg
- Key Type: RSA and RSA
- Key Length: 4096 bits
- Expiration: 1y
- Name: HCoop Debian Archive Signing Key
Email: admins@hcoop.net
Comment: CURRENT_YEAR
After generating, run sudo debarchiver gpg --list-secret-keys and copy the keyid of the private key that was generated to the debarchiver config option $gpgkey.
4.2. Installing Packages to the Archive
debarchiver is configured to scan /afs/hcoop.net/common/debian/archive/incoming/$dists every five minutes. The easiest way to install a package to the archive is to use dput on the .changes file. By uploading to a distinputdir, you can leave the distribution as unstable in the changelog, and upload a package to multiple releases. The package should be built using pbuilder for each target release, and the source tarballs must match.
You can to upload packages for backports into a distinput directory, but you still have to update the version in changelog.
Example ~/.dput.cf:
[hcoop-stretch] fqdn = local method = local incoming = /afs/hcoop.net/common/debian/archive/incoming/stretch/ [hcoop-stretch-backports] fqdn = local method = local incoming = /afs/hcoop.net/common/debian/archive/incoming/stretch_backports/ [hcoop] fqdn = local method = local incoming = /afs/hcoop.net/common/debian/archive/incoming
To upload a new package,
sign the changes file: debsign PACKAGE.changes
upload with dput: dput RELEASE PACKAGE.changes, for example to upload exim 4.89-2+deb9u6~hcoop11 to stretch: put hcoop-stretch exim4_4.89-2+deb9u6~hcoop11_amd64.changes
4.3. Checking for new versions
Many packages supply https://wiki.debian.org/debian/watch/][debian/watch files which allow for easy scanning of new upstream versions. Run uscan /afs/hcoop.net/common/debian/src/ occasionally to scan for new upstream versions.