RFE: 'cloudsmith push deb' parse .dsc `Files:` section for sources/changes

[nickbroon]

Currently to upload a debian source package:

cloudsmith push deb <repo/dist/version> foo_1.0.0-1.dsc --sources-file foo_1.0.0.orig.tar.bz2 --changes-file foo_1.0.0-1.debian.tar.xz

When scripting the CLI this means detecting/determining the sources/changes files (and they can have various suffix, sources: .orig.tar.gz, .orig.tar.bz2, changes: .debian.tar.xz, .diff.gz , etc).

The files that make up the source package are listed in the 'Files:' section of the .dsc file itself. https://manpages.debian.org/testing/dpkg-dev/dsc.5.en.html

It would be useful if 'cloudsmith push deb' when passed only a .dsc file could parse the 'Files:' to determine the additional sources/changes files it should upload.

If it helps, the https://pypi.org/project/pydpkg/ python library can parse .dsc files:

>>> dsc.source_files
['/tmp/testdeb_0.0.0.orig.tar.gz',
 '/tmp/testdeb_0.0.0-1.debian.tar.xz',
 '/tmp/testdeb_0.0.0.dsc' ]
 >>> dsc.all_files_present
True

[nickbroon]

A work around for this using dcmd would be:

    DEBTAR=$(dcmd -r --debtar "$DSC")
    cloudsmith push deb ${REPO}/any-distro/any-version "$DSC" --sources-file=$(dcmd --orig "$DSC") ${DEBTAR:+--changes-file="${DEBTAR}"}

However having cloudsmith push deb... accept a .dsc or .changes files and then use 'Files:' section in those, in much similar way to dput would more robust/flexible than the above shell snippet.

[nickbroon]

Another reason it would also be useful if cloudsmith push could parse the .dsc and upload all the files listed as part of the 'debian source package` is that debian source packages can consist of more than just a single source-file and single changes-files. It can have several sources-file components and also .asc signature files.

While the additional component sources files and asc signatures files are rare in practise, if the former are present the source package simply cannot be uploaded to CloudSmith, but for later they can be ignored and not uploaded.

https://manpages.debian.org/buster/dpkg-dev/dpkg-source.1.en.html#Format:3.0(quilt)

Format: 3.0 (quilt)
Supported since dpkg 1.14.17. A source package in this format contains at least an original tarball (.orig.tar.ext where ext can be gz, bz2, lzma and xz) and a debian tarball (.debian.tar.ext). It can also contain additional original tarballs (.orig-component.tar.ext). component can only contain alphanumeric (‘a-zA-Z0-9’) characters and hyphens (‘-’). Optionally each original tarball can be accompanied by a detached upstream signature (.orig.tar.ext.asc and .orig-component.tar.ext.asc), extraction supported since dpkg 1.17.20, building supported since dpkg 1.18.5.

[nickbroon]

Workaround, that includes warning/errors for unsupported cases now looks like:

$ cat cloudsmith_upload_debian_source_package.sh

#!/bin/bash

UPLOAD_TARGET=$1
DSC=$2
OTHER_ARGS=("${@:3}")

# Determine source file to upload
SOURCES=$(dcmd --orig "$DSC")
for SOURCE in $SOURCES
do
    if [[ "${SOURCE}" =~ \.asc$ ]]; then
        echo "WARNING: upstream signatures ($SOURCE) not supported by CloudSmith. Ignoring."
    elif [[ "${SOURCE}" =~ .*\.orig-[a-zA-Z0-9\-]*\. ]]; then
        echo "ERROR: additional upstream component sources ($SOURCE) not supported by CloudSmith"
        exit 1
    else
        # Assume it's the source to upload
        UPLOAD_SOURCE="${SOURCE}"
    fi
done

if [[ -z "${UPLOAD_SOURCE}" ]]; then
    echo "ERROR: Could not determine source file"
    exit 2
fi

# Determine optional changes file, only present for 'non-native' packages
DEBTAR=$(dcmd -r --debtar "$DSC")

cloudsmith upload deb "${UPLOAD_TARGET}" "$DSC" \
"--sources-file=${UPLOAD_SOURCE}" \
${DEBTAR:+--changes-file="${DEBTAR}"} \
"${OTHER_ARGS[@]}"

[lskillen]

This is awesome work, thank you @nickbroon!

Food for thought for the engineering team over here:

In order to support this on the client-side we'd likely need to do a bit of rearchitecting work for the CLI. Right now the package formats are programmatically driven by the API, in the sense that the CLI auto-generates the parameters for push.

However, it means that we can't (or don't) yet do any custom client-side logic per package format. So the CLI would need some form of "extensions" to understand that for "cloudsmith push deb", there's custom logic to apply for some of the parameters.

[nickbroon]

What would be even more amazing would be if CloudSmith were to implement the web endpoints needed to use the common native Debian dput tool for uploading packages (in a similar way that CloudSmith has web end points for native uploading some other package types it support, ie, cargo, Conan, docker, maven, npm, ruby)! :-)

cat <<EOF > dput.cf
[cloudsmith]
method = https
fqdn = <username>:<your_access_token>@api.cloudsmith.io
incoming = v1/packages/<owner>/<repo>/upload/deb/           <----- This looks like needs to directory path to which files can be directly PUT
EOF

dput --config=dput.cf --unchecked --no-upload-log cloudsmith <your_package>.changes

dput docs here:

• https://manpages.debian.org/bullseye/dput-ng/dput.1.en.html
• https://salsa.debian.org/debian/dput-ng

GitLab Package Registries has experimental support for this (Self hosted only, not Cloud) that might be a useful reference:

• https://docs.gitlab.com/ee/user/packages/debian_repository/#publish-a-package
• https://docs.gitlab.com/ee/api/packages/debian.html#upload-a-package-file