AVRO-3423: Add build.sh step to encapsulate all the steps needed during release

[zcsizmadia]

Make sure you have checked all steps below.

Jira

• [X] My PR addresses the following [Avro Jira](https://issues.apache.org/jira/browse/AVRO-3423
  • https://issues.apache.org/jira/browse/AVRO-3423
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

• [X] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• [X] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• [X] In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain Javadoc that explain what it does

[zcsizmadia]

@martin-g @RyanSkraba

This is a proposed way to simplify the release process for the C#. Step #9 in https://cwiki.apache.org/confluence/display/AVRO/How+To+Release#HowToRelease-Publishing could be reduced to the following if running from the ../lang/csharp folder:

NUGET_KEY="YOUR_KEY" ./build.sh release

This step requires that ./build.sh dist was executed earlier. That requirement can be eliminated by running dotnet pack --configuration Release Avro.sln as the first thing in release, however the dist step might be already done at the time when release is happening.

Other languages might follow this pattern so the release step would become a series of ./build.sh release from all language.

[zcsizmadia]

@martin-g I can check if the tar.gz exists and all the needed nupkgs exist, if not execute the dist() function as you described above. I have it implemented locally and I can commit it.

However, what about if there is no release action, but RELEASE=true NUGET_KEY="YOUR_KEY" ./build.sh dist needed to be executed:

  dist)
    ...
    if "$RELEASE"
    then
        ...
        # Bunch of nuget push ...
    fi
    ;;

[martin-g]

what about if there is no release action, but RELEASE=true NUGET_KEY="YOUR_KEY" ./build.sh dist

I like it!

[zcsizmadia]

This runs dist step and pushes the nupkgs to nuget.org:

RELEASE=1 NUGET_KEY="YOUR_KEY" ./build.sh dist

./build.sh dist is the same as it was before.

Is there any place where the NUGET_KEY is stored already on the "release" machine? SO I can default the value to it if NUGET_KEY is not set.

[zcsizmadia]

I still like the a new release step.This is another alternative to combine dist and release:

  dist|release)
     # dist steps
      ...
      if [ "$target" == "release" ]
      then
        # Push packages
        ...
      fi

[zcsizmadia]

Added a verify-release step to simplify and harden the verisifcation step after the release. It iterates through all the supported dotnet code SDKS to make sure that all the libs and tool can be installed from nuget for all the SDKs.

[RyanSkraba]

I really like one advantage of this -- I have tons of notes (in addition to the wiki) for releasing binary artifacts and doing the equivalent of verify-release. I've always intended to clean my notes up and contribute them back, and it would be really neat to do that in the build scripts themselves.

Unfortunately, I'm kind of skittish about delegating these steps to a script and generalizing them to all languages... I'd find it hard to "trust" doing a ./build.sh release from the root and have all of the binary artifacts deployed -- especially if there were failures midway or something unexpected happens (it always does).

Note that the Wiki process isn't immune to that either (check out the Python SDK release 1.9.2.1 as an example of fixing a bad deploy!)

I don't have any good ideas for that other than prompting (y/n/skip) at each command! Any ideas?

[zcsizmadia]

Adding a step function might be a good solution to help out skittish users ;)

function step
{
    ask if want to run [y/skip/abort]
    if no, return
    if abort, exit shole script
    execute $1
    if error code == 0 -> green message: step complete
    if error code != 0 -> red message: step failed (maybe question for retry/abort?)
}

   release)
       step "dotnet nuget push A.B.C.nupkg"
       step "dotnet nuget push A.B.C.D.nupkg"
       ...

Btw, couls you post how the release is happening now? Are you executing the steps from the wiki page in blocks, or each line one by one from those code snipptes for each step?

[zcsizmadia]

@RyanSkraba Added a ask function to the script and usage to the verify-release step as an example. It asks for each SDKs how to proceed. If there is any error anytime, the script stops.

Verify .NET SDK 3.1 ([y]es/[n]o/([a]bort)? y
...
Verify .NET SDK 5.0 ([y]es/[n]o/([a]bort)? y
...

[RyanSkraba]

(Please excuse the delay! I've been playing catch-up with emails)

Btw, could you post how the release is happening now? Are you executing the steps from the wiki page in blocks, or each line one by one from those code snipptes for each step?

Effectively, I've been doing commands from the wiki page -- the commands that can be reverted or backed out in blocks and the commands that (permanently) publish artifacts one at a time. Usually one SDK at a time. Even if it's a bit fussy, it's really only about an hours work!

I like the ask function and if we generalize this process (release and verify-release) to other language SDKs, we should probably also generalize that! If one day we get to the state where we're completely automating releases, this will be welcome. We probably don't need to ask to verify, right? It's fine though as it is!

Perhaps a future feature would be to not ask but to echo out the command instead of executing it.

[zcsizmadia]

I added soem better command line argument parsing and option fro dry run, answer yes to all questions .... It is better to have a more standard way of handling wehn the number of options start to inscrease.

It is possible to put most of these build.sh framework into /share/build-helper script, and pretty much all languages just source /share/build-helper and implenet their own commands with functions like function command_test {}. The buidl helper would execute bt something like this command_$1 for the supported commands.

[zcsizmadia]

@RyanSkraba I added a build-helper.sh to share fodler as a POC. the c# build.sh is using that helper code. That shows kinda the template what to do in other languages. As you can see the helper handles all the command line parsing and just executes the functions defined by the language specific build sh files.

Couple of objectives I wanted the help build script to meet:

1. Support ask (yes/no/abort)
2. Support --yes to auto-ytes any questions
3. Dry run
4. Colored message functions to show green, yellow or red messages
5. If the build.sh is successfull (exit code=0) it shows a green "Done." message at the end of the script or "FAILED!" if exit code!=0. This would help to recognise easily if the script failed. fyi colors can be disabled manually or thautomatically if the script is not running on a terminal

I am not sure if language specific commands or options are needed, however that can be addedd as well, with some nice trickery

[RyanSkraba]

Excellent :heart: ! How much work do you want to put into this before merging? I was just brainstorming that one idea for the future ... but I can see this serving as a framework for other language SDKs.

[zcsizmadia]

I tried to do a brute force attempt to convert other languages. I dont think it is too bad. It is still WIP, however it is promising. The only 2 languages I did not toch are c and c++

My plan is to make everything backward compatible, with the current executions

[martin-g]

Awesome work, @zcsizmadia !

[zcsizmadia]

I saw that some language (PHP?) used a lib/avro/version.txt, instead of share/version.txt. Was that intentional?

The build-helper shell script initializes the BUILD_VERSION and BUILD_ROOT vars so the actual build script can use them directly. That is the reason many places ../.. were replaced by BUILD_ROOT

[zcsizmadia]

The initial version is pretty green. I assume most of the PR build checks are using build.sh for the languages? The dist commands must be tested that they generate the dist files in the proper place.

Could you guys help me out and test individual steps so it works on your side for languages you use?

[martin-g]

Just tested the Rust module! clean, lint, test, dist, interop-data-generate and interop-data-test work just fine!

Before I add an impl for release I wanted to ask whether it would be OK to rebase to latest master ?

[zcsizmadia]

@martin-g Merged master into branch

[zcsizmadia]

Added cd "$(dirname "$0")" to the top of the build.sh scripts. Previously it was inconsistent , some scripts had it, some did not. If the build.sh is executed from tha lang folder where the build.sh script is, it has no effect. If the build.sh script is executed from a different folder, it will change into the proper folder and make sure the scripts will work. E.g. lang/csharp/build.sh dist will work.

[zcsizmadia]

Modded the C build.sh script as well. Only C++ is left. That script might need some rework, since it has some "non-standard" targets, like doc and 'format' ...

[zcsizmadia]

@martin-g Should be able to provide the CARGO_API_TOKEN from the command line as well? --cargo-api-token? I was thinking to add support for language specific options to the build.sh script. This would perfectly fit there

[martin-g]

Env var should work too. I'm fine with both.

[martin-g]

@jklamer Please review https://github.com/apache/avro/pull/1570/commits/69772c3d234e2686d9722c10259612f9eb6e12db After some experimentation I concluded that cargo package cannot be used for Workspace project with Virtual manifests when a dependency (e.g. avro_derive) is not yet released. The .crate (the source package) can be created only after cargo publishing the modules/manifests.

[jklamer]

@martin-g will review soon. Currently on vacation but will get to it ASAP