Alternate backend support for config files

[tpickett66]

I'm working on a release management web app to manage branching and picking of commits onto release branches and push those changes back out to other services. Because multiple nodes in the app need to have a unified view of the repository I'm experimenting with non-filesystem storage technologies (specifically Postgres and Amazon's DynamoDB) for the workload. So far the Ref and Object DB backend support has worked exactly as expected but support for alternate config storage is missing. I notice there is backend support for configs in libgit2 similar to the Ref and Object DBs and am happy to start implementing support for this feature in Rugged. This issue is more to solicit feedback and start a discussion before I dig into adding this feature.

[arthurschreiber]

Hey @tpickett66 - that sounds exciting. It's always nice to hear how people are using rugged to build cool new stuff.

The backend support is... a bit weird. :disappointed:

I don't even remember the reason for why it was implemented the way it is implemented, but I'm currently thinking about rewriting it and make it easier to use and extend. Another part that is missing, as you correctly pointed out, is pluggable config backends. I promise I'll take a look this weekend and come up with some suggestions / improvements on the backend support.

[tpickett66]

Fantastic, thanks for giving your two cents. Since we aren't totally sure what storage technology we're ultimately going to use I've decided to take an approach that will allow us to rapidly prototype a couple different storage backends. Basically I'm building a bridge that translates between libgit2 types and native ruby types and calls into a ruby class to do the actual storage.

So far I've got refs reading a writing against an in memory store (just a ruby hash) and was moving on to configs so I could start building toward cloning one of the libgit2 fixture repos.

An outline of the RefDB backend I've got so far:

// rugged_bridge_backend_refdb.c
typedef struct {
  git_refdb_backend parent;

  char *repo_path;
  VALUE adapter;
} rugged_bridge_backend_refdb;

//Queries the refdb backend for a given reference. A refdb implementation
//must provide this function.
static int rugged_bridge_backend_refdb__lookup(
    git_reference **ref_out,
    git_refdb_backend *_backend,
    const char *ref_name)
{
  rugged_bridge_backend_refdb *backend;
  int status = GIT_OK;
  VALUE rb_repo_name;
  VALUE rb_ref_name;
  VALUE rb_ref;

  assert(ref_name && _backend);

  backend = (rugged_bridge_backend_refdb *) _backend;
  assert(backend->adapter);
  rb_repo_name = rb_str_new2(backend->repo_path);
  rb_ref_name = rb_str_new2(ref_name);

  rb_ref = rb_funcall(backend->adapter, rb_intern("lookup"), 2, rb_repo_name, rb_ref_name);

  if (NIL_P(rb_ref)) {
    giterr_set_str(GITERR_REFERENCE, "Bridge refdb couldn't find ref");
    status = GIT_ENOTFOUND;
  } else {
    rugged_bridge_rb_reference_to_git(ref_out, rb_ref);
  }

  return status;
}

// Writes a new reference to the DB
static int rugged_bridge_backend_refdb__write(
    git_refdb_backend *_backend,
    const git_reference *ref,
    int force,
    const git_signature *sig,
    const char *message,
    const git_oid *oid,
    const char *oid_target)
{
  int status = GIT_OK;
  rugged_bridge_backend_refdb *backend;
  VALUE rb_repo_name;
  VALUE rb_reference;
  VALUE rb_signature;
  VALUE rb_force = (force) ? Qtrue : Qfalse;
  VALUE result;

  /* We aren't handling the OID arguments yet, I honestly have no idea
  // how they're used so let's bring things crashing down if they're
  // supplied so we know we need to take a look. - tpickett 2015-10-01 */
  assert(oid == NULL && oid_target == NULL);
  backend = (rugged_bridge_backend_refdb *) _backend;
  assert(backend->adapter);
  rb_repo_name = rb_str_new2(backend->repo_path);

  rb_reference = rugged_bridge_git_reference_to_rb(ref);
  rb_signature = rugged_bridge_git_signature_to_rb(sig);

  // TODO: forward oid and oid_target along to Ruby land
  result = rb_funcall(backend->adapter, rb_intern("write"), 4, rb_repo_name,
      rb_reference, rb_force, rb_signature);

  if (NIL_P(result)) {
    giterr_set_str(GITERR_REFERENCE, "REFDB storage was unable to store ref");
    status = GIT_ERROR;
  }

  return status;
}

And the ruby adapter:

module Rugged
  module Bridge
    Reference = Struct.new(:type, :name, :target, :peel)
    Signature = Struct.new(:name, :email, :timestamp, :offset)

    class MemoryRefDBAdapter
      def initialize
        @refs = {}
      end

      def method_missing(method_name, *args, &block)
        puts "#{method_name} called with #{args.inspect} on RefDB"
        super
      end

      def lookup(repo_name, ref_name)
        @refs[ref_key(repo_name, ref_name)]
      end

      def write(repo_name, reference, force, signature)
        # TODO: accept oid and oid_target from C land
        # TODO: use incoming signature
        if !exists(repo_name, reference.name) || force
          @refs[ref_key(repo_name, reference.name)] = reference
        else
          nil
        end
      end

      def exists(repo, ref)
        @refs.has_key?(ref_key(repo, ref))
      end

      private

      def ref_key(repo, name)
        [repo, name].join(':')
      end
    end
  end
end

# in the application a backend can then be built using these pieces
backend = Rugged::Bridge::Backend.new(MemoryODBAdapter.new, MemoryRefDBAdapter.new)

As you can tell this is still a long way off of being complete but is so far a working approach that should allow easier access for non-C writing Rubyists to extend Rugged to fit their needs.

Edit: Added Reference and Signature structs for Ruby.

[silkentrance]

Any news on this, especially the config backend?