Reimplementation of Orchestration, Scheduling, and Credential Management in Golang

[yasithdev]

🎯 Conceptual Model

This PR is more of a RFC to think through what the fundamental problem that airavata should tackle. This POC is built around 6 core domain interfaces that represent the fundamental operations of distributed task execution. The POC uses a gRPC-based worker system for task execution:

┌─────────────────────────────────────────────────────────────────┐
│                    Core Domain Interfaces                       │
├─────────────────────────────────────────────────────────────────┤
│  ResourceRegistry    │  CredentialVault     │  ExperimentOrch   │
│  • Register compute  │  • Secure storage    │  • Create exper   │
│  • Register storage  │  • Unix permissions  │  • Generate tasks │
│  • Validate access   │  • Encrypt/decrypt   │  • Submit for exec│
├─────────────────────────────────────────────────────────────────┤
│  TaskScheduler       │  DataMover          │  WorkerLifecycle  │
│  • Cost optimization │  • 3-hop staging    │  • Spawn workers  │
│  • Worker distrib    │  • Persistent cache │  • gRPC workers   │
│  • Atomic assignment │  • Lineage tracking │  • Task execution │
└─────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────┐
│                    gRPC Worker System                          │
├─────────────────────────────────────────────────────────────────┤
│  Worker Binary      │  Script Generation   │  Task Execution   │
│  • Standalone exec  │  • SLURM scripts     │  • Poll for tasks │
│  • gRPC client      │  • K8s manifests     │  • Execute tasks  │
│  • Auto-deployment  │  • Bare metal scripts│  • Report results │
└─────────────────────────────────────────────────────────────────┘

🔐 Credential Management Architecture

The Airavata Scheduler implements a three-layer credential architecture that separates authorization logic from storage for maximum security and scalability:

┌─────────────────────────────────────────────────────────────┐
│                    Application Layer                        │
│   (Experiments, Resources, Users, Groups)                   │
└────────────┬────────────────────────────────────────────────┘
             │
             ├──────────────────┬──────────────────┐
             │                  │                  │
┌────────────▼─────┐  ┌────────▼────────┐  ┌─────▼──────────┐
│   PostgreSQL     │  │    SpiceDB      │  │   OpenBao      │
│                  │  │                 │  │                │
│  Domain Data     │  │  Authorization  │  │  Secrets       │
│  - Users         │  │  - Permissions  │  │  - SSH Keys    │
│  - Groups        │  │  - Ownership    │  │  - Passwords   │
│  - Experiments   │  │  - Sharing      │  │  - Tokens      │
│  - Resources     │  │  - Hierarchies  │  │  (Encrypted)   │
└──────────────────┘  └─────────────────┘  └────────────────┘

Key Benefits

• 🔒 Separation of Concerns: Authorization (SpiceDB) separate from secret storage (OpenBao)
• 🛡️ Fine-grained Permissions: Read/write/delete permissions with hierarchical group inheritance
• 📋 Complete Audit Trail: All operations logged across all three systems
• 🔄 Credential Rotation: Support for automatic key rotation with zero downtime
• 👥 Group Management: Groups can contain groups with transitive permission inheritance
• 🔗 Resource Binding: Credentials bound to specific compute/storage resources