Unable to change monitor IP on secondary node; permission errors during mon IP modification

[hamedprog]

I'm setting up a 3-node MicroCeph cluster across three different subnets. After bootstrap and node join operations, I discovered slow operation errors in logs. Investigation revealed inconsistent IP binding between nodes:

1. First Node (Bootstrap):

   sudo microceph cluster bootstrap --cluster-network 0.0.0.0/0 --mon-ip <first-node-public-ip> --public-network 0.0.0.0/0 --microceph-ip <first-node-public-ip>
   

   • Binds correctly to public IP for both services:
     • 7443 (MicroCeph): <first-node-public-ip>
     • 6789 (Mon): <first-node-public-ip>

2. Second Node (Joined via token):

   sudo microceph cluster join <token> --microceph-ip <second-node-public-ip>
   

   • Binds inconsistently:
     • 7443 (MicroCeph): <second-node-public-ip> (correct)
     • 6789 (Mon): 192.168.1.137 (private IP, incorrect)

Attempted to change the monitor IP following Ceph's documentation but encountered permission errors:

sudo ceph mon -i {mon-id} --mkfs --monmap /tmp/mapfile --keyring /tmp/keyfile
# Error:
2025-06-10T07:32:13.032+0000 7f2b467fc6c0 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
[errno 13] RADOS permission denied (error connecting to the cluster)

Key Questions

1. Is there a way to specify the monitor IP (--mon-ip) during microceph cluster join?
2. If not, what's the correct procedure to change the monitor IP post-join without triggering permission errors?

Additional Context

• Cluster network configuration uses 0.0.0.0/0 (all interfaces)
• Nodes reside in different subnets, requiring explicit public IP binding
• Observed "slow ops" errors in logs likely due to cross-subnet communication issues

[syncronize-issues-to-jira[bot]]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CEPH-1334.

This message was autogenerated