vector icon indicating copy to clipboard operation
vector copied to clipboard
verified publisher icon vectordotdev

Auth.assume role doesn't work on an EC2 instance

Open RoeiSagi opened this issue 2 years ago • 4 comments

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I'm trying to retrive messages from an SQS queue on a diffrent account. in order to achieve that I have an EC2 role attached, this role can assume the other role that has access to the SQS. when trying to do I encounter an error:Failed to fetch SQS events. when doing the same with key and secret I'm able to acheve the desired soultuion

Configuration

# data_dir: "/var/lib/vector"

# Random Syslog-formatted logs
sources:
  sqs_logs:
    type: "aws_sqs"
    queue_url: "https://sqs.us-west-2.amazonaws.com/AccountID/QueueName"
    auth:
      assume_role: "arn:aws:iam::AcountID:role/RoleName"
    proxy:
      http: "MyProxy"
      https: "MyProxy"
      no_proxy: "169.254.169.254,localhost"

sinks:
  print:
    type: "console"
    inputs: ["sqs_logs"]
    encoding:
      codec: "json"

Version

vector 0.35.0 (x86_64-unknown-linux-gnu e57c0c0 2024-01-08 14:42:10.103908779)

Debug Output

2024-01-15T12:58:28.984429Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}: vector::topology::builder: Source starting.
2024-01-15T12:58:28.984714Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role: aws_config::sts::assume_role: retrieving assumed credentials
2024-01-15T12:58:28.984852Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Environment context=the credential provider was not enabled: environment variable not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "environment variable not set" }))
2024-01-15T12:58:28.984914Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}: aws_config::fs_util: loaded home directory src="HOME"
2024-01-15T12:58:28.984940Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/config"
2024-01-15T12:58:28.984970Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/config" after="/root/.aws/config"
2024-01-15T12:58:28.985015Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file not found path=~/.aws/config
2024-01-15T12:58:28.985034Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Config)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/config") size=0
2024-01-15T12:58:28.985048Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}: vector::topology::builder: Source pump supervisor starting.
2024-01-15T12:58:28.985068Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: performing home directory substitution home="/root" path="~/.aws/credentials"
2024-01-15T12:58:28.985106Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}: vector::topology::builder: Source pump starting.
2024-01-15T12:58:28.985127Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: home directory expanded before="~/.aws/credentials" after="/root/.aws/credentials"
2024-01-15T12:58:28.985170Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file not found path=~/.aws/credentials
2024-01-15T12:58:28.985193Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Profile}:load_config_file{file=Default(Credentials)}: aws_config::profile::parser::source: config file loaded path=Some("/root/.aws/credentials") size=0
2024-01-15T12:58:28.985229Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=Profile context=the credential provider was not enabled: No profiles were defined (CredentialsNotLoaded(CredentialsNotLoaded { source: NoProfilesDefined }))
2024-01-15T12:58:28.985273Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=WebIdentityToken context=the credential provider was not enabled: $AWS_WEB_IDENTITY_TOKEN_FILE was not set (CredentialsNotLoaded(CredentialsNotLoaded { source: "$AWS_WEB_IDENTITY_TOKEN_FILE was not set" }))
2024-01-15T12:58:28.985310Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: provider in chain did not provide credentials provider=EcsContainer context=the credential provider was not enabled: ECS provider not configured (CredentialsNotLoaded(CredentialsNotLoaded { source: "ECS provider not configured" }))
2024-01-15T12:58:28.985348Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loading credentials from IMDS
2024-01-15T12:58:28.985487Z DEBUG sink{component_kind="sink" component_id=print component_type=console}: vector::utilization: utilization=0.026805092930932584
2024-01-15T12:58:28.985518Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:async_map_request{name="attach_imds_token"}:send_operation{operation="get-token" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2024-01-15T12:58:28.991931Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:async_map_request{name="attach_imds_token"}:send_operation{operation="get-token" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2024-01-15T12:58:28.996648Z DEBUG hyper::proto::h1::io: flushed 242 bytes
2024-01-15T12:58:28.997854Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-01-15T12:58:28.997875Z DEBUG hyper::proto::h1::conn: incoming body is content-length (56 bytes)
2024-01-15T12:58:28.997889Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-01-15T12:58:28.998053Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2024-01-15T12:58:28.998462Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2024-01-15T12:58:28.998581Z DEBUG hyper::proto::h1::io: flushed 307 bytes
2024-01-15T12:58:28.999868Z DEBUG hyper::proto::h1::io: parsed 8 headers
2024-01-15T12:58:28.999915Z DEBUG hyper::proto::h1::conn: incoming body is content-length (11 bytes)
2024-01-15T12:58:28.999926Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-01-15T12:58:29.000029Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loaded profile profile=Splnuk-Role
2024-01-15T12:58:29.000125Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connecting to 169.254.169.254:80
2024-01-15T12:58:29.001812Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}:send_operation{operation="get" service="imds"}:dispatch: hyper::client::connect::http: connected to 169.254.169.254:80
2024-01-15T12:58:29.001921Z DEBUG hyper::proto::h1::io: flushed 318 bytes
2024-01-15T12:58:29.003371Z DEBUG hyper::proto::h1::io: parsed 8 headers
2024-01-15T12:58:29.003393Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1594 bytes)
2024-01-15T12:58:29.003410Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-01-15T12:58:29.003557Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=Ec2InstanceMetadata
2024-01-15T12:58:29.003599Z  INFO source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: credentials cache miss occurred; retrieved new AWS credentials (took 18.767825ms)
2024-01-15T12:58:29.004297Z DEBUG hyper::client::connect::dns: resolving host="sts.eu-west-1.amazonaws.com"
2024-01-15T12:58:29.006749Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:dispatch: hyper::client::connect::http: connecting to 67.220.226.247:443
2024-01-15T12:58:33.985866Z DEBUG sink{component_kind="sink" component_id=print component_type=console}: vector::utilization: utilization=0.0026805092930932578
2024-01-15T12:58:33.986036Z  INFO source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: credentials cache miss occurred; retrieved new AWS credentials (took 5.001327491s)
2024-01-15T12:58:33.986150Z ERROR source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}: vector::internal_events::aws_sqs: Failed to fetch SQS events. error=failed to construct request error_code="failed_fetching_sqs_events" error_type="request_failed" stage="receiving" internal_log_rate_limit=true
2024-01-15T12:58:33.986271Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role: aws_config::sts::assume_role: retrieving assumed credentials
2024-01-15T12:58:33.986351Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: loaded credentials from cache
2024-01-15T12:58:33.986557Z DEBUG hyper::client::connect::dns: resolving host="sts.eu-west-1.amazonaws.com"
2024-01-15T12:58:33.986617Z  INFO source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}: aws_credential_types::cache::lazy_caching: credentials cache miss occurred; retrieved new AWS credentials (took 349.142µs)
2024-01-15T12:58:33.986677Z ERROR source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}: vector::internal_events::aws_sqs: Internal log [Failed to fetch SQS events.] is being suppressed to avoid flooding.
2024-01-15T12:58:33.986772Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role: aws_config::sts::assume_role: retrieving assumed credentials

Example Data

No response

Additional Context

No response

References

No response

RoeiSagi avatar Jan 15 '24 13:01 RoeiSagi

+1 - I'm facing the same issue

Eilonh avatar Jan 16 '24 07:01 Eilonh

I have re ran the configuration with VECTOR_LOG set to aws_config=debug. And this is the output I get: 024-01-16T08:33:18.706447Z DEBUG assume_role:provide_credentials{provider=default_chain}: aws_config::meta::credentials::chain: loaded credentials provider=Ec2InstanceMetadata 2024-01-16T08:33:23.703064Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:23.703437Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:28.704227Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:28.704488Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:33.705387Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:33.705629Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:38.706193Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:38.706634Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:43.707497Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:43.707751Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:48.708481Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:48.709766Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:53.709926Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:53.711362Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:58.711798Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:33:58.712089Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:03.713969Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:03.714224Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:08.715318Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:08.715728Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:13.716401Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:13.717795Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:18.717117Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:18.719271Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:23.718694Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:23.720906Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:28.719643Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:28.722135Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:33.720682Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:33.723032Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:38.721529Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:38.723945Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:43.722488Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:43.725813Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:48.723595Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:48.727206Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:53.724732Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:53.728029Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:58.726596Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:34:58.729840Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:35:03.727350Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:35:03.731854Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:35:08.728769Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:35:08.733117Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:35:13.730737Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:35:13.734375Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:35:18.732942Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:35:18.735375Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:35:23.734792Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials 2024-01-16T08:35:23.736052Z DEBUG assume_role: aws_config::sts::assume_role: retrieving assumed credentials

RoeiSagi avatar Jan 16 '24 08:01 RoeiSagi

HI @RoeiSagi ,

I'm a little confused. The logs in the original issue seem to show credentials being fetched correctly:

2024-01-15T12:58:29.000029Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loaded profile profile=Splnuk-Role

What is the error you are seeing?

jszwedko avatar Jan 18 '24 00:01 jszwedko

HI @RoeiSagi ,

I'm a little confused. The logs in the original issue seem to show credentials being fetched correctly:

2024-01-15T12:58:29.000029Z DEBUG source{component_kind="source" component_id=sqs_logs component_type=aws_sqs}:send_operation{operation="ReceiveMessage" service="sqs"}:async_map_request{name="retrieve_credentials"}:assume_role:send_operation{operation="AssumeRole" service="sts"}:async_map_request{name="retrieve_credentials"}:lazy_load_credentials:provide_credentials{provider=default_chain}:load_credentials{provider=Ec2InstanceMetadata}: aws_config::imds::credentials: loaded profile profile=Splnuk-Role

What is the error you are seeing?

Hey @jszwedko . The the following log reffers to the instacne Role (Splnuk-Role). I have a diffrent role that this role needs to assume. so as far as the auth flow goes: Vector is assuming his instance profile role(Splnuk-Role), but then he is stuck on retriving credentials for the role he needs to assume.

RoeiSagi avatar Jan 23 '24 09:01 RoeiSagi

We are also facing this issue. The setup was working fine in 0.33.0 but broken in 0.36. Unfortunately due to the syslog source cannot parse IPv6 addresses in the hostname bug we cannot use 0.33.0. We are running Vector in Kubernets and using EKS Pod Identity Provider to set up a WebIdentityToken "environment" with a role in the current account but we want to assume another role in another account where we store our logs on S3.

akunszt avatar Mar 06 '24 16:03 akunszt

We are also facing this issue. The setup was working fine in 0.33.0 but broken in 0.36. Unfortunately due to the syslog source cannot parse IPv6 addresses in the hostname bug we cannot use 0.33.0. We are running Vector in Kubernets and using EKS Pod Identity Provider to set up a WebIdentityToken "environment" with a role in the current account but we want to assume another role in another account where we store our logs on S3.

Can you confirm if the versions between 0.33.0 and 0.36.0 work or not? For example, does 0.34.0 work? 0.35.0? You might be running into https://github.com/vectordotdev/vector/issues/19879 which was recently fixed and pending release.

jszwedko avatar Mar 06 '24 16:03 jszwedko

@jszwedko FYI:

0.33.0 - works 0.34.2 - works 0.35.1 - works 0.36.0 - does not work

#19879 looks like the same issue.

As it is already fixed in the master then it will be released in 0.36.1 or 0.37.0, right? I think we can use 0.35.1 but I have to test it first. If not then we have to wait for 0.37.0.

Thank you for you support. I am not the author of this ticket but I assume this also can be closed as a duplicate of #19879.

akunszt avatar Mar 07 '24 12:03 akunszt

Thanks for confirming @akunszt ! We'll be releasing it as 0.36.1 shortly (probably Monday). I'll close this out for now, but let us know if the new release doesn't work for you.

jszwedko avatar Mar 07 '24 15:03 jszwedko