AWS::KinesisFirehose::DeliveryStream as an Event Source

[amostech]

Description:

Original StackOverflow Post: https://stackoverflow.com/questions/49784126/automate-creation-of-kinesis-firehose-delivery-stream-splunk-destination-with

I have successfully setup a Kinesis Stream to send Data to Splunk by using the Splunk Add-on for Amazon Kinesis (https://splunkbase.splunk.com/app/3719/).

I followed the procedure described in the official documentation: https://docs.aws.amazon.com/firehose/latest/dev/create-destination.html#create-destination-splunk

All works well and fine. Data is succesfully sent from Kinesis stream into Splunk.

The problem is that now to use this in production we need to create a SAM (https://github.com/awslabs/serverless-application-model) to automate the deployment of the Firehose Delivery Stream.

Currently my yaml template looks like (https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis-example-use-app-spec.html):

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  KinesisStream:
    Type: AWS::Kinesis::Stream
    Properties:
      ShardCount: 2
      Name: my-kinesis-stream-name

  ProcessKinesisRecords:
    Type: AWS::Serverless::Function
    Properties:
      Handler: handler
      Runtime: runtime
      Policies:
       - AWSLambdaExecute # Managed Policy
       - Version: '2012-10-17' # Policy Document
         Statement:
           - Effect: Allow
             Action:
               - dynamodb:GetItem
               - dynamodb:Scan
               - dynamodb:Query
             Resource: !Join [ "", [ "arn:aws:dynamodb:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":table/TABLE_NAME" ] ]
      CodeUri: CodePackage.zip
      Events:
        Stream:
          Type: Kinesis
          Properties:
            Stream: !GetAtt KinesisStream.Arn
            BatchSize: 25
            StartingPosition: TRIM_HORIZON

This template works fine when I deploy it with CloudFormation and it successfully creates the Kinesis Stream, Lambda function and Roles.

How can I extend this template to also create a Firehose Delivery Stream using the SplunkDestinationConfiguration?

I've found the following piece of documentation in AWS (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesisfirehose-deliverystream.html) which points me towards adding the following block into my SAM template:

Type: "AWS::KinesisFirehose::DeliveryStream"
Properties: 
  DeliveryStreamName: String
  DeliveryStreamType: String
  ElasticsearchDestinationConfiguration:
    ElasticsearchDestinationConfiguration
  ExtendedS3DestinationConfiguration:
    ExtendedS3DestinationConfiguration
  KinesisStreamSourceConfiguration:
    KinesisStreamSourceConfiguration
  RedshiftDestinationConfiguration:
    RedshiftDestinationConfiguration
  S3DestinationConfiguration:
    S3DestinationConfiguration

But notice that there's no way to specify a "SplunkDestinationConfiguration" at least it isn't explained in the documentation for SAM. Can anyone help in understanding whether this is supported by SAM currently and if not how can one accomplish that?

Steps to reproduce the issue:

1. Create a SAM template that deploys a Simple Lambda Function and set the source as Kinesis Stream as described in: https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis-example-use-app-spec.html
2. Create a Kinesis Firehose Delivery Stream with Splunk as Destination https://www.splunk.com/blog/2017/11/29/ready-set-stream-with-the-kinesis-firehose-and-splunk-integration.html (No need to create a Splunk Server for that)
3. Try to follow the explanation in the documentation at https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kinesisfirehose-deliverystream.html to AUTOMATE the creation of the Kinesis Firehose Delivery Stream with Splunk Destination.

Observed result:

It currently doesn't seem to be possible to specify SplunkDestinationConfiguration in the SAM yaml template to automate the creation of the Kinesis Firehose Data Delivery Stream.

Expected result:

I would like to be able to create the Delivery Stream from my SAM template and CloudFormation automatically.

[ombozek]

SAM currently does not support AWS::KinesisFirehose::DeliveryStream as a serverless function event source. I'll flag this as a requested resource. In the meantime the following are supported event source types: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#event-source-types Another alternative would be using CloudFormation instead of SAM to define this lambda function and event source.