Use pipeline data for http post|put|patch|delete commands.
WIP
Description
Provides the ability to use http commands as part of a pipeline. Additionally, this pull requests extends the pipeline metadata to add a content_type field. The content_type metadata field allows commands such as to json to set the metadata in the pipeline allowing the http commands to use it when making requests.
This pull request also introduces the ability to directly stream http requests from streaming pipelines.
One other small change is that Content-Type will always be set if it is passed in to the http commands, either indirectly or throw the content type flag. Previously it was not preserved with requests that were not of type json or form data.
User-Facing Changes
-
http post,http put,http patch,http deletecan be used as part of a pipeline -
to text,to json,from jsonall set the content_type metadata field and the http commands will utilize them when making requests.
I'm also wondering if from json should also set the content-type metadata, just to be thorough?
I'm also wondering if
from jsonshould also set the content-type metadata, just to be thorough?
100%
I had another thought too, which is a problem with metadata that we experience right now and that is, every command that has the possibility of included data, like json, now has to ensure that the metadata is passed through and not eaten. We currently experience this with the ls datasource. You notice that the metadata is not present when the name column doesn't have the LS_COLORS coloring.
It would be nice to abstract this metadata up a level somehow so that each command doesn't have to worry about checking or it or manually including it.
I had another thought too, which is a problem with metadata that we experience right now and that is, every command that has the possibility of included data, like json, now has to ensure that the metadata is passed through and not eaten. We currently experience this with the ls datasource. You notice that the metadata is not present when the name column doesn't have the LS_COLORS coloring.
It would be nice to abstract this metadata up a level somehow so that each command doesn't have to worry about checking or it or manually including it.
I think this is a good point. We would still need the ability to mutate the metadata.. for instance when the open command is called it, it sets the source metadata field. The to * and from * commands preserve the source and add the content_type metadata field.
Since this would be a larger architectural change, I think I would prefer to address it separately.
Since this would be a larger architectural change, I think I would prefer to address it separately.
100%. I didn't mean to imply that it needed to be in this PR. I was just thinking out loud to see if others agree here because I'm not a huge fan of how it works today because it's so each to miss, "oh, i forgot to pass on or modify the metadata", and now things don't work as expected.
@fdncred, @devyn
This is ready for review.
I looked through it. It seems reasonable to me.