dynamic field type(flat_obejct) value missing

[kedbirhan]

dynamic field type(flat_obejct) field are missing from returned query result. For instance requestParameter and responseElement field of cloudtrail logs are commonly mapped to flat_object field type to avoid mapping explosion but when doing so the query result returned completely ignores this field on top of that we can't even use this dynamic nested fields for querying this is a big problem for storing cloud native logs that may have dynamic field values,

[dblock]

[Catch All Triage - 1, 2, 3]

[kedbirhan]

Query

SELECT requestParameters
from logs-cloudtrail WHERE 
requestParameters.nextToken is not null
LIMIT 25

Response

400 {error={reason=There was internal problem at backend, details=FLAT_OBJECT fieldName types are currently not supported., type=IllegalArgumentException}, status=500}

[kedbirhan]

SELECT * FROM logs-cloudtrail
WHERE (
        (eventName = 'CreateInstanceExportTask' AND eventSource = 'ec2.amazonaws.com') 
        AND NOT (
                (errorMessage LIKE '%%%' ) 
                OR (errorCode LIKE '%%%') OR (responseElements LIKE '%Failure%' )))

400 Bad Request: "{<EOL> "error": {<EOL> "reason": "Invalid SQL query",<EOL> "details": "can't resolve Symbol(namespace=FIELD_NAME, name=responseElements) in type env",<EOL> "type": "SemanticCheckException"<EOL> },<EOL> "status": 400<EOL>}"

[acarbonetto]

Related issue: https://github.com/opensearch-project/sql/issues/1604

[acarbonetto]

@kedbirhan would you consider mapping the fields you'd like to search to an object? Searches on flat_objects are limited.