uninitialized token indexes in some fragments

[vemoo]

Using Microsoft.SqlServer.TransactSql.ScriptDom version 161.9135.0.

For example:

select 1
option(recompile)

gets parsed as:

TOKENS:
0: Select "select"
1: WhiteSpace " "
2: Integer "1"
3: WhiteSpace "\n"
4: Option "option"
5: LeftParenthesis "("
6: Identifier "recompile"
7: RightParenthesis ")"
8: EndOfFile 

FRAGMENTS:
TSqlScript {
  FirstTokenIndex = 0,
  LastTokenIndex = 8,
  Batches = [
    TSqlBatch {
      FirstTokenIndex = 0,
      LastTokenIndex = 7,
      Statements = [
        SelectStatement {
          FirstTokenIndex = 0,
          LastTokenIndex = 7,
          QueryExpression = QuerySpecification {
            FirstTokenIndex = 0,
            LastTokenIndex = 2,
            UniqueRowFilter = NotSpecified,
            SelectElements = [
              SelectScalarExpression {
                FirstTokenIndex = 2,
                LastTokenIndex = 2,
                Expression = IntegerLiteral {
                  FirstTokenIndex = 2,
                  LastTokenIndex = 2,
                  LiteralType = Integer,
                  Value = 1,
                },
              },
            ],
          },
          OptimizerHints = [
            OptimizerHint {
              FirstTokenIndex = -1,
              LastTokenIndex = -1,
              HintKind = Recompile,
            },
          ],
        },
      ],
    },
  ],
}

where OptimizerHint ends up with -1 in FirstTokenIndex and LastTokenIndex.

I think the issue here is that here

https://github.com/microsoft/SqlScriptDOM/blob/d84cc30809b29cc5497809c2b0432bf23c412c69/SqlScriptDom/Parser/TSql/TSql80.g#L6276-L6294

tokens get assigned to vParent and should be assigned to vHint.

Another example:

if 1=1
begin
    select 1
end

gets parsed as:

TOKENS:
0: If "if"
1: WhiteSpace " "
2: Integer "1"
3: EqualsSign "="
4: Integer "1"
5: WhiteSpace "\n"
6: Begin "begin"
7: WhiteSpace "\n"
8: WhiteSpace "    "
9: Select "select"
10: WhiteSpace " "
11: Integer "1"
12: WhiteSpace "\n"
13: End "end"
14: EndOfFile 

FRAGMENTS:
TSqlScript {
  FirstTokenIndex = 0,
  LastTokenIndex = 14,
  Batches = [
    TSqlBatch {
      FirstTokenIndex = 0,
      LastTokenIndex = 13,
      Statements = [
        IfStatement {
          FirstTokenIndex = 0,
          LastTokenIndex = 13,
          Predicate = BooleanComparisonExpression {
            FirstTokenIndex = 2,
            LastTokenIndex = 4,
            ComparisonType = Equals,
            FirstExpression = IntegerLiteral {
              FirstTokenIndex = 2,
              LastTokenIndex = 2,
              LiteralType = Integer,
              Value = 1,
            },
            SecondExpression = IntegerLiteral {
              FirstTokenIndex = 4,
              LastTokenIndex = 4,
              LiteralType = Integer,
              Value = 1,
            },
          },
          ThenStatement = BeginEndBlockStatement {
            FirstTokenIndex = 6,
            LastTokenIndex = 13,
            StatementList = StatementList {
              FirstTokenIndex = -1,
              LastTokenIndex = -1,
              Statements = [
                SelectStatement {
                  FirstTokenIndex = 9,
                  LastTokenIndex = 11,
                  QueryExpression = QuerySpecification {
                    FirstTokenIndex = 9,
                    LastTokenIndex = 11,
                    UniqueRowFilter = NotSpecified,
                    SelectElements = [
                      SelectScalarExpression {
                        FirstTokenIndex = 11,
                        LastTokenIndex = 11,
                        Expression = IntegerLiteral {
                          FirstTokenIndex = 11,
                          LastTokenIndex = 11,
                          LiteralType = Integer,
                          Value = 1,
                        },
                      },
                    ],
                  },
                },
              ],
            },
          },
        },
      ],
    },
  ],
}

here StatementList is missing token indexes.

[llali]

@vemoo I added a new test in this PR to verify the bug https://github.com/microsoft/SqlScriptDOM/pull/94 but my test passes. I'm not sure what I'm missing

[vemoo]

the issue is not that there are no OptimizerHints, it's that FirstTokenIndex and LastTokenIndex are -1

[IVNSTN]

@llali I can confirm that some TSQLFragment descendants do never (or in some cases) have initialized FirstTokenIndex and LastTokenIndex values (they are both -1 as @vemoo already mentioned). I made a sample app with a single visitor method override Visit(TSQLFragment node) implemented to detect such defects and ran it on rather big code base.

For example, StatementList class instances seem to always have both FirstTokenIndex & LastTokenIndex == -1. Here are additional classes which also have these properties uninitialized:

• Microsoft.SqlServer.TransactSql.ScriptDom.AlterTableDropTableElement
• Microsoft.SqlServer.TransactSql.ScriptDom.DurabilityTableOption
• Microsoft.SqlServer.TransactSql.ScriptDom.EventRetentionSessionOption
• Microsoft.SqlServer.TransactSql.ScriptDom.FetchType
• Microsoft.SqlServer.TransactSql.ScriptDom.MaxDispatchLatencySessionOption
• Microsoft.SqlServer.TransactSql.ScriptDom.MemoryPartitionSessionOption
• Microsoft.SqlServer.TransactSql.ScriptDom.OptimizerHint
• Microsoft.SqlServer.TransactSql.ScriptDom.RetentionPeriodDefinition
• Microsoft.SqlServer.TransactSql.ScriptDom.StatementList
• Microsoft.SqlServer.TransactSql.ScriptDom.TemporalClause

Most of mentioned defects can be detected on this minimal piece of code:

CREATE TABLE dbo.foo
(
    id               INT          NULL
    , sys_start_time DATETIME2(0) GENERATED ALWAYS AS ROW START HIDDEN NOT NULL
    , sys_end_time   DATETIME2(0) GENERATED ALWAYS AS ROW END HIDDEN NOT NULL
    , PERIOD FOR SYSTEM_TIME(sys_start_time, sys_end_time)
)
WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.foo_history, HISTORY_RETENTION_PERIOD = 6 MONTHS));
GO

ALTER TABLE dbo.foo
DROP PERIOD FOR SYSTEM_TIME;
GO

SELECT t.id
FROM foo.bar FOR SYSTEM_TIME ALL AS t;
GO

CREATE TABLE dbo.foo (some_id INT NOT NULL)
WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA);
GO

FETCH NEXT FROM cr
INTO @id;
GO

CREATE EVENT SESSION MyXEventSession
    ON SERVER
    ADD EVENT sqlserver.sql_batch_starting
    (ACTION (sqlserver.sql_text, sqlserver.database_id, sqlserver.username)
     WHERE (sqlserver.database_name = N'asdf')
    )
    ADD TARGET package0.ring_buffer
    (SET max_memory = 4096)
    WITH
    (
        MAX_MEMORY = 4096KB
        , EVENT_RETENTION_MODE = NO_EVENT_LOSS
        , MAX_DISPATCH_LATENCY = 30 SECONDS
        , MAX_EVENT_SIZE = 0KB
        , MEMORY_PARTITION_MODE = NONE
        , TRACK_CAUSALITY = ON
        , STARTUP_STATE = OFF
    );
GO

ScriptDom version: 161.9142.1 Compatibility level: 150