Incorrect FirstTokenIndex returned by the CopyStatement.Options fragment

[dhiraj-n]

We have encountered a parsing issue in Microsoft Documentation in COPY INTO using TSql170Parser 170.128.0.

Example script

COPY INTO test_1
FROM 'https://myaccount.blob.core.windows.net/myblobcontainer/folder1/'
WITH (
    FILE_TYPE = 'CSV',
    CREDENTIAL=(IDENTITY= 'Shared Access Signature', SECRET='<Your_SAS_Token>'),
    --CREDENTIAL should look something like this:
    --CREDENTIAL=(IDENTITY= 'Shared Access Signature', SECRET='?sv=2018-03-28&ss=bfqt&srt=sco&sp=rl&st=2016-10-17T20%3A14%3A55Z&se=2021-10-18T20%3A19%3A00Z&sig=IEoOdmeYnE9%2FKiJDSHFSYsz4AkNa%2F%2BTx61FuQ%2FfKHefqoBE%3D'),
    FIELDQUOTE = '"',
    FIELDTERMINATOR = ';',
    ROWTERMINATOR = '0X0A',
    ENCODING = 'UTF8',
    MAXERRORS = 10,
    ERRORFILE = '/errorsfolder'--path starting from the storage container
)

Incorrect Behaviour: Instead of returning the FirstTokenIndex of FILE_TYPE token, it returns the index of 'CSV' instead. Similarly for other columns as well : CREDENTIAL, FIELDQUOTE, and so on.

Correct Behaviour is observed for queries like CREATE VECTOR INDEX. For eg:

CREATE VECTOR INDEX vec_idx
    ON [dbo].[wikipedia_articles] ([title_vector])
        WITH (METRIC = 'COSINE', TYPE = 'DISKANN');

Here the FirstTokenIndex is returned of the METRIC, or TYPE tokens are of correct tokens.