Dynamic environment variables in the config file not working in ubuntu based build agents

[nikhil-neu]

Describe the bug Env variables in config variables not working in ubuntu build agents . Getting the error Validation of config file completed. Update-PropertiesForObject: /root/.local/share/powershell/Modules/azure.datafactory.tools/0.95.1/private/Update-PropertiesFromFile.ps1:75 Line | 75 | … $null = Update-PropertiesForObject -o $_ -action $action -path $p … | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | Cannot bind argument to parameter 'value' because it is an | empty string.

To Reproduce Using this config file

Your script to reproduce the behaviour:

{
    "akv_ls_amnoneshd": [
      {
        "name": "$.properties.typeProperties.baseUrl",
        "value": "https://co-wus2-amnoneshd-kv-t01.vault.azure.net/",
        "action": "update"
      }
    ],
    "sql_ls_amie_amhdb": [
      {
        "name": "$.properties.connectVia.referenceName",
        "value": "ADFIntegrationRuntime",
        "action": "update"
      }
    ]
}

Expected behaviour Config file should be used to paramterise the json files

Version of PowerShell module of: latest

• ver.........

[NowinskiK]

I run (under UBUNTU agent) the subset of unit tests included in the module and it works fine:

As an additional check I run this PS script under Ubuntu agent:

$m = 'azure.datafactory.tools'
Install-Module $m -Scope CurrentUser
Import-Module $m

$RootFolder = 'X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\BigFactorySample2'
$config = 'X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\BigFactorySample2\deployment\config-c100.json'

Publish-AdfV2FromJson -RootFolder $RootFolder -ResourceGroupName 'as' -DataFactoryName 'xyz' -Stage $config -DryRun

The above code finished successfully either, until empty value met, which is expected:

All previous configuration items have been processed properly. Do double-check your values or variables. You can switch DEBUG mode ON to see more details.

[nikhil-neu]

@NowinskiK .. but my value isnt empty .. Here is my config file type,name,path,value linkedService,akv_ls_amnoneshd,typeProperties.baseUrl,"$Env:kvURI" linkedService,sql_ls_amie_amhdb,connectVia.referenceName,"intRunTimeAmnOneAmieDev" i enabled verbose ` Validation of config file completed. VERBOSE: - Performing: update for object(path): linkedService.akv_ls_amnoneshd(properties.typeProperties.baseUrl) VERBOSE: Entering Function: ConvertFrom-ArraysToOrderedHashTables VERBOSE: Processing PSCustomObject... VERBOSE: Properties: name type properties VERBOSE: Processing property 'name' of type String VERBOSE: Processing property 'type' of type String VERBOSE: Processing property 'properties' of type PSCustomObject VERBOSE: Converting PSCustomObject property using a recursive function call... VERBOSE: Entering Function: ConvertFrom-ArraysToOrderedHashTables VERBOSE: Processing PSCustomObject... VERBOSE: Properties: annotations type typeProperties VERBOSE: Processing property 'annotations' of type Object[] VERBOSE: Processing property 'type' of type String VERBOSE: Processing property 'typeProperties' of type PSCustomObject VERBOSE: Converting PSCustomObject property using a recursive function call... VERBOSE: Entering Function: ConvertFrom-ArraysToOrderedHashTables VERBOSE: Processing PSCustomObject... VERBOSE: Properties: baseUrl VERBOSE: Processing property 'baseUrl' of type String Update-PropertiesForObject: /root/.local/share/powershell/Modules/azure.datafactory.tools/0.95.1/private/Update-PropertiesFromFile.ps1:75 Line | 75 | … $null = Update-PropertiesForObject -o $_ -action $action -path $p … | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | Cannot bind argument to parameter 'value' because it is an | empty string.

##[error]PowerShell exited with code '1'. `

The same task can be passing here , when run on a windows agent with everything remaining same

VERBOSE: Properties: annotations type typeProperties VERBOSE: Processing property 'annotations' of type Object[] VERBOSE: Processing property 'type' of type String VERBOSE: Processing property 'typeProperties' of type PSCustomObject VERBOSE: Converting PSCustomObject property using a recursive function call... VERBOSE: Entering Function: ConvertFrom-ArraysToOrderedHashTables VERBOSE: Processing PSCustomObject... VERBOSE: Properties: baseUrl VERBOSE: Processing property 'baseUrl' of type String VERBOSE: Entering Function: ConvertFrom-OrderedHashTablesToArrays VERBOSE: Processing PSCustomObject... VERBOSE: Properties: name type properties VERBOSE: Processing property 'name' of type String VERBOSE: Processing property 'type' of type String VERBOSE: Processing property 'properties' of type PSCustomObject VERBOSE: Converting PSCustomObject property using a recursive function call... VERBOSE: Entering Function: ConvertFrom-OrderedHashTablesToArrays VERBOSE: Processing PSCustomObject... VERBOSE: Properties: annotations type typeProperties VERBOSE: Processing property 'annotations' of type Object[] VERBOSE: Processing property 'type' of type String VERBOSE: Processing property 'typeProperties' of type PSCustomObject VERBOSE: Converting PSCustomObject property using a recursive function call... VERBOSE: Entering Function: ConvertFrom-OrderedHashTablesToArrays VERBOSE: Processing PSCustomObject... VERBOSE: Processing property 'baseUrl' of type String VERBOSE: - Performing: update for object(path): linkedService.sql_ls_amie_amhdb(properties.connectVia.referenceName) VERBOSE: Entering Function: ConvertFrom-ArraysToOrderedHashTables VERBOSE: Processing PSCustomObject...

[NowinskiK]

Can you attach both files of Linked Services?

[nikhil-neu]

@NowinskiK akv_ls_amnoneshd

{ "name": "akv_ls_amnoneshd", "type": "Microsoft.DataFactory/factories/linkedservices", "properties": { "annotations": [], "type": "AzureKeyVault", "typeProperties": { "baseUrl": "https://co-wus2-xxxxxxx-kv-d01.vault.azure.net/" } } }

sql_ls_amie_amhdb { "name": "sql_ls_xxxx_amhdb", "type": "Microsoft.DataFactory/factories/linkedservices", "properties": { "annotations": [], "type": "SqlServer", "typeProperties": { "connectionString": { "type": "AzureKeyVaultSecret", "store": { "referenceName": "akv_ls_amnoneshd", "type": "LinkedServiceReference" }, "secretName": "xxxdb-sqlconnection-adf" } }, "connectVia": { "referenceName": "intRunTimexxxOneAmieDev", "type": "IntegrationRuntimeReference" } } }

[nikhil-neu]

@NowinskiK ... any work around for this ?

[NowinskiK]

I'm on holiday right now. I will back to this as soon as I can. In the meantime, you can investigate it on your own.

[nikhil-neu]

@NowinskiK .. i have been still unable to find a work around for this , would you be having ideas ?

[NowinskiK]

Unfortunately, I don't have too much capacity now, so we must ask someone from the community to investigate this issue deeper.