feature : in server location add more token as {year}, {month}, {quarter} , {day} and {date} as {model}

[dmaresma]

https://github.com/datacontract/datacontract-cli/blob/bf47fea74d10e2f43f04ca4f5c12f4c3f57623ee/datacontract/engines/soda/connections/duckdb_connection.py#L37

is it possible to add more token as date parts, this should not introduce regression.

...
date = datetime.today()
  month_to_quarter = { 1: "Q1", 2: "Q1", 3: "Q1", 4: "Q2", 5: "Q2", 6: "Q2",
  7: "Q3", 8: "Q3", 9: "Q3",10: "Q4", 11: "Q4", 12: "Q4" }

  model_path = model_path.format(model=model_name, 
                                 year=date.strftime('%Y'),
                                 month=date.strftime('%m'),
                                 day=date.strftime('%d'), 
                                 date=date.strftime('%Y-%m-%d'),
                                 quarter=month_to_quarter.get(date.month))

example of implementation into odcs

...
servers
  - server : datalake_prd
      type: azure
      storageAccount: storageaccount
      location: abfss://[email protected]/entity={model}/year={year}/month={month}/day={day}/*.json
... 

[jochenchrist]

These are usually modeled as globs

location: abfss://[email protected]/entity{model}/year=*/month=*/day=*/*.json

Do you have a usecase where you need to determine the date information dynamically?

[dmaresma]

Hi @jochenchrist, Here’s how I’m using data contracts in my workflows:

In the ingestion pipeline, I use data contracts to monitor changes in JSON payload definitions. Before triggering the Matillion ETL process, the contract validates the structure and quality of the incoming data as a daily sanity check (Pull scenario). In the egress pipeline, at the end of the process, we perform a final validation on the output payload to ensure its quality before delivery (Push scenario). Day-to-day operations: Data contracts help speed up validation and reduce the amount of data we need to parse by focusing only on the latest payload file. Thanks for your attention

[dmaresma]

Complément : we work in silo, we don't have guarantee or notification the EAI (mulesoft, biztalk) change the structure of the payload.