[Feature Request] Convert json/yaml to pkl object/class

[bielikb]

Problem statement: The adoption of jsons/yamls is pretty much spread across the community/orgs. pkl goal's to provide configuration-as-code language with rich validation and tooling. For the cases when pkl tool is the right fit this however entails a rather higher ramp up time for people converting the existing configurations -> pkl -> existing configuration format (json/yaml).

Feature Request: A converter that would map the existing configurations into pkl templates + modules. Aware of the complexity of the existing json/yaml configurations, the first versions of the converter could support the simplest mappings && let the configuration authors to convert more complex definitions manually.

[jackkleeman]

Hey @bielikb! This should be fairly easy to do with a pkl file itself (very meta, I know). pkl files can read yaml files, and output pcf, which is a static subset of pkl that you can then just copy straight into a new file (in fact, you can use pkl to write that file for you, thus making a script that does the full conversion)

Here's a simplified example:

vars.yaml

varReference:
  - kind: HTTPRoute
    path: spec/rules/matches/headers/value
  - kind: HTTPRoute
    path: spec/hostnames
  - kind: HTTPRoute
    path: spec/rules/filters/urlRewrite/path/replaceFullPath
  - kind: HTTPRoute
    path: spec/rules/filters/requestRedirect/hostname
  - kind: HTTPRoute
    path: spec/rules/filters/requestHeaderModifier/set/value
  - kind: AuthenticationFilter
    path: spec/jwtProviders/audiences

converter.pkl

import "pkl:yaml"

file = read("file:vars.yaml")
parsed = new yaml.Parser {}.parse(file)

output {
  value = parsed
}

$ pkl eval converter.pkl

varReference {
  new {
    kind = "HTTPRoute"
    path = "spec/rules/matches/headers/value"
  }
  new {
    kind = "HTTPRoute"
    path = "spec/hostnames"
  }
  new {
    kind = "HTTPRoute"
    path = "spec/rules/filters/urlRewrite/path/replaceFullPath"
  }
  new {
    kind = "HTTPRoute"
    path = "spec/rules/filters/requestRedirect/hostname"
  }
  new {
    kind = "HTTPRoute"
    path = "spec/rules/filters/requestHeaderModifier/set/value"
  }
  new {
    kind = "AuthenticationFilter"
    path = "spec/jwtProviders/audiences"
  }
}

This is essentially a Dynamic format, but with a bit of work you can copy the fields over to the equivalent typed objects (eg, there are typed for all the Kubernetes objects) using reflection. In time hopefully we can provide tools that do this for you :)

[holzensp]

@jackkleeman is entirely right and this is the starting point of convert.pkl scripts we sometimes write for specific schema (see this example for k8s). Converting JSON/YAML files to a Pkl template can't be done, because the input does not specify the types. One might make educated guesses about the types, but getting it wrong will confuse people who then use the generated templates and believe them to be authoritative.

I wanted to highlight a couple of exiting tools that might help push this along, though;

• if you have a JSON Schema, you can convert it to Pkl schema with org.json_schema
• if you have a Pkl schema and a bunch of JSON/YAML data, you can use @jackkleeman's solution and use deepToTyped to convert it from Dynamic to YourSchema.

[HalilIbrahimAndic]

Hi, I wrote a short article about conversion between Pkl and other static languages and used this page as a reference. Thank you for your contribution @jackkleeman @holzensp Here is the article: https://blog.protein.tech/pkl-from-apple-899310b43a96?source=friends_link&sk=286bfd74d0cd0e012098e198b9d18225

[moritztim]

I would love for this to be integrated directly into the cli. That would really speed up the process of migration

[moritztim]

Easy access to this would also be a great way to learn pkl without having to read the whole tutorial or even know what the types you're dealing with are called.

[jackkleeman]

I think in principle we could have a pkl-pantry module to help with this and it could be called with a one line like pkl eval module://... -p file=a.json

[bvalyou]

I think adding Parser to the Language Reference would be valuable - I had to search for this issue to find the syntax to parse yaml to an object.

[holzensp]

It's not a general language construct; it's part of the standard library. We document individual modules (like "pkl:json" and "pkl:yaml") there.

As to CLI accessibility; it's not entirely clear what that would do / look like. You can't derive the schema from an untyped source. If it's only about data conversion, you can always:

pkl eval -x 'new Parser {}.parse("[0, 1, 2, 3]")' pkl:json

or, if you have foo.json:

pkl eval -p input="$(cat foo.json)" -x 'new Parser {}.parse(read("prop:input"))' pkl:json

That could be somewhat shorter, but the wins have a low ceiling here

[moritztim]

I guess it's better in the long run anyways, for new users to learn the syntax instead of using shorthands.

PS. @holzensp pls check my issue template pr. I'm stuck there (I will likely not work on it til may tho, cuz of finals)

[holzensp]

PS. @holzensp pls check my issue template pr. I'm stuck there (I will likely not work on it til may tho, cuz of finals)

Done. Apologies for letting it sit and best of luck with finals.