purpleidea
Json decoding function
We'd like to implement a function that decodes an mcl string of json data, and turns it into the correct mcl type. This is a difficult challenge, but definitely doable.
Design
mcl is a statically typed language (like golang) and since it's impossible to guarantee the "shape" (type) of the input data, then we need to employ some tricks. Let's start off with the function signature and an example:
$mcl_value = json_decode("[]str", $data)
The first arg is a string in the mcl type format. This can be parsed with types.NewType(...) $data should be a string of json data.
Internal API's
Since we build functions of different types depending on the input type, we'll need the fancier function signature of:
FuncInfer(partialType *types.Type, partialValues []types.Value) (*types.Type, []*interfaces.UnificationInvariant, error)
which, before type unification, lets you look at what we happen to already know statically, and build a function that fits that expectation! Of course if the type isn't known, or if it's an invalid type, we just error early, and your program won't compile!
Bad input data?
If the $data doesn't match the expected type, or if it changes at runtime, to a new version that doesn't match the expected type, then it's perfectly legal to fail. As a fancy bonus, if it happens that the data is known statically at compile time, then we should definitely parse it early. This will happen for more and more data as the compiler gets more clever about knowing what data is static.
Hints
A rough POC to show this is possible is here. This should hint that the function should be recursive. If it's possible to combine it with one of the existing functions in lang/types/ then that's even better.
// one example:
// {
// "foo": [
// "a",
// "b",
// "c"
// ],
// "bar": [
// "a",
// "b",
// "c"
// ]
// }
package main
import (
"encoding/json"
"fmt"
"os"
"io"
"reflect"
"github.com/purpleidea/mgmt/lang/types"
//"github.com/purpleidea/mgmt/lang/funcs"
"github.com/purpleidea/mgmt/lang/interfaces"
)
type JSONDecodeFunc struct {
}
func (obj *JSONDecodeFunc) FuncInfer(partialType *types.Type, partialValues []types.Value) (*types.Type, []*interfaces.UnificationInvariant, error) {
panic("not implemented")
}
func (obj *JSONDecodeFunc) Build(typ *types.Type) (*types.Type, error) {
panic("not implemented")
}
func main() {
fmt.Printf("hello\n")
f, err := os.Open("f.json")
if err != nil {
panic(err)
}
defer f.Close()
in, err := io.ReadAll(f)
if err != nil {
panic(err)
}
//var out map[string]interface{}
var out interface{}
if err := json.Unmarshal([]byte(in), &out); err != nil {
panic(err)
}
fmt.Printf("type: %T\n", out)
fmt.Printf("value: %+v\n", out)
is, ok := out.(map[string]interface{})
fmt.Printf("is: %+v\n", ok)
if !ok {
panic("not ok")
}
out2 := is["foo"]
is2, ok := out2.([]interface{})
fmt.Printf("is2: %+v\n", ok)
if !ok {
panic("not ok2")
}
// TODO: loop...
out3 := is2[0]
is3, ok := out3.(string)
fmt.Printf("is3: %+v\n", ok)
if !ok {
panic("not ok3")
}
rvalue := reflect.ValueOf(is3)
opts := []types.TypeOfOption{
//StructTagOpt(StructTag),
//StrictStructTagOpt(false),
//SkipBadStructFieldsOpt(false),
types.AllowInterfaceTypeOpt(true),
}
typ, err := types.ConfigurableTypeOf(rvalue.Type(), opts...)
if err != nil {
panic(err)
}
fmt.Printf("typ: %+v\n", typ)
// v, err := types.ValueOfGolang(out)
// if err != nil {
// panic(err)
// }
// fmt.Printf("mcl: %+v\n", v)
}
Testing
You'll need a few tests for this patch, including the obvious cases listed here:
https://pkg.go.dev/encoding/json
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
Future
If it turns out to be possible, we could also consider accepting a single arg variant of the function which requires that it can statically read the json data at compile time to determine the type. I don't know if this is possible with the golang library, we'd have to figure that out first.
Sugar
Obviously if the type argument changes at runtime, then we would have to error/shutdown. For this reason, it may be worth adding this with some compiler sugar so that it's not possible to do so.
Questions?
Don't be shy, let me know if you have questions.
Is there any situation where inferring types dynamically would be acceptable in mcl?
Is there any situation where inferring types dynamically would be acceptable in mcl?
Need more information about your question:
I think the function graph needs to have all static types, but there are situations where we decide a type at compile time, but allow a different type to be a runtime error... For example, theoretically we might allow this to be func(str, str) str but the actual type represented by the first arg (string representation) and the json, could be variable and it works as long as they match.
Something along these lines is now merged in git master.
Should be obvious how to write one of these for other formats if someone is interested.
Of course a better format that has a strict/static type system instead of lossy json would be even better, because we could inspect the data statically (when it's statically) and maybe that found type part of the type unification.
That feature is implemented in the json implementation I merged, so you can copy that if you want to do the same. Even though it can't work in all json cases.