#19601 | Process fields of POJOs recursively

[DatApplePy]

Fixes #19601 (and fixes #19557) | [REQ][Java][Spring] Process and generate variables/objects (and their annotations) recursively in mustache

1st round

• Changes in mustache templates related to POJO and bean validation
  • Generates field, parameter and return type recursively along with their annotations and wrappers
  • Separates the use of Optional and JsonNullable
  • Field gets @Valid if it's a model
  • Apply @NotNull only if the field is required and not nullable
  • New vendor tags to define custom validation messages
    • x-not-null-message
    • x-size-message
    • x-min-message
    • x-max-message
• Fix issues in test cases caused by these changes

2nd round

• Redesign the abstraction of getTypeDeclaration
• Recursively process properties, where it is possible
• Rework clone in CodegenProperty to get proper deep copies
• Add functionality to generate Optional.of(...) for default values
• Introduce isResolvedEnum to CodegenProperty to decide whether the enum itself can be used as a type, or its base type should be used

@cachescrubber (2022/02) @welshm (2022/02) @MelleD (2022/02) @atextor (2022/02) @manedev79 (2022/02) @javisst (2022/02) @borsch (2022/02) @banlevente (2022/02) @Zomzog (2022/09) @martin-mfg (2023/08)

PR checklist

• [X] Read the contribution guidelines.
• [X] Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
• [X] Run the following to build the project and update samples:

  ./mvnw clean package 
  ./bin/generate-samples.sh ./bin/configs/*.yaml
  ./bin/utils/export_docs_generators.sh
  

  (For Windows users, please run the script in Git BASH) Commit all changed files. This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master. These must match the expectations made by your contribution. You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*. IMPORTANT: Do NOT purge/delete any folders/files (e.g. tests) when regenerating the samples as manually written tests may be removed.
• [X] File the PR against the correct branch: master (upcoming 7.x.0 minor release - breaking changes with fallbacks), 8.0.x (breaking changes without fallbacks)
• [X] If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

[DatApplePy]

@wing328

[altro3]

@DatApplePy Don't need to add different extensions for decimal and int values, It is enough to add x-min-message and x-max-message. Because annotations @DecimalMin and @Min (@DecimalMax and @Max) will never be used together.

Need to remove x-decimalMin-message and x-decimalMax-message.

Also you can add support x-not-null-message extension for @NotNull annotation message

[DatApplePy]

@altro3 Nice catch, thanks! I'm gonna take care of them :D

[DatApplePy]

Please read this before reviewing the changes!

Originally, the plan was to modify only the mustache templates so that the data member types and their corresponding annotations are processed recursively and the final code is generated based on this. Because of this, a significant part of the getTypeDeclaration method body in the AbstractJavaCodegen, as well as the other (getTypeDeclarationForArray, getBeanValidation, isUseBeanValidation, get<type>BeanValidation) methods, became unnecessary. After I attempted to delete the unnecessary part of the getTypeDeclaration, the tests of the other 13 codegens (except for SpringCodegen) started failing one after the other. Therefore, I came up with the plan to completely delete the getTypeDeclaration method from the AbstractJavaCodegen and move the existing implementation down one level, into these 14 inheriting classes. In my opinion, an Abstract<language>Codegen class should not implement code that could differ due to framework or other dependencies, or generate computed values. Everything can be handled with mustache templates. I believe that abstract classes should only contain code related to the language itself, such as types, reserved keywords, in the case of Java, Optional, language configurations, etc.

Potential questions:

Q: Why didn’t you keep getTypeDeclaration in AbstractJavaCodegen and just override it where necessary? A: I didn’t keep it because there is a return super.getTypeDeclaration(target) at the end of the method, which would result in calls going from SpringCodegen back to AbstractJavaCodegen, but we need it to go to DefaultCodegen.

Q: So now there will be duplicate code in 14 Codegens? A: Until (possibly) all Codegens switch to not generating code with computed values, yes. But does this allow us to edit them one by one without breaking many others? Yes.

Final note

This is not a dogma, it's my personal opinion formed after 3 weeks of research, thinking, and testing. If anyone has any counterarguments, I’m all ears. If anything is unclear, feel free to ask. Thank you for reading!

[DatApplePy]

@wing328 @welshm @altro3

[martin-mfg]

Hi @DatApplePy, thank you for the PR! I appreciate your effort and enthusiasm.

Among your improvements unfortunately I also see a couple of problems:

• One side effect of the current changes is that @NotNull is added in places where it wasn't present before, e.g. here. The corresponding specification does not say nullable: true. Such changes will cause problems for many existing users. To illustrate this, see e.g. #18735 - a long discussion about problems caused by a relatively small change and about the question what the generator should assume if nullable is not specified.
• You explained in your - quite helpful - comment that this PR introduces 14-times duplicated code. I don't think this is a good approach. And especially I think it is not necessary. To solve the problem that SpringCodegen needs to override getTypeDeclaration but can't directly call DefaultCodegen.getTypeDeclaration, I propose a solution like this: in AbstractJavaCodegen:

   @Override
   public String getTypeDeclaration(Schema p) {
       return getTypeDeclaration(p, true);
   }
  
   public String getTypeDeclaration(Schema p, boolean handleContainers) {
       Schema<?> schema = unaliasSchema(p);
       Schema<?> target = ModelUtils.isGenerateAliasAsModel() ? p : schema;
       if(handleContainers) {
           if (ModelUtils.isArraySchema(target)) {
               Schema<?> items = getSchemaItems(schema);
               String typeDeclaration = getTypeDeclarationForArray(items);
               return getSchemaType(target) + "<" + typeDeclaration + ">";
           } else if (ModelUtils.isMapSchema(target)) {
               // Note: ModelUtils.isMapSchema(p) returns true when p is a composed schema that also defines
               // additionalproperties: true
               Schema<?> inner = ModelUtils.getAdditionalProperties(target);
               if (inner == null) {
                   LOGGER.error("`{}` (map property) does not have a proper inner type defined. Default to type:string", p.getName());
                   inner = new StringSchema().description("TODO default missing map inner type to string");
                   p.setAdditionalProperties(inner);
               }
               return getSchemaType(target) + "<String, " + getTypeDeclaration(inner) + ">";
           }
       }
       return super.getTypeDeclaration(target);
   }
  

  in SpringCodegen:

   @Override
   public String getTypeDeclaration(Schema p) {
       return super.getTypeDeclaration(p, false);
   }
  

• I didn't check further in detail for now, because the changes are quite big and complex; i.e. hard to review. But I also saw e.g. some changes involving @Valid, which might need a closer inspection.

As a way forward to get things merged, I'd like to suggest this: Maybe you can break your changes into smaller PRs. Since your original issue was about custom validation messages, maybe change only this aspect in the first PR. By the way, I am wondering if you could maybe simply include the custom messages in the annotations that we generate as string e.g. here. Something vaguely like return String.format(Locale.ROOT, "@Min(value=%s, message=\"%s\")", items.getMinimum(), items.getMinMessage());? Ideally, the existing samples shouldn't change in this first PR, unless they make use of custom validation messages. And then you could create additional PRs for the other topics you were working on and we can discuss them separately.

Again, thanks for your work! I'm looking forward to having your improvements merged. If you would like some more input along the way, you can reach me on Slack.

[DatApplePy]

Hi @martin-mfg! Thanks for your huge reply. In this comment, I will now only answer the NotNull part. I have yet to review the rest. (Just a little reminder for me)

		required	<<<
		true	false
nullable	true	can be any value, except undefined	can be any value
^^^	false	cannot be null nor undefined, but any other	can be any value, except null

If we combine required and nullable, we have 4 cases for field values. In pure Java (if we don't use OpenApiNullable) there is no undefined as a value. required on its own controls whether the value of a field can be undefined. In terms of Java, this is not so interesting. Well, nullable is different. In Java, due to the existence of null, there is an exception among many called NullpointerException. To avoid this, I thought that implicit nullable should be false.

The reason that NotNull appeared in new places is that I removed the use of beanValidationCore in pojo.mustache and nullableDataTypeBeanValidation.mustache and put beanValidation in their place.

[DatApplePy]

Reply to part 2 of @martin-mfg comment I reviewed your suggestion and I have no objections to it, I like it. I will apply it ASAP.

[DatApplePy]

Due to lack of free time, I am unable to complete this change. I am very sorry to have to say this after months of waiting. Unfortunately, the code is confusing enough that such a change cannot be easily resolved in a way that is readable and maintainable in the future (in my opinion).