Entitas with Roslyn Code Generation via ISourceGenerator

[studentutu]

Hi,

I have a suggestion and would want to contribute. As in official code generation guide by Unity - https://docs.unity3d.com/Manual/roslyn-analyzers.html - we can use separate project and create dll which unity will use right in it's code compilation steps.

We already have a separate project for it - Jehny, all we need is to make sure that code gen is done by Microsoft Roslyn Source Generators, and put a label “RoslynAnalyzer” for the DLL inside the release branch (create a Unity package)

This way we don't need to use Jehny Server for constant monitoring of changes, and it will help clean up the workflow.

Here's some code that we need to use for Roslyn Source Generators

using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Text;
using System.Text

[Generator]
public class ExampleSourceGenerator : ISourceGenerator
{
}

Hope it will be helpful.

[sschmid]

@studentutu thanks! Roslyn Source Generators is definitely sth I would like to research at some point! That might be useful for Entitas. Jenny however follows a different idea where the input can be anything, not just source code or assemblies. It more general purpose for any kind of code generation

[sschmid]

One downside is

Note: Roslyn analyzers are only compatible with the IDEs that Unity publically supports, which are Visual Studio and JetBrains Rider.

But I'll have a look

[sschmid]

Looks like we would need to switch to Unity when we make changes in order to recompile, which would take too long.

[sschmid]

Ok, building in the IDE works too 👍

[rubenwe]

I've written a few Source Generators. My note would be to look into Incremental Source Generators if you decide to go this route. The perf of a normal ISourceGenerator isn't that great.

Unity 2022.2 supports the required newer Microsoft packages for that. They just haven't fixed their documentation yet.

[studentutu]

yep, definitely better with IIncrementalGenerator

[rubenwe]

One thing of note though: I'm having severe problems with AdditionalFiles in Unity 2021.3 - that's files you want to pass along to source generators outside of source code. So for example if you had some csv files you would want to use as a source to generate from.

The Unity docs on this are kind of sparse - and the one page that does mention this describes an implementation that: a) does not work - at all - as described b) is of questionable design, given that one needs to rename files to somename.[Gernerator].additionalfile

So if that is something that one wants to support I'd wait for Unity to actually switch to permanent, modern csproj files and using MsBuild, as outlined in their roadmap talk. Although it is a bit bold to assume that this will just work then. They usually manage to do stuff that is not in line with what .NET devs are used to ;).

[sschmid]

Hi, I started testing IIncrementalGenerator and started a new branch. I added Entitas.Generators and Entitas.Generators.Tests projects to get started.

https://github.com/sschmid/Entitas/tree/roslyn-source-generators/src/Entitas.Generators

https://github.com/sschmid/Entitas/tree/roslyn-source-generators/tests/Entitas.Generators.Tests

So far I'am happy with IIncrementalGenerator performance and will look more into it.

[sschmid]

If all goes well I can imagine that it can replace Jenny and setting up Entitas will be much easier.

I started with snapshot testing to verify the output of the source generators, it's pretty cool. A test looks like this

[Fact]
public Task Test() => TestHelper.Verify(
    GetFixture("NamespacedComponentWithOneField"),
    new ComponentGenerator());

More about snapshot testing: https://andrewlock.net/creating-a-source-generator-part-2-testing-an-incremental-generator-with-snapshot-testing/

[sschmid]

I made some progress using IIncrementalGenerator. It's great. But now I was testing it with the latest Unity 2022.3 LTS and it looks like IIncrementalGenerator is not yet supported. Can this be true? I hope I'm wrong!

[sschmid]

Ok, got it to work!

Unity docs say you should use this specific version: Microsoft.CodeAnalysis 3.8 https://docs.unity3d.com/Manual/roslyn-analyzers.html

This version does not contain IIncrementalGenerator.

The current version is 4.6.0, but that one does not work in Unity. I manually checked each version to find the latest one that works, which is

<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.1.0" />

And you can use netstandard2.1 <TargetFramework>netstandard2.1</TargetFramework>

[sschmid]

Currently stuck, because it seems like you cannot resolve generated attributes, e.g. similar to the current approach I wanted to generate a convenience context attribute for each context, so you can add this attribute to components as usual

[MyApp.Main.Context, Other.Context]
partial MovableComponent : IComponent { }

But since those attributes are generated, the don't seem to be part of the compilation for looking up symbols 😭

This was easily possible with Jenny...

Any ideas how to solve this?

[sschmid]

To be more specific: Testing with non-generated attributes, I can easily get the attribues like this

var attribute = symbol.GetAttributes().First();

With the generated ones the same code returns ErrorTypeSymbol instead of the attributes

[studentutu]

@sschmid you can use following

1. Create a list of ICustomGenerators with Execute(GeneratorExecutionContext cx, Compilation compilation, other params if needed)

2. For each generator create a separate class:

/// <summary>
///   Generates systems for step ..... for all components.
/// </summary>
[Generator(LanguageNames.CSharp)]
public class SystemsDescriptorGenerator : IIncrementalGenerator, ICustomGenerators 
{

void ICustomGenerators.Execute(GeneratorExecutionContext cx, Compilation compilation, other params)
  {
    // Do work.
        cx.AddSource($"{cx}.{Attribute}CustomStep{component.Name}.generated", GeneratedCode(cx, component));
  }
}

3. Don't forget to add to the YourComponents.Generators.csproj

    <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.0.1" PrivateAssets="all" />
    <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.3" PrivateAssets="all" />
  </ItemGroup>

And for the actual game /Runtime .csproj:

    <ItemGroup>
    <ProjectReference Include="..\YourComponents.Generators.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />
  </ItemGroup>

[Ants-Aare]

afaik step 3. and 4. are already part of the current setup.

Can you provide documentation for ICustomGenerators interface? Is it part of roslyn or a concept you introduced yourself? what's calling ICustomGenerators.Execute?

I don't see how this makes generated code of other incremental generators available to the input compilation of this generator. Can you explain how your approach solves/bypasses this?

[studentutu]

@Ants Aare ICustomGenerators is just a custom interface, for use with a single generator (ISourceGenerator)

Basically, you begin with a normal Generator (ISourceGenerator), that includes a list of other generators (list of ICustomGenerators ).

Then in the main Generator, you simply execute each custom one by providing GeneratorExecutionContext and Compilation into them.

So by using a predefined ordering, you will get a correct compilation of source generators.

[studentutu]

By the way, if you will use IIncrementalGenerator - there is no need for custom ordering, as all IIncrementalGenerator's are executed before final compilation.

[Ants-Aare]

Maybe I'm understanding it wrong, but I don't think this solves the problem(?) What you're describing is just a way to call multiple methods one after each other through an interface inside a regular sourcegenerator. First of all this doesn't include recompilation steps inbetween the calls to ICustomGenerators, which would be neccessary to get the symbols as INamedTypeSymbols using GetAttributes(). Secondly this would downgrade the current solution to a normal ISourceGenerator instead of an IIncrementalGenerator. Btw: The order in which we call cx.AddSource is irrelevant as all files will be added to the compilation in the same pass once the generator finishes executing. No matter if we're using SourceProductionContext or GeneratorExecutionContext

[studentutu]

right, missed the part of the "generated source code with new attributes".

[rubenwe]

But since those attributes are generated, they don't seem to be part of the compilation for looking up symbols 😭

Yes, this is part of the design of source generators. This avoids having to have multiple runs and allows the caching mechanisms that make them so efficient.

For my ECS approach I'm not using generated attributes because of this.

The user defines partial classes for the contexts and components implement IComponent<TContext1, ...>.

[sschmid]

Hi, a quick update and some thoughts:

I have a working proof of concept using IIncrementalGenerator. Main goal of this new generator is easy of use (no Jenny setup needed anymore) and support for namespaces in components and contexts + support for Unity asmdefs. As previously explained in other GitHub issues, it's necessary to change the generated code to not used partial on the Entity classes (and others), but instead use static C# extension methods. This way it's possible to get a similar and familiar API as we have now, but at the same time support asmdefs. We also cannot pre-generate all component indexes anymore, as new components may be introduced from other dlls. So there are still a few challenges ahead, but here's a quick sample of how it could like based on my current generators:

Contexts

Now with namespace support! Works with or without a namespace.

You can define contexts in your code like this:

// MainContext.cs
namespace MyApp
{
    partial class MainContext : Entitas.IContext { }
}

// OtherContext.cs
partial class OtherContext : Entitas.IContext { }

---

Components

Now with namespace support! Works with or without a namespace.

You can define components in your code like this:

// MovableComponent.cs
namespace MyFeature
{
    [MyApp.Main.Context, Other.Context]
    partial class MovableComponent : Entitas.IComponent { }
}

// PositionComponent.cs
namespace MyFeature
{
    [MyApp.Main.Context, Other.Context]
    partial class PositionComponent : Entitas.IComponent
    {
        public int X;
        public int Y;
    }
}

The generated component extensions work for all specified contexts and can be chained:

MainContext mainContext = new MyApp.MainContext();
MyApp.Main.Entity mainEntity = mainContext.CreateEntity()
    .AddMovable()
    .ReplaceMovable()
    .RemoveMovable()
    .AddPosition(1, 2)
    .ReplacePosition(3, 4)
    .RemovePosition();

OtherContext otherContext = new OtherContext();
Other.Entity otherEntity = otherContext.CreateEntity()
    .AddMovable()
    .ReplaceMovable()
    .RemoveMovable()
    .AddPosition(1, 2)
    .ReplacePosition(3, 4)
    .RemovePosition();

Matchers

I currently generate component indexes for each context, e.g. MyFeaturePositionComponentIndex. They also include the namespace. I might use them later to assign component indexes ones the app starts.

Matcher.AllOf(stackalloc[]
{
    MyFeaturePositionComponentIndex.Value,
    MyFeatureMovableComponentIndex.Value
});

[sschmid]

More updates: Yay, added component deconstructors

var (x, y) = entity.GetPosition();
x.Should().Be(1);
y.Should().Be(2);

Also, since the only purpose of ContextAttributes is for code generators, I added a [Conditional] attribute, so all those attributes are stripped from components once compiled.

[sschmid]

Fyi, for those who are interested about the changes, see branch: roslyn-source-generators https://github.com/sschmid/Entitas/tree/roslyn-source-generators

[sschmid]

More updates: I'm currently testing alternatives to the currently generated ComponentLookup. The current approach doesn't allow Unity's asmdefs or multiple projects.

The new approach should work with multiple assemblies per solution. At some point however, you would need to assign an index to each component. With the following idea you can build up your game with multiple separate assemblies and the main project that consumes them can implement a partial method per context and add the ContextInitializationAttribute to it:

public static partial class ContextInitialization
{
    [MyApp.Main.ContextInitialization]
    public static partial void Initialize();
}

This will be picked up by the generator and it will generate everything that used to be in ComponentLookup, e.g.:

namespace Entitas.Generators.IntegrationTests
{
public static partial class ContextInitialization
{
    public static partial void Initialize()
    {
        MyFeatureMovableComponentIndex.Index = new ComponentIndex(0);
        MyFeaturePositionComponentIndex.Index = new ComponentIndex(1);

        MyApp.MainContext.ComponentNames = new string[]
        {
            "MyFeature.Movable",
            "MyFeature.Position"
        };

        MyApp.MainContext.ComponentTypes = new System.Type[]
        {
            typeof(MyFeature.MovableComponent),
            typeof(MyFeature.PositionComponent)
        };
    }
}
}

[sschmid]

More updates: I improved the overall caching performance by using multiple pipelines. This means, only affected files should be regenerated leaving most of the files untouched.

I can recommend this cookbook for incremental generators: https://github.com/dotnet/roslyn/blob/main/docs/features/incremental-generators.md

Next problem: if something in the generator fails, nothing will be generated. This can easily be reproduced by declaring the same component twice which will break everything. I tried to wrap all spc.AddSource() calls in a try catch block, because they fail when filenames are not unique, but this didn't help either.

Does anyone know how to make handle exceptions in a source generator?

[studentutu]

@sschmid you can use tests, and check snapshots? https://andrewlock.net/creating-a-source-generator-part-2-testing-an-incremental-generator-with-snapshot-testing/

Otherwise - you can even use simpler diagnostic as a log inside the compilation See https://github.com/dotnet/roslyn-sdk/blob/main/src/Microsoft.CodeAnalysis.Testing/README.md

await new CSharpAnalyzerTest<SomeAnalyzerType, XUnitVerifier>
{
    TestState =
    {
        Sources = { @"original source code" },
        ExpectedDiagnostics = null,
        AdditionalFiles =
        {
            ("File1.ext", "content1"),
            ("File2.ext", "content2"),
        },
    },
}.RunAsync();

// Inside try-catch, catch report diagnostic
 context.ReportDiagnostic(Diagnostic.Create(ExceptionRule,
            constructorParameter.Locations[0],
            error));

[rubenwe]

As @studentutu hinted it's probably a good idea to emit diagnostics for these problems if you run into generation failures.

On top of that, you can also be proactive and write additional implementations of DiagnosticAnalyzer for common problem scenarios. Those can also offer code fixes if the class of error has an easy way to fix it. That's the lightbulb fixes that are popping up in Visual Studio / Rider.

In my ECS prototype I used those to validate Queries (so impossible queries produce errors).

[sschmid]

More updates:

I have a working Entitas protoype that supports namespaced contexts and components with multiple Unity asmdefs

• one "Contexts" asmdef for contexts, this one could be used as a core package for features, no engine references
• one "FeatureOne" asmdef for features with components and systems (including generated code, not committed to git) that references "Contexs"
• a Test Monobehaviour in Assets folder that creates an entity

[sschmid]

I'm very happy with that now and I will proceed to implement all missing generators to get the full feature set of the current Jenny generators.

[Ants-Aare]

have you thought about using enums perhaps? you can just collect all component names together into one file and make an enum out of it like this

public enum AppContextComponents
{
    Test1Component,
    Test2Component,
    Test3Component,
}

I feel like doing both the MyFeatureMovableComponentIndex.Index = new ComponentIndex(0); and MyApp.MainContext.ComponentNames = new string[]

feels like it's basically the same thing as an enum. (ints in the lower level code that also have a string version if you need it). Plus you can just declare the enum using the component names without worrying about the actual numbers etc. the only thing that matters is that they're unique, right? enums should guarantee that. Maybe I'm understanding something wrong though.