microsoft
.Net: Integrate IEmbeddingGenerator
Adds support for using IEmbeddingGenerator to generate embeddings from arbitrary user property types.
- An IEmbeddingGenerator can be configured at the property, collection or store level. The DI methods also look up an IEmbeddingGenerator in DI, and if they find it, configure it as the store level one.
- When calling SearchAsync and UpsertAsync, we detect vector properties that have an IEmbeddingGenerator configured, and call it (note that for batching UpsertAsync, we call it once for all records, per property). When upserting, the generated embeddings are then passed into the mapper, and are used instead of the .NET property on the user's CLR type.
- Connectors are in charge of specifying what embedding types are supported (float, Half...), in order of priority. On the other hand, the VectorStoreRecordVectorPropertyModel is responsible for specifying which input types are supported (string, DataContent). This means that the user can simply have a string property on their record type, have an IEmbeddingGenerator registered in DI, and as long as that generator implements
IEmbeddingGeneratorfor string inputs, and for an embedding output that's supported by their connector, everything will just work. - The default VectorStoreRecordVectorPropertyModel supports string and DataContent, which are the "base" types expected to be implemented by actual embedding generation services. To support arbitrary input types, the user must use the newly-introduced generic
VectorStoreRecordVectorPropertyModel<TInput>(the generic part is necessary to make everything work in a trimming-safe manner). This allows arbitrary user types to be used for vector properties, and as long as the correct IEmbeddingGenerator is registered, everything will work.
Various notes
- Various bits here are going to looking a bit strange - because I've made sure to make everything compatible with trimming and NativeAOT. In older times I'd have passed around a bunch of Types for embedding types and done reflection to check what interfaces they implement - but we can't do that. This makes the code more a bit more convoluted.
- There's also definitely potential for deduplicating the added code. However, this is more complicated than it looks - I've left this for a later cleanup.
- Azure AI Search does not currently support IEmbeddingGenerator; this is because it uses an Azure API to directly handle the .NET CLR type, so we have no place to integrate the embedding generation. We should explore this in the future.
- However, Azure AI Search's built-in embedding generation is of course still supported via SearchAsync (though this is text-only, not sure what happens with e.g. images).
- InMemory required extra work: we now have a wrapper around the user POCO, which holds the generated embeddings in a separate Dictionary. This is because when using an IEmbeddingGenerator, the user POCO only has a source property (e.g. string), and we need to put the generated embeddings somewhere.
- Added VectorDataStrings.resx for exception messages; these are checked in the conformance tests.
- Made the SupportedTypes properties on VectorStoreRecordModelBuildingOptions non-nullable, which required cleaning up how connectors populate these.
Closes #10492 Closes #11527
@RogerBarreto:
Shouldn't we just use Skip in the facts or does it introduces compilation errors? If the second maybe Exclude from project with the comment in the .csproj xml would be better.
These are very temporary - the idea is that we need to figure out what we want to do with the samples around embedding generation (and yeah, IIRC there are compilation issues). So for now I just "commented out" the problematic parts.
@westey-m @adamsitnik thanks for the quick and thorough reviewing - as always.
I've gone over all comments and addressed them, this should be ready for the next round.
@markwallace-microsoft just a reminder that we still need to figure out what to do with the code coverage failure in CI etc.
@adamsitnik
From the pure coding perspective, I would like to request changes in the regard of InMemoryCollection: the static cache should be removed.
As discussed offline, I'm removing the static cache and in general the (new) ability to specify a store name/share data between multiple InMemoryVectorStore instances. This very slightly reduces InMemory test coverage but nothing meaningful - thanks again for insisting on this, I believe it's the better way.
On the other stuff:
From the usability perspective, I am not sure about how we should approach the problem of storing both the input value and its embedding (generated by the generator on the fly).
First of all, it seems to me that it makes no sense to perist only one of them (as there is no easy way of reversing the process).
We just had a design discussion with you and @eavanvalkenburg, I think we more or less reached a consensus, and I summarized the discussion and possible future work in #11734 and #11736. If my understanding is correct, we're OK to proceed with the approach in this PR, and think that the possible improvements can be added in the future (without breaking).
@westey-m @adamsitnik I've pushed more commits and addressed all outstanding comments. Note that there's a specific separate commit which does extensive work on VectorStoreTextSearch (see https://github.com/microsoft/semantic-kernel/pull/11682#discussion_r2061494517).
FYI pushed another commit to address comments. I also did a quick pass to make sure all the options are used and coalesced correctly in the search APIs, as suggested by @adamsitnik above.
@roji
Sorry to butt in, but what is the supposed migration path if we want to keep generating the embeddings as vectorized and not use the automatic embedding generation?
The migration docs only metion the replacement, but shows no concrete example of how.
Assuming I have
ITextEmbeddingGenerationService embeddingGenerator = kernel.GetRequiredService<ITextEmbeddingGenerationService>();
ReadOnlyMemory<float> embeddings = await embeddingGenerator.GenerateEmbeddingAsync(term, kernel, HttpContext.RequestAborted);
VectorSearchResults<VectorModels.Employee> empoyeesResult = await employeeCollection.VectorizedSearchAsync(embeddings, 10);
And want to keep it that way with IEmbeddingGenerator<string, Embedding<float>> w/o using the automatic embeddings ? previously I used to pass IReadOnlyMemory<float> to the VectorizedSearchAsync (now SearchAsync)?
I'm in the middle of upgrading and want do as little changes as possible to not change behavior until I can test out the new embeddings generator stuff. Neiter April 2025 nor the May 2025 vector store changes guides outline that case specifically.
@TsengSR you can pass embeddings (e.g. as ReadOnlyMemory<float>) directly to the new SearchAsync method - this is covered in this section of the May 2025 notes.
@roji
@TsengSR you can pass embeddings (e.g. as
ReadOnlyMemory<float>) directly to the new SearchAsync method - this is covered in this section of the May 2025 notes.
Yea that is how it roked before, but GenerateAsyncin the new api returns Embeddings<float>. Does it work with that too, since it's a type residing in Microsoft.Extensions.Ai? The new usage is confusing at best, since the methods aren't strong typed anymore (Search<T> not being constraint to something)
@roji
@TsengSR you can pass embeddings (e.g. as
ReadOnlyMemory<float>) directly to the new SearchAsync method - this is covered in this section of the May 2025 notes.Yea that is how it roked before, but
GenerateAsyncin the new api returnsEmbeddings<float>. Does it work with that too, since it's a type residing inMicrosoft.Extensions.Ai? The new usage is confusing at best, since the methods aren't strong typed anymore (Search<T>not being constraint to something)
@TsengSR, All vector stores now support Embedding
