From eee8e1e4d47cb72ee4818302e066c0a76cda86c8 Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Mon, 10 Feb 2025 19:05:27 -0500 Subject: [PATCH 01/19] Add Postgres connector code samples --- semantic-kernel/concepts/vector-store-connectors/code-samples.md | 1 + 1 file changed, 1 insertion(+) diff --git a/semantic-kernel/concepts/vector-store-connectors/code-samples.md b/semantic-kernel/concepts/vector-store-connectors/code-samples.md index 5080ff66..ba58be1d 100644 --- a/semantic-kernel/concepts/vector-store-connectors/code-samples.md +++ b/semantic-kernel/concepts/vector-store-connectors/code-samples.md @@ -50,6 +50,7 @@ vector store they create to use with the common code. - [Azure AI Search vector search with common code](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Memory/VectorStore_VectorSearch_MultiStore_AzureAISearch.cs) - [InMemory vector search with common code](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Memory/VectorStore_VectorSearch_MultiStore_InMemory.cs) +- [Postgres vector search with common code](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Memory/VectorStore_VectorSearch_MultiStore_Postgres.cs) - [Qdrant vector search with common code](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Memory/VectorStore_VectorSearch_MultiStore_Qdrant.cs) - [Redis vector search with common code](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Memory/VectorStore_VectorSearch_MultiStore_Redis.cs) From 1c05ab7a44249b831cf4108b55b60bf3502e768e Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Mon, 10 Feb 2025 19:11:23 -0500 Subject: [PATCH 02/19] Update postgres connector docs. Includes some expanded docs around dotnet and adds docs for the python --- .../postgres-connector.md | 423 ++++++++++++++++-- 1 file changed, 388 insertions(+), 35 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 8eb24e45..ed9d9394 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -41,19 +41,43 @@ Add the Postgres Vector Store connector NuGet package to your project. dotnet add package Microsoft.SemanticKernel.Connectors.Postgres --prerelease ``` -You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by Semantic Kernel. +You can add the vector store to the dependency injection container available on the `KernelBuilder` or to the `IServiceCollection` dependency injection container using extension methods provided by Semantic Kernel. -In this case, an instance of the `Npgsql.NpgsqlDataSource` class, which has vector capabilities enabled, will also be registered with the container. +```csharp +using Microsoft.SemanticKernel; + +// Using Kernel Builder. +var kernelBuilder = Kernel + .CreateBuilder() + .AddPostgresVectorStore(""); +``` ```csharp using Microsoft.SemanticKernel; // Using IServiceCollection with ASP.NET Core. var builder = WebApplication.CreateBuilder(args); -builder.Services.AddPostgresVectorStore("Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;"); +builder.Services.AddPostgresVectorStore(""); ``` -Extension methods that take no parameters are also provided. These require an instance of the `Npgsql.NpgsqlDataSource` class to be separately registered with the dependency injection container. +Where `` is a connection string to the Postgres instance, in the format that [Npgsql](https://www.npgsql.org/) expects. + +Extension methods that take no parameters are also provided. These require an instance of [NpgsqlDataSource](https://www.npgsql.org/doc/api/Npgsql.NpgsqlDataSource.html) to be separately registered with the dependency injection container. Note that "UseVector" must be called on the builder to enable vector support via [pgvector-dotnet](https://github.com/pgvector/pgvector-dotnet?tab=readme-ov-file#npgsql-c): + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Microsoft.SemanticKernel; +using Npgsql; + +// Using Kernel Builder. +var kernelBuilder = Kernel.CreateBuilder(); +kernelBuilder.Services.AddSingleton(sp => { + NpgsqlDataSourceBuilder dataSourceBuilder = new(""); + dataSourceBuilder.UseVector(); + return dataSourceBuilder.Build(); +}); +kernelBuilder.AddPostgresVectorStore(); +``` ```csharp using Microsoft.Extensions.DependencyInjection; @@ -62,13 +86,11 @@ using Npgsql; // Using IServiceCollection with ASP.NET Core. var builder = WebApplication.CreateBuilder(args); -builder.Services.AddSingleton(sp => -{ - NpgsqlDataSourceBuilder dataSourceBuilder = new("Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;"); +kernelBuilder.Services.AddSingleton(sp => { + NpgsqlDataSourceBuilder dataSourceBuilder = new(""); dataSourceBuilder.UseVector(); return dataSourceBuilder.Build(); }); - builder.Services.AddPostgresVectorStore(); ``` @@ -76,77 +98,408 @@ You can construct a Postgres Vector Store instance directly. ```csharp using Microsoft.SemanticKernel.Connectors.Postgres; -using Npgsql; +using Postgres.Client; -NpgsqlDataSourceBuilder dataSourceBuilder = new("Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;"); +NpgsqlDataSourceBuilder dataSourceBuilder = new(""); dataSourceBuilder.UseVector(); -var dataSource = dataSourceBuilder.Build(); - -var connection = new PostgresVectorStore(dataSource); +using NpgsqlDataSource dataSource = dataSourceBuilder.Build(); +var vectorStore = new PostgresVectorStore(dataSource); ``` It is possible to construct a direct reference to a named collection. ```csharp using Microsoft.SemanticKernel.Connectors.Postgres; -using Npgsql; +using Postgres.Client; -NpgsqlDataSourceBuilder dataSourceBuilder = new("Host=localhost;Port=5432;Username=postgres;Password=example;Database=postgres;"); +NpgsqlDataSourceBuilder dataSourceBuilder = new(""); dataSourceBuilder.UseVector(); -var dataSource = dataSourceBuilder.Build(); - -var collection = new PostgresVectorStoreRecordCollection(dataSource, "skhotels"); +using NpgsqlDataSource dataSource = dataSourceBuilder.Build(); +var collection = new PostgresVectorStoreRecordCollection(dataSource, "skhotels"); ``` ## Data mapping -The Postgres Vector Store connector provides a default mapper when mapping from the data model to storage. -This mapper does a direct conversion of the list of properties on the data model to the columns in Postgres. +The Postgres connector provides a default mapper when mapping data from the data model to storage. +The default mapper uses the model annotations or record definition to determine the type of each property and to map the model +into a Dictionary that can be serialized to Postgres. + +- The data model property annotated as a key will be mapped to the PRIMARY KEY in the Postgres table. +- The data model properties annotated as data will be mapped to a table column in Postgres. +- The data model properties annotated as vectors will be mapped to a table column that has the pgvector `VECTOR` type in Postgres. It's also possible to override the default mapper behavior by providing a custom mapper via the `PostgresVectorStoreRecordCollectionOptions.DictionaryCustomMapper` property. ### Property name override -You can override property names to use in storage that is different to the property names on the data model. -The property name override is done by setting the `StoragePropertyName` option via the data model property attributes or record definition. +For data properties and vector properties (if using named vectors mode), you can provide override field names to use in storage that is different from the +property names on the data model. + +The property name override is done by setting the `StoragePropertyName` option via the data model attributes or record definition. -Here is an example of a data model with `StoragePropertyName` set on its attributes and how that will be represented in Postgres query. +Here is an example of a data model with `StoragePropertyName` set on its attributes and how that will be represented in Postgres as a table, assuming the Collection name is `Hotels`. ```csharp using Microsoft.Extensions.VectorData; public class Hotel { - [VectorStoreRecordKey] + [VectorStoreRecordKey(StoragePropertyName = "hotel_id")] public ulong HotelId { get; set; } [VectorStoreRecordData(StoragePropertyName = "hotel_name")] - public string? HotelName { get; set; } + public string HotelName { get; set; } [VectorStoreRecordData(StoragePropertyName = "hotel_description")] - public string? Description { get; set; } + public string Description { get; set; } - [VectorStoreRecordVector(Dimensions: 4, DistanceFunction.CosineDistance)] + [VectorStoreRecordVector(Dimensions: 4, DistanceFunction: DistanceFunction.CosineDistance, IndexKind: IndexKind.Hnsw, StoragePropertyName = "hotel_description_embedding")] public ReadOnlyMemory? DescriptionEmbedding { get; set; } } ``` ```sql -CREATE TABLE public."Hotels" ( - "HotelId" INTEGER NOT NULL, - "hotel_name" TEXT , - "hotel_description" TEXT , - "DescriptionEmbedding" VECTOR(4) , - PRIMARY KEY ("HotelId") +CREATE TABLE Hotels ( + hotel_id INTEGER PRIMARY KEY, + hotel_name TEXT, + hotel_description TEXT, + hotel_description_embedding VECTOR(4) ); ``` +## Using with Entra Authentication + +Azure Database for PostgreSQL provides the ability to connect to your database using [Entra authentication](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-azure-ad-authentication). +This removes the need to store a username and password in your connection string. +To use Entra authentication for an Azure DB for PostgreSQL database, you can use the following Npgsql extension method and set a connection string that does not have a username or password: + +```csharp +using System.Text; +using System.Text.Json; +using Azure.Core; +using Azure.Identity; +using Npgsql; + +namespace Npgsql; + +public static class NpgsqlDataSourceBuilderExtensions +{ + private static readonly TokenRequestContext s_azureDBForPostgresTokenRequestContext = new([Constants.AzureDBForPostgresScope]); + + public static NpgsqlDataSourceBuilder UseEntraAuthentication(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default) + { + credential ??= new DefaultAzureCredential(); + + if (dataSourceBuilder.ConnectionStringBuilder.Username == null) + { + var token = credential.GetToken(s_azureDBForPostgresTokenRequestContext, default); + SetUsernameFromToken(dataSourceBuilder, token.Token); + } + + SetPasswordProvider(dataSourceBuilder, credential, s_azureDBForPostgresTokenRequestContext); + + return dataSourceBuilder; + } + + public static async Task UseEntraAuthenticationAsync(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default, CancellationToken cancellationToken = default) + { + credential ??= new DefaultAzureCredential(); + + if (dataSourceBuilder.ConnectionStringBuilder.Username == null) + { + var token = await credential.GetTokenAsync(s_azureDBForPostgresTokenRequestContext, cancellationToken).ConfigureAwait(false); + SetUsernameFromToken(dataSourceBuilder, token.Token); + } + + SetPasswordProvider(dataSourceBuilder, credential, s_azureDBForPostgresTokenRequestContext); + + return dataSourceBuilder; + } + + private static void SetPasswordProvider(NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential credential, TokenRequestContext tokenRequestContext) + { + dataSourceBuilder.UsePasswordProvider(_ => + { + var token = credential.GetToken(tokenRequestContext, default); + return token.Token; + }, async (_, ct) => + { + var token = await credential.GetTokenAsync(tokenRequestContext, ct).ConfigureAwait(false); + return token.Token; + }); + } + + private static void SetUsernameFromToken(NpgsqlDataSourceBuilder dataSourceBuilder, string token) + { + var username = TryGetUsernameFromToken(token); + + if (username != null) + { + dataSourceBuilder.ConnectionStringBuilder.Username = username; + } + else + { + throw new Exception("Could not determine username from token claims"); + } + } + + private static string? TryGetUsernameFromToken(string jwtToken) + { + // Split the token into its parts (Header, Payload, Signature) + var tokenParts = jwtToken.Split('.'); + if (tokenParts.Length != 3) + { + return null; + } + + // The payload is the second part, Base64Url encoded + var payload = tokenParts[1]; + + // Add padding if necessary + payload = AddBase64Padding(payload); + + // Decode the payload from Base64Url + var decodedBytes = Convert.FromBase64String(payload); + var decodedPayload = Encoding.UTF8.GetString(decodedBytes); + + // Parse the decoded payload as JSON + var payloadJson = JsonSerializer.Deserialize(decodedPayload); + + // Try to get the username from 'upn', 'preferred_username', or 'unique_name' claims + if (payloadJson.TryGetProperty("upn", out var upn)) + { + return upn.GetString(); + } + else if (payloadJson.TryGetProperty("preferred_username", out var preferredUsername)) + { + return preferredUsername.GetString(); + } + else if (payloadJson.TryGetProperty("unique_name", out var uniqueName)) + { + return uniqueName.GetString(); + } + + return null; + } + + private static string AddBase64Padding(string base64) => (base64.Length % 4) switch + { + 2 => base64 + "==", + 3 => base64 + "=", + _ => base64, + }; +} +``` + +Now you can use the `UseEntraAuthentication` method to set up the connection string for the Postgres connector: + +```csharp +var dataSourceBuilder = new NpgsqlDataSourceBuilder(connectionString); +dataSourceBuilder.UseEntraAuthentication(); +dataSourceBuilder.UseVector(); +using var datasource = dataSourceBuilder.Build(); + +var vectorStore = new PostgresVectorStore(dataSource); +``` + +By default, the `UseEntraAuthentication` method uses the [DefaultAzureCredential](https://learn.microsoft.com/en-us/dotnet/api/azure.identity.defaultazurecredential?view=azure-dotnet) to authenticate with Azure AD. You can also provide a custom `TokenCredential` implementation if needed. + ::: zone-end ::: zone pivot="programming-language-python" -## Coming soon +## Getting started + +Install semantic kernel with the Postgres extras, which includes the [Postgres client](https://github.com/Postgres/Postgres-client). + +```cli +pip install semantic-kernel[postgres] +``` + +You can then create a vector store instance using the `PostgresStore` class. +You can pass a `psycopg_pool` [AsyncConnectionPool](https://www.psycopg.org/psycopg3/docs/api/pool.html#psycopg_pool.AsyncConnectionPool) directly, +or use the `PostgresSettings` to create a connection pool from environment variables. + +```python + +from semantic_kernel.connectors.memory.Postgres import PostgresStore, PostgresSettings -More info coming soon. +settings = PostgresSettings() +with settings.create_connection_pool() as pool: + vector_store = PostgresStore(pool=pool) + ... +``` + +You can set `POSTGRES_CONNECTION_STRING` in your environment, or the environment variables `PGHOST`, `PGPORT`, `PGUSER`, `PGPASSWORD`, `PGDATABASE`, and optionally `PGSSLMODE` as defined for [libpq](https://www.postgresql.org/docs/current/libpq-envars.html). +These values will be used by the `PostgresSettings` class to create a connection pool. + +You can also create a collection directly. The Collection itself is a context manager, so you can use it in a `with` block. If you don't pass +in a connection pool, the collection will create one using the `PostgresSettings` class. + +```python +from semantic_kernel.connectors.memory.Postgres import PostgresCollection + +collection = PostgresCollection(collection_name="skhotels", data_model_type=Hotel) +with collection: # This will create a connection pool using PostgresSettings + ... +``` + +## Data mapping + +The Postgres connector provides a default mapper when mapping data from the data model to storage. +The default mapper uses the model annotations or record definition to determine the type of each property and to map the model +into a `dict` that can be serialized to Postgres rows. + +- The data model property annotated as a key will be mapped to the PRIMARY KEY in the Postgres table. +- The data model properties annotated as data will be mapped to a table column in Postgres. +- The data model properties annotated as vectors will be mapped to a table column that has the pgvector `VECTOR` type in Postgres. + +```python +from pydantic import BaseModel + +from semantic_kernel.data import ( + DistanceFunction, + IndexKind, + VectorStoreRecordDataField, + VectorStoreRecordKeyField, + VectorStoreRecordVectorField, + vectorstoremodel, +) + +@vectorstoremodel +class Hotel(BaseModel): + hotel_id: Annotated[int, VectorStoreRecordKeyField()] + hotel_name: Annotated[str, VectorStoreRecordDataField()] + hotel_description: Annotated[str, VectorStoreRecordDataField(has_embedding=True, embedding_property_name="description_embedding")] + description_embedding: Annotated[ + list[float] | None, + VectorStoreRecordVectorField( + index_kind=IndexKind.HNSW, + dimensions=4, + distance_function=DistanceFunction.COSINE_SIMILARITY, + ), + ] = None +``` + +```sql +CREATE TABLE Hotels ( + hotel_id INTEGER PRIMARY KEY, + hotel_name TEXT, + hotel_description TEXT, + description_embedding VECTOR(4) +); +``` + +## Using with Entra Authentication + +Azure Database for PostgreSQL provides the ability to connect to your database using [Entra authentication](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-azure-ad-authentication). +This removes the need to store a username and password in your connection string. +To use Entra authentication for an Azure DB for PostgreSQL database, you can use the following custom AsyncConnection class: + +```python +import base64 +import json +import logging +from functools import lru_cache + +from azure.core.credentials import TokenCredential +from azure.core.credentials_async import AsyncTokenCredential +from azure.identity import DefaultAzureCredential +from psycopg import AsyncConnection + +AZURE_DB_FOR_POSTGRES_SCOPE = "https://ossrdbms-aad.database.windows.net/.default" + +logger = logging.getLogger(__name__) + + +async def get_entra_token_async(credential: AsyncTokenCredential) -> str: + """Get the password from Entra using the provided credential.""" + logger.info("Acquiring Entra token for postgres password") + + async with credential: + cred = await credential.get_token(AZURE_DB_FOR_POSTGRES_SCOPE) + return cred.token + + +def get_entra_token(credential: TokenCredential | None) -> str: + """Get the password from Entra using the provided credential.""" + logger.info("Acquiring Entra token for postgres password") + credential = credential or get_default_azure_credentials() + + return credential.get_token(AZURE_DB_FOR_POSTGRES_SCOPE).token + + +@lru_cache(maxsize=1) +def get_default_azure_credentials() -> DefaultAzureCredential: + """Get the default Azure credentials. + + This method caches the credentials to avoid creating new instances. + """ + return DefaultAzureCredential() + + +def decode_jwt(token): + """Decode the JWT payload to extract claims.""" + payload = token.split(".")[1] + padding = "=" * (4 - len(payload) % 4) + decoded_payload = base64.urlsafe_b64decode(payload + padding) + return json.loads(decoded_payload) + + +async def get_entra_conninfo(credential: TokenCredential | AsyncTokenCredential | None) -> dict[str, str]: + """Fetches a token returns the username and token.""" + # Fetch a new token and extract the username + if isinstance(credential, AsyncTokenCredential): + token = await get_entra_token_async(credential) + else: + token = get_entra_token(credential) + claims = decode_jwt(token) + username = claims.get("upn") or claims.get("preferred_username") or claims.get("unique_name") + if not username: + raise ValueError("Could not extract username from token. Have you logged in?") + + return {"user": username, "password": token} + + +class AsyncEntraConnection(AsyncConnection): + """Asynchronous connection class for using Entra auth with Azure DB for PostgreSQL.""" + + @classmethod + async def connect(cls, *args, **kwargs): + """Establish an asynchronous connection using Entra auth with Azure DB for PostgreSQL.""" + credential = kwargs.pop("credential", None) + if credential and not isinstance(credential, (TokenCredential, AsyncTokenCredential)): + raise ValueError("credential must be a TokenCredential or AsyncTokenCredential") + if not kwargs.get("user") or not kwargs.get("password"): + credential = credential or get_default_azure_credentials() + entra_conninfo = await get_entra_conninfo(credential) + if kwargs.get("user"): + entra_conninfo.pop("user", None) + kwargs.update(entra_conninfo) + return await super().connect(*args, **kwargs) +``` + +You can use the custom connection class with the `PostgresSettings.get_connection_pool` method to create a connection pool. + +```python +from semantic_kernel.connectors.memory.Postgres import PostgresSettings, PostgresStore + +settings = PostgresSettings(connection_class=AsyncEntraConnection) +with settings.create_connection_pool() as pool: + vector_store = PostgresStore(pool=pool) + ... +``` + +By default, the `AsyncEntraConnection` class uses the [DefaultAzureCredential](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) to authenticate with Azure AD. +You can also provide another `TokenCredential` in the kwargs if needed: + +```python +from azure.identity import ManagedIdentityCredential + +settings = PostgresSettings(connection_class=AsyncEntraConnection, credential=ManagedIdentityCredential()) +with settings.create_connection_pool() as pool: + vector_store = PostgresStore(pool=pool) + ... +``` ::: zone-end ::: zone pivot="programming-language-java" From a05198bb19abc449728e10f1176714252f1b95d0 Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Mon, 10 Feb 2025 19:19:30 -0500 Subject: [PATCH 03/19] Address link feedback from leran-build-service-prod --- .../out-of-the-box-connectors/postgres-connector.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index ed9d9394..a0ba52c9 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -169,7 +169,7 @@ CREATE TABLE Hotels ( ## Using with Entra Authentication -Azure Database for PostgreSQL provides the ability to connect to your database using [Entra authentication](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-azure-ad-authentication). +Azure Database for PostgreSQL provides the ability to connect to your database using [Entra authentication](/azure/postgresql/flexible-server/concepts-azure-ad-authentication). This removes the need to store a username and password in your connection string. To use Entra authentication for an Azure DB for PostgreSQL database, you can use the following Npgsql extension method and set a connection string that does not have a username or password: @@ -302,7 +302,7 @@ using var datasource = dataSourceBuilder.Build(); var vectorStore = new PostgresVectorStore(dataSource); ``` -By default, the `UseEntraAuthentication` method uses the [DefaultAzureCredential](https://learn.microsoft.com/en-us/dotnet/api/azure.identity.defaultazurecredential?view=azure-dotnet) to authenticate with Azure AD. You can also provide a custom `TokenCredential` implementation if needed. +By default, the `UseEntraAuthentication` method uses the [DefaultAzureCredential](/dotnet/api/azure.identity.defaultazurecredential) to authenticate with Azure AD. You can also provide a custom `TokenCredential` implementation if needed. ::: zone-end ::: zone pivot="programming-language-python" @@ -391,7 +391,7 @@ CREATE TABLE Hotels ( ## Using with Entra Authentication -Azure Database for PostgreSQL provides the ability to connect to your database using [Entra authentication](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-azure-ad-authentication). +Azure Database for PostgreSQL provides the ability to connect to your database using [Entra authentication](/azure/postgresql/flexible-server/concepts-azure-ad-authentication). This removes the need to store a username and password in your connection string. To use Entra authentication for an Azure DB for PostgreSQL database, you can use the following custom AsyncConnection class: @@ -489,7 +489,7 @@ with settings.create_connection_pool() as pool: ... ``` -By default, the `AsyncEntraConnection` class uses the [DefaultAzureCredential](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python) to authenticate with Azure AD. +By default, the `AsyncEntraConnection` class uses the [DefaultAzureCredential](/python/api/azure-identity/azure.identity.defaultazurecredential) to authenticate with Azure AD. You can also provide another `TokenCredential` in the kwargs if needed: ```python From c5ccebea0be236edaea82206291eb01b5fe4917b Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:13:37 -0400 Subject: [PATCH 04/19] Remove reference of KernelBuilder extension --- .../postgres-connector.md | 37 +++---------------- 1 file changed, 6 insertions(+), 31 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index a0ba52c9..a1fa8d38 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -41,23 +41,13 @@ Add the Postgres Vector Store connector NuGet package to your project. dotnet add package Microsoft.SemanticKernel.Connectors.Postgres --prerelease ``` -You can add the vector store to the dependency injection container available on the `KernelBuilder` or to the `IServiceCollection` dependency injection container using extension methods provided by Semantic Kernel. +You can add the vector store to the `IServiceCollection` dependency injection container using extension methods provided by Semantic Kernel. ```csharp using Microsoft.SemanticKernel; -// Using Kernel Builder. -var kernelBuilder = Kernel - .CreateBuilder() - .AddPostgresVectorStore(""); -``` - -```csharp -using Microsoft.SemanticKernel; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -builder.Services.AddPostgresVectorStore(""); +var kernelBuilder = Kernel.CreateBuilder(); +kernelBuilder.Services.AddPostgresVectorStore(""); ``` Where `` is a connection string to the Postgres instance, in the format that [Npgsql](https://www.npgsql.org/) expects. @@ -69,29 +59,14 @@ using Microsoft.Extensions.DependencyInjection; using Microsoft.SemanticKernel; using Npgsql; -// Using Kernel Builder. var kernelBuilder = Kernel.CreateBuilder(); -kernelBuilder.Services.AddSingleton(sp => { - NpgsqlDataSourceBuilder dataSourceBuilder = new(""); - dataSourceBuilder.UseVector(); - return dataSourceBuilder.Build(); -}); -kernelBuilder.AddPostgresVectorStore(); -``` - -```csharp -using Microsoft.Extensions.DependencyInjection; -using Microsoft.SemanticKernel; -using Npgsql; - -// Using IServiceCollection with ASP.NET Core. -var builder = WebApplication.CreateBuilder(args); -kernelBuilder.Services.AddSingleton(sp => { +kernelBuilder.Services.AddSingleton(sp => +{ NpgsqlDataSourceBuilder dataSourceBuilder = new(""); dataSourceBuilder.UseVector(); return dataSourceBuilder.Build(); }); -builder.Services.AddPostgresVectorStore(); +kernelBuilder.Services.AddPostgresVectorStore(); ``` You can construct a Postgres Vector Store instance directly. From 0c6a9c279f758f79c2a684a19219ee6cf5c1a43c Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:13:49 -0400 Subject: [PATCH 05/19] Add example of connection string --- .../out-of-the-box-connectors/postgres-connector.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index a1fa8d38..8f80c34a 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -50,7 +50,7 @@ var kernelBuilder = Kernel.CreateBuilder(); kernelBuilder.Services.AddPostgresVectorStore(""); ``` -Where `` is a connection string to the Postgres instance, in the format that [Npgsql](https://www.npgsql.org/) expects. +Where `` is a connection string to the Postgres instance, in the format that [Npgsql](https://www.npgsql.org/) expects, for example `Host=localhost;Port=5432;Database=postgres;Username=postgres;Password=postgres`. Extension methods that take no parameters are also provided. These require an instance of [NpgsqlDataSource](https://www.npgsql.org/doc/api/Npgsql.NpgsqlDataSource.html) to be separately registered with the dependency injection container. Note that "UseVector" must be called on the builder to enable vector support via [pgvector-dotnet](https://github.com/pgvector/pgvector-dotnet?tab=readme-ov-file#npgsql-c): From f48c1a727b88d37549f8619ea562841366d20f3f Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:14:06 -0400 Subject: [PATCH 06/19] Fix client name --- .../out-of-the-box-connectors/postgres-connector.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 8f80c34a..2f25477a 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -73,7 +73,7 @@ You can construct a Postgres Vector Store instance directly. ```csharp using Microsoft.SemanticKernel.Connectors.Postgres; -using Postgres.Client; +using Npgsql; NpgsqlDataSourceBuilder dataSourceBuilder = new(""); dataSourceBuilder.UseVector(); @@ -85,7 +85,7 @@ It is possible to construct a direct reference to a named collection. ```csharp using Microsoft.SemanticKernel.Connectors.Postgres; -using Postgres.Client; +using Npgsql; NpgsqlDataSourceBuilder dataSourceBuilder = new(""); dataSourceBuilder.UseVector(); From b7fbf8c8da3a9b15138b859d3482b8b856088fa4 Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:14:42 -0400 Subject: [PATCH 07/19] Fix generics --- .../out-of-the-box-connectors/postgres-connector.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 2f25477a..7b104d2c 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -90,7 +90,7 @@ using Npgsql; NpgsqlDataSourceBuilder dataSourceBuilder = new(""); dataSourceBuilder.UseVector(); using NpgsqlDataSource dataSource = dataSourceBuilder.Build(); -var collection = new PostgresVectorStoreRecordCollection(dataSource, "skhotels"); +var collection = new PostgresVectorStoreRecordCollection(dataSource, "skhotels"); ``` ## Data mapping @@ -120,7 +120,7 @@ using Microsoft.Extensions.VectorData; public class Hotel { [VectorStoreRecordKey(StoragePropertyName = "hotel_id")] - public ulong HotelId { get; set; } + public int HotelId { get; set; } [VectorStoreRecordData(StoragePropertyName = "hotel_name")] public string HotelName { get; set; } From 146d9a2feeb4ebd4d3da5e64802f94ed3faf283d Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:15:20 -0400 Subject: [PATCH 08/19] Update create table statement --- .../out-of-the-box-connectors/postgres-connector.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 7b104d2c..57d9da6e 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -134,8 +134,8 @@ public class Hotel ``` ```sql -CREATE TABLE Hotels ( - hotel_id INTEGER PRIMARY KEY, +CREATE TABLE IF NOT EXISTS public."Hotels" ( + "hotel_id" INTEGER PRIMARY KEY NOT NULL, hotel_name TEXT, hotel_description TEXT, hotel_description_embedding VECTOR(4) From fddbe9653466a28cfe396b6fa97558569b53b4a9 Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:15:30 -0400 Subject: [PATCH 09/19] Fix missing using in example --- .../out-of-the-box-connectors/postgres-connector.md | 1 + 1 file changed, 1 insertion(+) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 57d9da6e..468f892f 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -115,6 +115,7 @@ The property name override is done by setting the `StoragePropertyName` option v Here is an example of a data model with `StoragePropertyName` set on its attributes and how that will be represented in Postgres as a table, assuming the Collection name is `Hotels`. ```csharp +using System; using Microsoft.Extensions.VectorData; public class Hotel From 03e74d97a39e05d3d9e2d82185e82043390e5a0a Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:16:06 -0400 Subject: [PATCH 10/19] Fix dotnet entra auth --- .../out-of-the-box-connectors/postgres-connector.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 468f892f..f17ea67a 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -156,11 +156,11 @@ using Azure.Core; using Azure.Identity; using Npgsql; -namespace Npgsql; +namespace Program; public static class NpgsqlDataSourceBuilderExtensions { - private static readonly TokenRequestContext s_azureDBForPostgresTokenRequestContext = new([Constants.AzureDBForPostgresScope]); + private static readonly TokenRequestContext s_azureDBForPostgresTokenRequestContext = new(["https://ossrdbms-aad.database.windows.net/.default"]); public static NpgsqlDataSourceBuilder UseEntraAuthentication(this NpgsqlDataSourceBuilder dataSourceBuilder, TokenCredential? credential = default) { From d514fa648da4f9e83b136d6f2e04d9a54c0ec52b Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:16:27 -0400 Subject: [PATCH 11/19] dotnet doc small fix --- .../out-of-the-box-connectors/postgres-connector.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index f17ea67a..8c338cc5 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -52,7 +52,7 @@ kernelBuilder.Services.AddPostgresVectorStore(""); Where `` is a connection string to the Postgres instance, in the format that [Npgsql](https://www.npgsql.org/) expects, for example `Host=localhost;Port=5432;Database=postgres;Username=postgres;Password=postgres`. -Extension methods that take no parameters are also provided. These require an instance of [NpgsqlDataSource](https://www.npgsql.org/doc/api/Npgsql.NpgsqlDataSource.html) to be separately registered with the dependency injection container. Note that "UseVector" must be called on the builder to enable vector support via [pgvector-dotnet](https://github.com/pgvector/pgvector-dotnet?tab=readme-ov-file#npgsql-c): +Extension methods that take no parameters are also provided. These require an instance of [NpgsqlDataSource](https://www.npgsql.org/doc/api/Npgsql.NpgsqlDataSource.html) to be separately registered with the dependency injection container. Note that `UseVector` must be called on the builder to enable vector support via [pgvector-dotnet](https://github.com/pgvector/pgvector-dotnet?tab=readme-ov-file#npgsql-c): ```csharp using Microsoft.Extensions.DependencyInjection; @@ -112,7 +112,7 @@ property names on the data model. The property name override is done by setting the `StoragePropertyName` option via the data model attributes or record definition. -Here is an example of a data model with `StoragePropertyName` set on its attributes and how that will be represented in Postgres as a table, assuming the Collection name is `Hotels`. +Here is an example of a data model with `StoragePropertyName` set on its attributes and how it will be represented in Postgres as a table, assuming the Collection name is `Hotels`. ```csharp using System; From fca6d8d0e669fbc4ea22aa9fc295389d99d6bbea Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:16:42 -0400 Subject: [PATCH 12/19] Add vector index section --- .../postgres-connector.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 8c338cc5..09929a3d 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -143,6 +143,15 @@ CREATE TABLE IF NOT EXISTS public."Hotels" ( ); ``` +## Vector indexing + +The `hotel_description_embedding` in the above `Hotel` model is a vector property with `IndexKind.HNSW` +indexing. This index will be created automatically when the collection is created. +HNSW is the only index type supported for index creation. IVFFlat index building requires that data already +exist in the table at index creation time, and so it is not appropriate for the creation of an empty table. +You are free to create and modify indexes on tables outside of the connector, which will +be used by the connector when performing queries. + ## Using with Entra Authentication Azure Database for PostgreSQL provides the ability to connect to your database using [Entra authentication](/azure/postgresql/flexible-server/concepts-azure-ad-authentication). @@ -365,6 +374,15 @@ CREATE TABLE Hotels ( ); ``` +## Vector indexing + +The `hotel_description_embedding` in the above `Hotel` model is a vector property with `IndexKind.HNSW` +indexing. This index will be created automatically when the collection is created. +HNSW is the only index type supported for index creation. IVFFlat index building requires that data already +exist in the table at index creation time, and so it is not appropriate for the creation of an empty table. +You are free to create and modify indexes on tables outside of the connector, which will +be used by the connector when performing queries. + ## Using with Entra Authentication Azure Database for PostgreSQL provides the ability to connect to your database using [Entra authentication](/azure/postgresql/flexible-server/concepts-azure-ad-authentication). From 4c92e7665b84f08a1424a9d2070e0f5623e551a7 Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:17:21 -0400 Subject: [PATCH 13/19] Fix incorrect Python examples --- .../postgres-connector.md | 34 +++++++++++-------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 09929a3d..d807fcd4 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -305,12 +305,13 @@ You can pass a `psycopg_pool` [AsyncConnectionPool](https://www.psycopg.org/psyc or use the `PostgresSettings` to create a connection pool from environment variables. ```python - -from semantic_kernel.connectors.memory.Postgres import PostgresStore, PostgresSettings +from semantic_kernel.connectors.memory.postgres import PostgresStore, PostgresSettings settings = PostgresSettings() -with settings.create_connection_pool() as pool: - vector_store = PostgresStore(pool=pool) + +pool = await settings.create_connection_pool() +async with pool: + vector_store = PostgresStore(connection_pool=pool) ... ``` @@ -321,10 +322,10 @@ You can also create a collection directly. The Collection itself is a context ma in a connection pool, the collection will create one using the `PostgresSettings` class. ```python -from semantic_kernel.connectors.memory.Postgres import PostgresCollection +from semantic_kernel.connectors.memory.postgres import PostgresCollection collection = PostgresCollection(collection_name="skhotels", data_model_type=Hotel) -with collection: # This will create a connection pool using PostgresSettings +async with collection: # This will create a connection pool using PostgresSettings ... ``` @@ -341,6 +342,7 @@ into a `dict` that can be serialized to Postgres rows. ```python from pydantic import BaseModel +from semantic_kernel.connectors.memory.postgres import PostgresCollection from semantic_kernel.data import ( DistanceFunction, IndexKind, @@ -475,11 +477,12 @@ class AsyncEntraConnection(AsyncConnection): You can use the custom connection class with the `PostgresSettings.get_connection_pool` method to create a connection pool. ```python -from semantic_kernel.connectors.memory.Postgres import PostgresSettings, PostgresStore +from semantic_kernel.connectors.memory.postgres import PostgresSettings, PostgresStore + -settings = PostgresSettings(connection_class=AsyncEntraConnection) -with settings.create_connection_pool() as pool: - vector_store = PostgresStore(pool=pool) +pool = await PostgresSettings().create_connection_pool(connection_class=AsyncEntraConnection) +async with pool: + vector_store = PostgresStore(connection_pool=pool) ... ``` @@ -487,11 +490,14 @@ By default, the `AsyncEntraConnection` class uses the [DefaultAzureCredential](/ You can also provide another `TokenCredential` in the kwargs if needed: ```python -from azure.identity import ManagedIdentityCredential +from azure.identity import ManagedIdentityCredential -settings = PostgresSettings(connection_class=AsyncEntraConnection, credential=ManagedIdentityCredential()) -with settings.create_connection_pool() as pool: - vector_store = PostgresStore(pool=pool) + +pool = await PostgresSettings().create_connection_pool( + connection_class=AsyncEntraConnection, credential=ManagedIdentityCredential() +) +async with pool: + vector_store = PostgresStore(connection_pool=pool) ... ``` From 08afa2fc64b87e169b055bb7a2de04fc9d988ea7 Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:17:49 -0400 Subject: [PATCH 14/19] python: Improve data mapping section --- .../postgres-connector.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index d807fcd4..a1c4183e 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -356,8 +356,8 @@ from semantic_kernel.data import ( class Hotel(BaseModel): hotel_id: Annotated[int, VectorStoreRecordKeyField()] hotel_name: Annotated[str, VectorStoreRecordDataField()] - hotel_description: Annotated[str, VectorStoreRecordDataField(has_embedding=True, embedding_property_name="description_embedding")] - description_embedding: Annotated[ + hotel_description: Annotated[str, VectorStoreRecordDataField(has_embedding=True, embedding_property_name="hotel_description_embedding")] + hotel_description_embedding: Annotated[ list[float] | None, VectorStoreRecordVectorField( index_kind=IndexKind.HNSW, @@ -365,14 +365,19 @@ class Hotel(BaseModel): distance_function=DistanceFunction.COSINE_SIMILARITY, ), ] = None + +collection = PostgresCollection(collection_name="Hotels", data_model_type=Hotel) + +async with collection: + await collection.create_collection_if_not_exists() ``` ```sql -CREATE TABLE Hotels ( - hotel_id INTEGER PRIMARY KEY, - hotel_name TEXT, - hotel_description TEXT, - description_embedding VECTOR(4) +CREATE TABLE IF NOT EXISTS public."Hotels" ( + "hotel_id" INTEGER PRIMARY KEY NOT NULL, + "hotel_name" TEXT, + "hotel_description" TEXT, + "hotel_description_embedding" VECTOR(4) ); ``` From 0c40d8b2ad42b4837e4909fd65e339e2c84e0890 Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:18:05 -0400 Subject: [PATCH 15/19] formatting --- .../out-of-the-box-connectors/postgres-connector.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index a1c4183e..ebc09c0f 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -411,7 +411,6 @@ AZURE_DB_FOR_POSTGRES_SCOPE = "https://ossrdbms-aad.database.windows.net/.defaul logger = logging.getLogger(__name__) - async def get_entra_token_async(credential: AsyncTokenCredential) -> str: """Get the password from Entra using the provided credential.""" logger.info("Acquiring Entra token for postgres password") @@ -420,7 +419,6 @@ async def get_entra_token_async(credential: AsyncTokenCredential) -> str: cred = await credential.get_token(AZURE_DB_FOR_POSTGRES_SCOPE) return cred.token - def get_entra_token(credential: TokenCredential | None) -> str: """Get the password from Entra using the provided credential.""" logger.info("Acquiring Entra token for postgres password") @@ -428,7 +426,6 @@ def get_entra_token(credential: TokenCredential | None) -> str: return credential.get_token(AZURE_DB_FOR_POSTGRES_SCOPE).token - @lru_cache(maxsize=1) def get_default_azure_credentials() -> DefaultAzureCredential: """Get the default Azure credentials. @@ -437,7 +434,6 @@ def get_default_azure_credentials() -> DefaultAzureCredential: """ return DefaultAzureCredential() - def decode_jwt(token): """Decode the JWT payload to extract claims.""" payload = token.split(".")[1] @@ -445,7 +441,6 @@ def decode_jwt(token): decoded_payload = base64.urlsafe_b64decode(payload + padding) return json.loads(decoded_payload) - async def get_entra_conninfo(credential: TokenCredential | AsyncTokenCredential | None) -> dict[str, str]: """Fetches a token returns the username and token.""" # Fetch a new token and extract the username @@ -460,7 +455,6 @@ async def get_entra_conninfo(credential: TokenCredential | AsyncTokenCredential return {"user": username, "password": token} - class AsyncEntraConnection(AsyncConnection): """Asynchronous connection class for using Entra auth with Azure DB for PostgreSQL.""" From 1c3f6e07f366287a59caef113a93305512c4a5c8 Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:22:09 -0400 Subject: [PATCH 16/19] dotnet: fix var name --- .../out-of-the-box-connectors/postgres-connector.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index ebc09c0f..c64b6075 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -282,7 +282,7 @@ Now you can use the `UseEntraAuthentication` method to set up the connection str var dataSourceBuilder = new NpgsqlDataSourceBuilder(connectionString); dataSourceBuilder.UseEntraAuthentication(); dataSourceBuilder.UseVector(); -using var datasource = dataSourceBuilder.Build(); +using var dataSource = dataSourceBuilder.Build(); var vectorStore = new PostgresVectorStore(dataSource); ``` From 9c2a63faa6708b8ff69855898e5164f8fa5ffccb Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:28:03 -0400 Subject: [PATCH 17/19] Give example of entra conn str --- .../out-of-the-box-connectors/postgres-connector.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index c64b6075..e754d31e 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -279,6 +279,9 @@ public static class NpgsqlDataSourceBuilderExtensions Now you can use the `UseEntraAuthentication` method to set up the connection string for the Postgres connector: ```csharp +using Microsoft.SemanticKernel.Connectors.Postgres; + +var connectionString = "Host=mydb.postgres.database.azure.com;Port=5432;Database=postgres;SSL Mode=Require;"; // No Username or Password var dataSourceBuilder = new NpgsqlDataSourceBuilder(connectionString); dataSourceBuilder.UseEntraAuthentication(); dataSourceBuilder.UseVector(); From a7a747731334c9b634c80a8fc54f88ca7701b6c0 Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Tue, 11 Mar 2025 10:50:48 -0400 Subject: [PATCH 18/19] dotnet: Update CREATE TABLE --- .../out-of-the-box-connectors/postgres-connector.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index e754d31e..27f6a5e1 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -377,13 +377,15 @@ async with collection: ```sql CREATE TABLE IF NOT EXISTS public."Hotels" ( - "hotel_id" INTEGER PRIMARY KEY NOT NULL, + "hotel_id" INTEGER NOT NULL, "hotel_name" TEXT, "hotel_description" TEXT, "hotel_description_embedding" VECTOR(4) + PRIMARY KEY ("hotel_id") ); ``` + ## Vector indexing The `hotel_description_embedding` in the above `Hotel` model is a vector property with `IndexKind.HNSW` From a2ee2a86764e808971aeccb63660d5613991ad55 Mon Sep 17 00:00:00 2001 From: Rob Emanuele Date: Fri, 21 Mar 2025 13:26:04 -0400 Subject: [PATCH 19/19] Fix mention of named vectors --- .../out-of-the-box-connectors/postgres-connector.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md index 27f6a5e1..d3ed6d9e 100644 --- a/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md +++ b/semantic-kernel/concepts/vector-store-connectors/out-of-the-box-connectors/postgres-connector.md @@ -107,8 +107,9 @@ It's also possible to override the default mapper behavior by providing a custom ### Property name override -For data properties and vector properties (if using named vectors mode), you can provide override field names to use in storage that is different from the -property names on the data model. +You can provide override field names to use in storage that is different from the +property names on the data model. This allows you to match table column names even +if they don't match the property names on the data model. The property name override is done by setting the `StoragePropertyName` option via the data model attributes or record definition.