VALUE_AVRO_SCHEMA_FULL_NAME does not appear to work

[gpaniagua013]

I am trying to set up the schema namespace and name of a topic created by KSQL. Base in the documentation I can do this in the following way:

" By default, KSQL registered avro schemas have the same name (KsqlDataSourceSchema) and the same namespace (io.confluent.ksql.avro_schemas). You can override this behaviour by providing an VALUE_AVRO_SCHEMA_FULL_NAME property in the WITH clause where you set the VALUE_FORMAT to 'AVRO'. As the name suggests, this property will override the default name/namespace with the provided one. For Example, com.mycompany.MySchema registers a schema with the MySchema name and the com.mycompany namespace. "

In am trying to do this:

CREATE STREAM Test_GameStart_Rekeyed_KS WITH (VALUE_FORMAT='AVRO', VALUE_AVRO_SCHEMA_FULL_NAME='com.mycompany.MySchema', PARTITIONS=128) AS SELECT column_sample AS column_sample FROM [TABLE] PARTITION BY key;

And I'm getting this error:

io.confluent.ksql.util.KsqlException: Invalid config variable in the WITH clause: VALUE_AVRO_SCHEMA_FULL_NAME.

[xs005]

I just tried the same command on ksql-server 5.2.2 and it works well. Maybe you need to update ksql-server. Are you familiar with KSQL logging, I am trying to include the row data but cannot make it, need some help.

And I am curious why do you need your own namespace for the avro schema?

[gpaniagua013]

Well, I am using the latest docker image of KSQL. I am not familiar with KSQL logging.

I have a .net consumer and at the moment of the deserialization, I am getting a schema mismatch error, because we have our own namespace for the .net Avro entities. Like a workaround, we had to create the entities in the that KSQL namespace and the same name. But I don't like it.

[xs005]

Try 5.2.2 for testing. I always use the version a little bit earlier.

[gpaniagua013]

Yes in that version is working. Thanks.

[marghidanu]

I tried this option in 5.3.1 and it doesn't seem to work. Is there a way to specify the schema subject name when creating a stream?

[rmoff]

Trying this out in ksqlDB 0.6.

Create two new streams, one using the config, one not:

ksql> CREATE STREAM TEST_AVRO WITH (VALUE_AVRO_SCHEMA_FULL_NAME='FOO', VALUE_FORMAT='AVRO')
AS SELECT * FROM TEST;


 Message
----------------------------------------------------------------------------------------
 Stream TEST_AVRO created and running. Created by query with query ID: CSAS_TEST_AVRO_4
----------------------------------------------------------------------------------------
ksql>
ksql> CREATE STREAM TEST_AVRO2 WITH (VALUE_FORMAT='AVRO') AS SELECT * FROM TEST;

 Message
------------------------------------------------------------------------------------------
 Stream TEST_AVRO2 created and running. Created by query with query ID: CSAS_TEST_AVRO2_5
------------------------------------------------------------------------------------------
ksql>

Both get created with the standard subject name, and VALUE_AVRO_SCHEMA_FULL_NAME is not used:

$ curl -s -XGET localhost:8081/subjects/
["TEST_AVRO2-value","TEST_AVRO-value"]

[big-andy-coates]

FYI, VALUE_AVRO_SCHEMA_FULL_NAME doesn't change the name of the subject in the schema registry, so TEST_AVR02-value and TEST_AVRO-value are what I'd expect to see.

What VALUE_AVRO_SCHEMA_FULL_NAME changes is the name of the record within the schema. So you'd need to pull back the content of both of those schemas to see if there is an actual issue or not.

[msillence]

wiht Server v0.15.0 we find that VALUE_AVRO_SCHEMA_FULL_NAME ='com.company.ClassName" correctly creates the value schema but the key schema ends up as:

  "name": "ClassNameKey",
  "namespace": "io.confluent.ksql.avro_schemas",

we were expecting:

  "name": "ClassNameKey",
  "namespace": "com.company",

similar to how the value works

[waytoalpit]

@msillence Can you please confirm if this is working?

[msillence]

@msillence Can you please confirm if this is working?

I can, it's working well thank you so much for fixing this

[waytoalpit]

Actually I am another developer who is struggling with the issue where avro subject name strategy to record type. I thought this can help provide the avro schema name explicitly.

Can you help me how did you fix it and what was the ksqldb version used?

ksql> CREATE STREAM TESTSTREAM WITH (KAFKA_TOPIC='topic-name', VALUE_FORMAT='AVRO', VALUE_AVRO_SCHEMA_FULL_NAME='com.example.schema'); Schema for message values on topic 'topic-name' does not exist in the Schema Registry. Subject: topic-name-value

[msillence]

We're using release 0.28.2 as I remember this has helped where we had something that was't going to map to the right name. I think we may be in the situation where we automatically create all the schemas without ksql.