Can we connect confluent cloud ksqldb instance using this?

[Vasistan]

Am tried connecting with confluent cloud instance with this

` var ksqlDbUrl = @"https://XYZX.aws.confluent.cloud:443";
var contextOptions = new KSqlDBContextOptions(ksqlDbUrl);
contextOptions.SetBasicAuthCredentials("KEY", "PASSWORD");

            await using var context = new KSqlDBContext(contextOptions, _logger);`

This throws unauthorized error. Can someone help me on this. Am trying to query the stream i have created on ksqldb instance

[tomasfabian]

Hi @Vasistan,
I don't know what kind of auth. you would like to use. Could you please try out to set your auth. in the following way?

Http interception with ksqldb .NET client.

Let us know how it went.

Regards Tomas.

[tbraun-hk]

Hi @Vasistan , @tomasfabian,

we're currently trying the same - connection to KSQL provided by Confluent Cloud. We've generated an api-key with corresponding secret.
Did you succeed with a custom http interceptor as described in the documentation?

Regards,
Tobias

[tomasfabian]

Hi @tbraun-hk,
I don't have a Confluent Cloud account, but I assume that you could try the following HTTP auth. header:

Authorization: Basic <BASE64-encoded-key-and-secret>

The above mentioned BASE64-encoded-key-and-secret should be composed in this way:

$KSQLDB_API_KEY:$KSQLDB_API_SECRET

using ksqlDB.RestApi.Client.KSql.Query.Context.Options;

var contextOptions = new KSqlDbContextOptionsBuilder()
  .UseKSqlDb(ksqlDbUrl)
  .SetBasicAuthCredentials(apiKey, apiSecret)
  .Options;

await using var context = new KSqlDBContext(contextOptions);

Check also your ksqldb security settings and try to connect from CLI with the same API secret and key.

If it doesn't work please try to find the corresponding Confluent Cloud documentation for authentication or post a question in Stackoverflow how to POST a query via REST eg POSTMAN.

@Vasistan did you manage to connect to your Confluent Cloud ksqldb instance, please? If yes could you share your approach please?

Thank you Tomas.

[tomasfabian]

@tbraun-hk based on this hands on it is possible to connect to ksqlDB in Confluent Cloud with REST.
Could you please verify your access rights / permissions granted to your role or whether you can query your stream/table with your generated API_KEY?
In the section Provisioning ksqlDB on Confluent Cloud I see that you can use "Granular" or "Global" accesses.
The easiest for you could be to initially try to query your ksqlDB from eg Postman, Fiddler and try out your credentials from there. In case you will succeed you should know how to set up also ksqlDB.RestApi.Client.

Related articles:
ksql-rbac
access a ksqlDB with an Api key
similar question in Stackoverflow

Please let us know if you proceed further.

Thanks a lot! Tomas

[tomasfabian]

[thinnesf]

Hi @Vasistan , @tomasfabian,
I am working together with @tbraun-hk and I tried your workarounds. Fortunately, it is as simple as that.

var httpClientFactory = new HttpClientFactory(new Uri(ksqlDbUrl));

var restApiClient = new KSqlDbRestApiClient(httpClientFactory);
restApiClient.SetCredentials(new BasicAuthCredentials(apiKey, apiSecret));

Thank you for your help!
Franziska

[tomasfabian]

Hi @thinnesf so are you able to use both KSqlDBContext and KSqlDbRestApiClient with the provided SetCredentials methods in the package ksqlDB.RestApi.Client to authenticate against Confluent Cloud?

If yes then that's great to know.

[thinnesf]

Hi @tomasfabian ,
unfortunately, BasicAuth does not work for KSqlDBContext with the given SetBasicAuthCredentials in the ContextOptionsBuilder.

[tomasfabian]

Hi @thinnesf, that is strange. KSqlDBContext and KSqlDbRestApiClient are setting the same auth. header.
I see the following info in Fiddler when I'm trying the sample query with KSqlDBContext locally:

POST http://localhost:8088/query-stream HTTP/1.1
Host: localhost:8088
Accept: application/vnd.ksqlapi.delimited.v1
Authorization: basic ZnJlZDpsZXRtZWlu
Content-Type: application/json; charset=utf-8
Content-Length: 296

{"sql":"SELECT Id, Title, Release_Year, RowTime FROM Movies\r\nWHERE Title != \u0027E.T.\u0027 AND LCASE(Title) LIKE LCASE(\u0027%hard%\u0027) OR (Id = 1) AND RowTime \u003E= 1510923225000 EMIT CHANGES LIMIT 2;","properties":{"auto.offset.reset":"earliest","processing.guarantee":"exactly_once"}}

In case of KSqlDbRestApiClient the outcome is this:

POST http://localhost:8088/ksql HTTP/1.1
Host: localhost:8088
Authorization: basic ZnJlZDpsZXRtZWlu
Accept: application/vnd.ksql.v1+json
Content-Type: application/vnd.ksql.v1+json; charset=utf-8
Content-Length: 281

{"ksql":"CREATE OR REPLACE TABLE lead_actor (\r\n        title VARCHAR PRIMARY KEY,\r\n        actor_name VARCHAR\r\n      ) WITH (\r\n        KAFKA_TOPIC=\u0027lead_actor\u0027,\r\n        PARTITIONS=1,\r\n        VALUE_FORMAT=\u0027JSON\u0027\r\n      );","streamsProperties":{}}

Edit:
Please verify that you are not overriding the default BasicAuthHandler with injecting a different HttpClientFactory if you changed something on the ksqldb server side.

It would be nice to figure this out somehow. thx

[thinnesf]

Hi @tomasfabian ,
Finally, I managed to query streams with the KSqlDBContext. I can authenticate against Confluent Cloud using the ContextOptionsBuilder with the provided SetBasicAuthCredentials method.

Thank you for your help.

[tomasfabian]

Hi @thinnesf,
that is great!

Could you please share what was wrong, please? Did you have to change something (config, permissions) on the server side?

Thanks too.

[thinnesf]

Hi @tomasfabian ,
no, I did not have to change anything. Your code works fine. It was a proxy issue on my side.
My simple test code:

var recordList = new List();
var contextOptions = new KSqlDbContextOptionsBuilder()
.UseKSqlDb(settings.EndpointUrl)
.SetBasicAuthCredentials("KsqlApiKey", "KsqlApiSecret")
.Options;

var context = new KSqlDBContext(contextOptions);
var enumerableResult = context.CreatePullQuery("myrecordstream")
.Where(c => c.name == "test_name").GetManyAsync();

    await foreach (var myRecordItem in enumerableResult.ConfigureAwait(false))
    {
        recordList.Add(myRecordItem);
    }