Evaluating Dekaf and would like to contribute upstream

[kaminski-dev]

Hey @visortelle ,

I'm Daniel, and I work on Raiffeisen Bank International's Apache Pulsar platform. We are evaluating Dekaf for our business administrators and, later, other teams using our Pulsar service.

Dekaf covers an important gap for us. We need a practical way to inspect tenants, namespaces, topics, subscriptions, backlogs, schemas, and message payloads via a UI.

Before we can provide a web UI beyond a small administrator group, we must meet our security requirements:

• authenticate users through our enterprise identity provider
• attribute Pulsar actions to individual users instead of a shared account
• prevent users from accessing tenants and resources they are not entitled to, through both the UI and direct API calls
• separate read access from destructive operations such as deleting topics, clearing backlogs, and changing policies
• build and scan a pinned container image with a defined patching process

Our image build, deployment, monitoring, and approval processes are specific to RBI and will remain internal. The authentication and authorization work may be useful to other Dekaf users.

We have already found one compatibility issue. Our OAuth profile uses the client-credentials grant with the private_key_jwt client-authentication method.

At Dekaf commit 7267f381, Dekaf uses Pulsar Java client 3.2.1. Its OAuth implementation reads client_id and client_secret from a JSON credentials file and sends the secret to the token endpoint. Despite the Dekaf field being named privateKey, it does not appear to create a signed client_assertion.

We are testing this against our identity provider now. If support is missing, we would prefer a provider-neutral solution. The right place for that solution may be Dekaf, the Pulsar Java client, or a separate authentication plugin.

Multi-user access is the larger topic. Hiding tenants in the navigation is not sufficient. The server must enforce access for every resource and operation. We are considering:

• optional user authentication
• server-validated identity mapping or delegation
• tenant and resource-level authorization
• role controls for destructive operations
• negative tests proving that one user cannot access another tenant through navigation, deep links, or direct API requests

We also want to preserve existing deployments. The current single-user setup and defaultPulsarAuth configuration should continue to work. New authentication and authorization behavior should be opt-in for existing installations. Once enabled, multi-user mode must fail closed and must not fall back to shared credentials.

We have imported the public repository into our internal development environment so we can build and test it under our controls. We do not want to maintain a separate RBI edition of Dekaf. Our goal is to stay aligned with your releases, rebuilt and verified through our internal process.

We would keep RBI-specific deployment code and configuration internal. Generic changes would be proposed upstream as focused pull requests with tests and documentation.

Before we change the architecture, I would appreciate your view:

1. Would you accept optional multi-user authentication and tenant-level authorization if existing deployments remain compatible?
2. Is there existing or planned work in this area that we should follow?
3. Would you prefer external OIDC authentication with a defined identity handoff, native OIDC support in Dekaf, or another approach?
4. Where do you think private_key_jwt support belongs?
5. How would you like us to divide the work into issues and pull requests?

We are still testing compatibility. I would rather align now than arrive later with a large patch that takes Dekaf in a direction you do not want.

[visortelle]

Hi @kaminski-dev,

Thank you for the detailed proposal, and especially for checking alignment before investing in a large patch.

For roughly the pasty year, I’ve been developing the similar set of features you just described.
Currently it's a private fork project. You can call it Dekaf EE.

From what I see, all the points you mentioned are satisfied. The auth part is on top of OIDC.
It also includes support for multiple Pulsar instances, improved geo-replication support, Grafana-style Pulsar metric views, audit log and data-masking.

It doesn't break the current Dekaf deployments, as this repo remains Apache 2.0 and will continue to be a perfect fit for single cluster Pulsar deployments, small teams, a well as the useful entrypoint for Pulsar beginners.
Pulsar IO, Functions support and many more are planned for the Dekaf open-source.

My current licensing idea, which is still open to feedback, is:

• free of charge for companies with both fewer than 50 employees and less than US$5 million in annual revenue
• US$2,500 per year for up to ten managed Pulsar clusters for larger organizations

Revenue from the enterprise edition would help fund an additional developer working on the open-source Dekaf project and contributing upstream to Apache Pulsar.

Would you be willing to help turn RBI’s requirements into a prioritized set of scenarios and acceptance criteria?

[visortelle]

@kaminski-dev feel free to contact me at kiryl.valkovich@proton.me

[kaminski-dev]

Hi @visortelle ,

Thank you for the explanation. Dekaf EE sounds very close to what we need.

RBI follows an open-source-first approach for our platform. It is based on OpenShift, Apache Pulsar, Argo CD, Kargo, and other open-source projects. A commercially licensed Dekaf edition is currently not an option for us, so we need to work with the Apache 2.0 version.

We are willing to implement the features ourselves and contribute the generic parts back, including tests and documentation. I am also happy to provide a prioritized list of features, scenarios, and acceptance criteria.

Our strong preference is to work with you and get these changes accepted upstream. We do not want to maintain a separate RBI version. However, if the required features remain exclusive to Dekaf EE and cannot be accepted into the open-source project, we will need to maintain a fork to meet our security requirements.

Before starting implementation, I would like to understand which features or building blocks you would be willing to accept upstream. Once we agree on that boundary, I can share the detailed scenarios and we can split the work into manageable issues and pull requests.

I look forward to finding an approach that works for both sides.

[visortelle]

@kaminski-dev When you’re talking about open-source, you mean that the code should be available and the build is reproducible by some security requirements?
Or you mean it should be free to use without any limitations.

Overall I don’t have objections on PRs if it’s implemented and works correctly. It’s an open-source project.

Have in mind you need to think how to deploy Dekaf in geo-distributed environment and support multiple replicas with blue-green deployments.

Of course I’d rather prefer to have my own fork to be merged to this open-source repository and have the Raiffeisen Bank International in the list of platinum sponsors :)

[kaminski-dev]

I can clarify on my side if this is possible to sponsor support your open source project. I can't promise you anything when it comes to that but let's see.

Overall I am happy to share the journey with you on how we set it up and everything, just let me know if you are interested.

[visortelle]

Sure, I'm interested.

[visortelle]

@kaminski-dev also could you provide a link to your LinkedIn profile with verified job place in the RBI, so I could check that you is actually you.
You can send it to my email if you wish.

I’ll do additional checks here.

Unfortunately, there are lot of bots nowadays.

[kaminski-dev]

Sure, here is my LinkedIn profile https://www.linkedin.com/in/daniel-kaminski-/