Alternatives to Kafka

This page documents constraints when integrating Waterstream with systems that expose Kafka-compatible APIs but are not Apache Kafka (for example, Redpanda or Azure Event Hubs).

Azure Event Hubs

Azure Event Hubs provides Kafka protocol compatibility, but not full parity with Apache Kafka. The limitations below focus on Waterstream operational behavior.

Microsoft references used for this section:

• Apache Kafka protocol support in Azure Event Hubs

• Azure Event Hubs tier comparison

• Azure Event Hubs quotas and limits

Note

Azure quotas and preview feature availability can change. Validate current limits before production rollout.

Known limitations and Waterstream workarounds

Limitation	Waterstream impact	Recommended workaround/configuration	Tier mitigation
No message compression support on Event Hubs Kafka endpoint	Higher network and storage consumption when Waterstream publishes high-volume MQTT traffic.	Set KAFKA_COMPRESSION_TYPE=none to align with Event Hubs behavior and avoid misleading compression expectations in client tuning.	None. Not mitigated by Standard, Premium, or Dedicated.
No idempotent producer in non-transactional Event Hubs mode	Retries can produce duplicates for at-least-once delivery paths.	Set KAFKA_ENABLE_IDEMPOTENCE=false and design consumers to be idempotent (deduplicate by message key or application identifier).	Use Premium or Dedicated with transaction-capable mode when strongest QoS 2 behavior is required.
No Kafka transactions in Standard plan	Strongest transaction-backed QoS 2 guarantees are not available.	Keep KAFKA_TRANSACTIONAL_ID= (empty/unset) on Standard. If strict QoS 2 consistency is required, move to Premium or Dedicated and configure a stable, unique KAFKA_TRANSACTIONAL_ID per Waterstream instance.	Premium and Dedicated currently document transaction support as public preview.
No log compaction	Session and retained-message state topics rely on retention windows instead of compaction, so state can expire and require rebuild/recovery behavior after retention lapses.	Tune time-based retention to exceed expected offline session windows. Document recovery implications (session rehydration and retained-message repopulation) in runbooks.	None. Compaction is not enabled by tier.
Topic count limits (Standard: 10 Event Hubs per namespace)	Restricts multi-topic routing and environment isolation patterns. Waterstream required topics plus internal topics consume the quota quickly.	Consolidate topic design, avoid unnecessary per-tenant/per-feature topic fan-out, and reserve capacity for internal topics.	Premium and Dedicated provide higher limits.
Consumer group limits (Standard: 20 per Event Hub)	Limits parallel independent consumers and integration patterns built on multiple consumer groups.	Reuse consumer groups where possible and plan group budgets per workload.	Premium and Dedicated provide higher limits.
Partition count limits	Upper bound on throughput parallelism and scaling by partitioning.	Size partition counts early, load-test realistic traffic, and scale up tier when partition ceilings are reached.	Premium and Dedicated provide higher partition ceilings.
Message size limit (1 MB)	Large MQTT payloads can be rejected by Kafka endpoint writes.	Keep payloads below 1 MB including protocol overhead. For larger payloads, publish pointers (for example, object storage URLs) instead of full payload bodies.	None. Treat as a hard design constraint unless Microsoft updates limits.
Kafka Streams internal topic overhead	Internal changelog/repartition topics count toward Event Hub quotas, reducing available topic budget for user data.	Include internal topics (for example, waterstream-kafka-table-mqtt_sessions-changelog) in namespace topic budget planning.	Premium and Dedicated reduce pressure by offering larger quotas.
Kafka API version/feature compatibility gaps vs Apache Kafka	Some client API versions/features may be unsupported or partially supported, affecting advanced producer/consumer behaviors.	Validate Waterstream/Kafka-client compatibility against Microsoft Kafka support docs before upgrades. Re-test connectivity and critical flows after broker/client version changes.	Premium and Dedicated improve capacity-related limits, but API compatibility is endpoint-specific, not full Apache Kafka parity.

Tier comparison (Standard vs Premium vs Dedicated)

Azure Event Hubs tier comparison for Waterstream planning

Capability	Standard	Premium	Dedicated
Topics per namespace	10 Event Hubs per namespace	Higher than Standard (see current Azure quotas)	Highest tier limits (see current Azure quotas)
Consumer groups per Event Hub	20	Higher than Standard (see current Azure quotas)	Highest tier limits (see current Azure quotas)
Max partitions per Event Hub	Lower ceiling than Premium/Dedicated	Higher ceiling than Standard	Highest ceiling
Message retention options	Time-based retention	Longer/higher retention options than Standard	Largest retention envelope
Compression support	No Kafka compression parity	No Kafka compression parity	No Kafka compression parity
Transaction support	Not available	Available (public preview per Microsoft docs)	Available (public preview per Microsoft docs)

Operational guidance

• Always include Event Hubs quotas in capacity planning for both user topics and internal Waterstream topics.

• For strict QoS 2 requirements, use Premium or Dedicated and validate current transaction support status.

• Keep a regression checklist for Event Hubs API compatibility during Waterstream/Kafka client upgrades.