Alternatives to Kafka ===================== This page documents constraints when integrating Waterstream with systems that expose Kafka-compatible APIs but are not Apache Kafka (for example, Redpanda or Azure Event Hubs). .. _azure-event-hubs-limitations: Azure Event Hubs ---------------- Azure Event Hubs provides Kafka protocol compatibility, but not full parity with Apache Kafka. The limitations below focus on Waterstream operational behavior. Microsoft references used for this section: - `Apache Kafka protocol support in Azure Event Hubs `_ - `Azure Event Hubs tier comparison `_ - `Azure Event Hubs quotas and limits `_ .. note:: Azure quotas and preview feature availability can change. Validate current limits before production rollout. Known limitations and Waterstream workarounds ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :widths: 19 28 29 24 :header-rows: 1 * - Limitation - Waterstream impact - Recommended workaround/configuration - Tier mitigation * - No message compression support on Event Hubs Kafka endpoint - Higher network and storage consumption when Waterstream publishes high-volume MQTT traffic. - Set ``KAFKA_COMPRESSION_TYPE=none`` to align with Event Hubs behavior and avoid misleading compression expectations in client tuning. - None. Not mitigated by Standard, Premium, or Dedicated. * - No idempotent producer in non-transactional Event Hubs mode - Retries can produce duplicates for at-least-once delivery paths. - Set ``KAFKA_ENABLE_IDEMPOTENCE=false`` and design consumers to be idempotent (deduplicate by message key or application identifier). - Use Premium or Dedicated with transaction-capable mode when strongest QoS 2 behavior is required. * - No Kafka transactions in Standard plan - Strongest transaction-backed QoS 2 guarantees are not available. - Keep ``KAFKA_TRANSACTIONAL_ID=`` (empty/unset) on Standard. If strict QoS 2 consistency is required, move to Premium or Dedicated and configure a stable, unique ``KAFKA_TRANSACTIONAL_ID`` per Waterstream instance. - Premium and Dedicated currently document transaction support as public preview. * - No log compaction - Session and retained-message state topics rely on retention windows instead of compaction, so state can expire and require rebuild/recovery behavior after retention lapses. - Tune time-based retention to exceed expected offline session windows. Document recovery implications (session rehydration and retained-message repopulation) in runbooks. - None. Compaction is not enabled by tier. * - Topic count limits (Standard: 10 Event Hubs per namespace) - Restricts multi-topic routing and environment isolation patterns. Waterstream required topics plus internal topics consume the quota quickly. - Consolidate topic design, avoid unnecessary per-tenant/per-feature topic fan-out, and reserve capacity for internal topics. - Premium and Dedicated provide higher limits. * - Consumer group limits (Standard: 20 per Event Hub) - Limits parallel independent consumers and integration patterns built on multiple consumer groups. - Reuse consumer groups where possible and plan group budgets per workload. - Premium and Dedicated provide higher limits. * - Partition count limits - Upper bound on throughput parallelism and scaling by partitioning. - Size partition counts early, load-test realistic traffic, and scale up tier when partition ceilings are reached. - Premium and Dedicated provide higher partition ceilings. * - Message size limit (1 MB) - Large MQTT payloads can be rejected by Kafka endpoint writes. - Keep payloads below 1 MB including protocol overhead. For larger payloads, publish pointers (for example, object storage URLs) instead of full payload bodies. - None. Treat as a hard design constraint unless Microsoft updates limits. * - Kafka Streams internal topic overhead - Internal changelog/repartition topics count toward Event Hub quotas, reducing available topic budget for user data. - Include internal topics (for example, ``waterstream-kafka-table-mqtt_sessions-changelog``) in namespace topic budget planning. - Premium and Dedicated reduce pressure by offering larger quotas. * - Kafka API version/feature compatibility gaps vs Apache Kafka - Some client API versions/features may be unsupported or partially supported, affecting advanced producer/consumer behaviors. - Validate Waterstream/Kafka-client compatibility against Microsoft Kafka support docs before upgrades. Re-test connectivity and critical flows after broker/client version changes. - Premium and Dedicated improve capacity-related limits, but API compatibility is endpoint-specific, not full Apache Kafka parity. Tier comparison (Standard vs Premium vs Dedicated) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: Azure Event Hubs tier comparison for Waterstream planning :widths: 24 25 25 26 :header-rows: 1 * - Capability - Standard - Premium - Dedicated * - Topics per namespace - 10 Event Hubs per namespace - Higher than Standard (see current Azure quotas) - Highest tier limits (see current Azure quotas) * - Consumer groups per Event Hub - 20 - Higher than Standard (see current Azure quotas) - Highest tier limits (see current Azure quotas) * - Max partitions per Event Hub - Lower ceiling than Premium/Dedicated - Higher ceiling than Standard - Highest ceiling * - Message retention options - Time-based retention - Longer/higher retention options than Standard - Largest retention envelope * - Compression support - No Kafka compression parity - No Kafka compression parity - No Kafka compression parity * - Transaction support - Not available - Available (public preview per Microsoft docs) - Available (public preview per Microsoft docs) Operational guidance ~~~~~~~~~~~~~~~~~~~~ - Always include Event Hubs quotas in capacity planning for both user topics and internal Waterstream topics. - For strict QoS 2 requirements, use Premium or Dedicated and validate current transaction support status. - Keep a regression checklist for Event Hubs API compatibility during Waterstream/Kafka client upgrades.