Introduce to Reactive Messaging and Reactive Streams ?

[bsideup]

Hi!

Given the rising popularity of reactive APIs (also the Flow API in Java 9), such initiative as OpenMessaging should provide the first-class APIs for the reactive world.

To define the "reactive":

• Standard Publisher/Subscriber API - should be compatible with http://www.reactive-streams.org, so that many existing reactive solutions can easily integrate (RxJava, Akka, Reactor, etc...)
• Push/pull model - at least on end-user API level
• Back-pressure awareness - clients should be able to request their demand and producers should respect it
• [?] Smart resources utilization - ThreadPool-ing, async, non-blocking API

I pre-checked some of the points I think already provided. So far

The proven beauty of reactive APIs is that you can easily convert them to pull/push/streaming APIs, but not the other way around.
That might serve a foundation for the underlying implementations, but it's important to have it as an API, not the detail of implementation.

[zhouxinyu]

Hi thanks for your advice, we may consider providing reactive APIs in openmessaging, but we need more research and discussions.
Currently, we are focusing on spec doc, message model, and non-reactive APIs, any advice is welcome.

[vongosling]

@bsideup Do you have any suggestion for reactive API style, I think it is a high-levelAPI base on our core API now.

[bsideup]

@vongosling I don't think that the reactive APIs can be efficiently implemented on top of the APIs you have atm.

Reactive style has a concept of a dynamic "demand" known upfront.
The closest is StreamingConsumer but it doesn't support the demand, means that the consumer will inefficiency predict the demand and will not be able to react on back-pressure (i.e. loading 10 messages instead of 1000 because the consumer can only consume 10 messages per second)

[harrycodawang]

@vongosling I don't think that the reactive APIs can be efficiently implemented on top of the APIs you have atm.

Reactive style has a concept of a dynamic "demand" known upfront.
The closest is StreamingConsumer but it doesn't support the demand, means that the consumer will inefficiency predict the demand and will not be able to react on back-pressure (i.e. loading 10 messages instead of 1000 because the consumer can only consume 10 messages per second)

We may consider integrating RSocket to OpenMessaging. What do you say?

[bsideup]

@harrycodawang
I think that's a great idea and I'm happy to answer your questions about RSocket if you have any 👍

RSocket fits incredibly well into this story and I already had a very successful experience integrating it into an event gateway:
https://github.com/bsideup/liiklus

[vongosling]

@bsideup @harrycodawang very glad that we could pick up the topic about Reactive Streams :-) RSocket may not be an API level, I am not sure what's your point about here.Also, I have talked about with the micro-profile team from the Eclipse Foundation, who have tried a little bit in here, We could try to define and keep discussing.

[bsideup]

@vongosling RSocket alone may not be, but RSocket RPC provides gRPC-like APIs (also defined as Protobuf services) with the back-pressure support and other nice features

[harrycodawang]

@bsideup Do you any thoughts that can be illustrated by some charts how RSocket can be integrated to OpenMessaging?

Thanks for referring to the project liiklus, could you please help share some illustration about the system architecture with RSocket, liiklus and Event Brokers? One of the questions in my head is that "Does it mean two network connections i.e. a RSocket connection plus a Broker Client/Server connection? Or we can use RSocket to implement the Broker Client/Server protocol, since as you said, reactive APIs have the most expressive foundation to implement all the APIs "

[bsideup]

@harrycodawang

Do you any thoughts that can be illustrated by some charts how RSocket can be integrated to OpenMessaging?

I do, I can draft a simple RSocket RPC example for the OpenMessaging API

"Does it mean two network connections i.e. a RSocket connection plus a Broker Client/Server connection? Or we can use RSocket to implement the Broker Client/Server protocol, since as you said, reactive APIs have the most expressive foundation to implement all the APIs "

An RSocket connection is sufficient to cover the Broker Client/Server protocol.
Think about it as a "TCP with item flow control, not just bytes".

could you please help share some illustration about the system architecture with RSocket, liiklus and Event Brokers

Ok, I'll add it to Liiklus' description to help understand it better.
Basically, Liiklus to Event systems is what API Gateway (let's say, Nginx) to Http servers

[harrycodawang]

@bsideup

Ok, I'll add it to Liiklus' description to help understand it better.
Basically, Liiklus to Event systems is what API Gateway (let's say, Nginx) to Http servers
Yah, that is the question in my head.
In the API Gateway/Nignx case, usually the gateway is transforming the protocols or proxy packages between two connections. So if mapped to Liiklus case, that may mean a RSocket server acting as a broker Client. Let's say it's topology 1.
Or in another topology, RSocket connection is acting as the transport binding for the Broker Client/Server protocol. Let's say it's topology 2.
So Gateway illustrates topology 1 and TCP illustrates topology 1, which makes me a little confused.
But anyway, RSocket can also be a proxy, in which case RSocket is wrapping the Broker client, right?

[bsideup]

@harrycodawang
I see where the confusion comes from. Sorry, I should have said that Liiklus could also be used as a non-transparent proxy with content modifications (e.g. encryption for GDPR), checks (e.g. JSON Schema), re-routing, etc.

Currently, it looks like this:

                              /-(Kafka's native protocol, not reactive)--> Kafka
clients ==(RSocket)==> Liiklus -(Pulsar native protocol, async)------> Pulsar
                              \--(OpenMessaging's protocol adapter)--> Any OM broker

(Where = is RSocket, client's protocol, and - is brocker's specific protocols). if OM will add RSocket support (and other vendors will start using it :) ), it will look more like this:

clients ==(RSocket)==> Liiklus ==(OpenMessaging's RSocket)==> Any OM broker

So, even if the Event Systems will support RSocket, Liiklus will stay (although dramatically simplified on the "event store plugin" part, visualized on the diagram on the right side).
But it will improve the whole system because client's demand will be transparently delivered from the client to brokers (currently it is done by the event storage plugins, specific for each storage)

[vongosling]

I think we could combine the issue to address how to support reactive programming paradigms. I've seen a lot of guys(Microprofile Reactive Streams, Spring Reactor, RSocket, Akka Streams, JDK8&JDK9, RxJava and so on.) work hard in this area but none of them is fittable for messaging. So, It's really worth exploring. After all, building an API like this was not our original intention.