ARTEMIS-3163 Experimental support for Netty IO_URING incubator #3479

franz1981 · 2021-03-07T14:19:45Z

https://issues.apache.org/jira/browse/ARTEMIS-3163

michaelandrepearce · 2021-03-07T16:50:15Z

...server/src/main/java/org/apache/activemq/artemis/core/remoting/impl/netty/NettyAcceptor.java

@@ -401,6 +406,21 @@ public ActiveMQThreadFactory run() {
            acceptorType = KQUEUE_ACCEPTOR_TYPE;

            logger.debug("Acceptor using native kqueue");
+         } else if (useIoUring && CheckDependencies.isIoUringAvailable()) {


I would expect io uring to be primary option later with fallback to epoll and then nio, as such maybe this should be first in order of preference. Its off by default but this way going forwards correct preference order would exist.

Yes :) correct

Damn karaf and its feature checks :O it works for reflection too? :O

You can probably add some metadata to artemis-server-osgi let it know the io.netty.incubator.channel.uring package usages are optional.

franz1981 · 2021-03-07T17:00:28Z

These are my results by using a single single-threaded acceptor for both clients and replication (on the live broker) to fairly compare epoll vs io_uring under load.
The test is similar to the one on https://issues.apache.org/jira/browse/ARTEMIS-2852 with 32 JMS core clients, 100 persistent bytes messages and IO_URING transport has been used only on the live server, leaving the rest as it is ie backup + clients

NOTE: These are just preliminary results, so I won't share HW configuration or anything to make this reproducible, but it should give the magnitude of improvement offered by io_uring.

master:

**************
EndToEnd Throughput: 22582 ops/sec
**************
EndToEnd SERVICE-TIME Latencies distribution in MICROSECONDS
mean               1410.83
min                 333.82
50.00%             1368.06
90.00%             1679.36
99.00%             2293.76
99.90%             3489.79
99.99%            13107.20
max               16187.39
count               320000

this pr:

**************
EndToEnd Throughput: 30540 ops/sec
**************
EndToEnd SERVICE-TIME Latencies distribution in MICROSECONDS
mean               1052.52
min                 329.73
50.00%             1007.62
90.00%             1286.14
99.00%             1736.70
99.90%             4653.06
99.99%            13893.63
max               16711.68
count               320000

The profile data collected with https://github.com/jvm-profiling-tools/async-profiler/ are attached on https://issues.apache.org/jira/browse/ARTEMIS-3163

But the important bits are:

Replication event loop thread: 935 (epoll) vs 775 (io_uring) samples -> ~94% cpu usage vs 78% cpu usage
SYSCALLs samples:
epoll: ~61% samples

io_uring: ~31% samples

The io_uring version is far more efficient while using resources then epoll despite our replication process already try to batch writes as much as possible to amortize syscall cost: would be interesting to compare io_uring with some Open-OnLoad kernel bypass driver using epoll :P

IMPORTANT:
Why I've chosen to use a single thread for everything?
Don't be tempted to use the default configuration, because it uses 3 * available cores for the replication/client acceptors: the io_uring version is that much efficient then epoll then the Netty event loops tends to go idle most of the time and need to be awaken, causing application threads to always pay the cost to wakeup event loop threads...this can make the io_uring version to look worse then epoll, while is right the opposite(!!)

michaelandrepearce · 2021-03-07T19:15:37Z

...nt/src/main/java/org/apache/activemq/artemis/core/remoting/impl/netty/CheckDependencies.java

+
+   public static boolean isIoUringAvailable() {
+      try {
+         return Env.isLinuxOs() && (boolean) (Class.forName("io.netty.incubator.channel.uring.IOUring")


Should just call IOUring.isAvailable() like we do with kqueue and epoll, dont do this via reflection

It's not available via maven afaik so it's up to the users to compile it from source and use it

<dependency> <groupId>io.netty.incubator</groupId> <artifactId>netty-incubator-transport-native-io_uring</artifactId> <version>0.0.3.Final</version> <classifier>linux-x86_64</classifier> </dependency>

Its available. These is actually in the docs on the netty github

In fact 0.0.4 is latest

https://mvnrepository.com/artifact/io.netty.incubator/netty-incubator-transport-native-io_uring/0.0.4.Final

Wow ok, will remove the reflection then, thanks to have checked!

michaelandrepearce · 2021-03-07T19:17:58Z

...-core-client/src/main/java/org/apache/activemq/artemis/core/client/ActiveMQClientLogger.java

@@ -420,6 +425,11 @@
           format = Message.Format.MESSAGE_FORMAT)
   void unableToCheckEpollAvailabilitynoClass();

+   @LogMessage(level = Logger.Level.WARN)
+   @Message(id = 212079, value = "IoUring is not available, please add to the classpath or configure useIoUring=false to remove this warning",


Please put in correct place in order or the code. So people know what the high watermark for next code is without needing to hunt. All others are in asc order, please follow

michaelandrepearce · 2021-03-07T19:18:33Z

...-core-client/src/main/java/org/apache/activemq/artemis/core/client/ActiveMQClientLogger.java

@@ -395,6 +395,11 @@
           format = Message.Format.MESSAGE_FORMAT)
   void unableToCheckEpollAvailability(@Cause Throwable e);

+   @LogMessage(level = Logger.Level.WARN)
+   @Message(id = 212080, value = "Unable to check IoUring availability ",


Please put in correct order place in file based on id. See other comment for why

michaelandrepearce · 2021-03-07T19:21:49Z

...server/src/main/java/org/apache/activemq/artemis/core/remoting/impl/netty/NettyAcceptor.java

@@ -112,6 +113,7 @@
   public static final String NIO_ACCEPTOR_TYPE = "NIO";
   public static final String EPOLL_ACCEPTOR_TYPE = "EPOLL";
   public static final String KQUEUE_ACCEPTOR_TYPE = "KQUEUE";
+   public static final String IOURING_ACCEPTOR_TYPE = "IO_URING";


Why not just IOURING we dont add under score seperator for KQueue or EPoll. Just also avoids any escaping issues on urls or other things in config urls or admin console

Perhaps Franz wanted to avoid a ring of IOUs...but more probably its just that io_uring actually has the underscore in its name whilst I dont think the others do. Reasonable point about URL based config...though I thought it was just a boolean there?

michaelandrepearce · 2021-03-07T19:25:18Z

Docs need updating.

michaelandrepearce · 2021-09-06T10:30:03Z

@franz1981 whats status on this one? Im keen to merge it, happy to help contribute any last bits like slight code re-org on if statements, and docs that are needed myself, if needed next week.

franz1981 · 2021-09-07T07:25:50Z

whats status on this one? Im keen to merge it, happy to help contribute any last bits like slight code re-org on if statements, and docs that are needed myself, if needed next week.

That seems a nice: i just would like to perform some better testing to help users to know which kernel version (and incubator version) to use. Let it parks here for a week and then we can move on and maybe decide with a public vote if people are interested and would like to know what it is/its purpose. We can have a call too with the community to explain it :)

jbertram · 2021-09-17T00:56:10Z

@franz1981, just a friendly reminder that it's been a week. This looks like a nice feature so it would be nice to get it out to the community if it's ready.

franz1981 · 2021-09-17T06:19:04Z

@jbertram

just a friendly reminder that it's been a week. This looks like a nice feature so it would be nice to get it out to the community if it's ready.

I don't know yet if we should bring this in without any reflection usage (as @michaelandrepearce suggested) and we need to add a doc paragraph that state we have no idea (yet) if SSL or other features works as expected ie EXPERIMENTAL tag on everything.
If any of the community people would like to raise the appropriate issues after this will be merged I can happily work on Netty side too, to fix things.
Let me know if it makes sense to bring this on the community list.

gemmellr · 2021-09-17T08:51:45Z

...server/src/main/java/org/apache/activemq/artemis/core/remoting/impl/netty/NettyAcceptor.java

@@ -112,6 +113,7 @@
   public static final String NIO_ACCEPTOR_TYPE = "NIO";
   public static final String EPOLL_ACCEPTOR_TYPE = "EPOLL";
   public static final String KQUEUE_ACCEPTOR_TYPE = "KQUEUE";
+   public static final String IOURING_ACCEPTOR_TYPE = "IO_URING";


Perhaps Franz wanted to avoid a ring of IOUs...but more probably its just that io_uring actually has the underscore in its name whilst I dont think the others do. Reasonable point about URL based config...though I thought it was just a boolean there?

gemmellr · 2021-09-17T09:18:18Z

...t/src/main/java/org/apache/activemq/artemis/core/remoting/impl/netty/TransportConstants.java

@@ -61,6 +61,8 @@

   public static final String USE_KQUEUE_PROP_NAME = "useKQueue";

+   public static final String USE_IOURING_PROP_NAME = "useIoUring";


Your recent comments Franz noted flagging everything with caution that its experimental...this might be a simple spot to trivially bang the point home to a user, making the option name reflect it...e.g "useIoUringExperimental"

I disagree, i think what franz has here is fine/better, and that should just be clear in the docs, else we end up with mangled config , it will be off by default anyhow so someone will have to have read the docs to know its there of which it will be mentioned very clearly this is an incubator feature, and explicitly turned it on.

Having experimental bits using explicit non-final options or even APIs prior to completion to signal their status isnt mangled config IMO. Its fairly common, e.g the JDK has many instances of this. The explicit incubator GAV for the netty bit is another example.

gemmellr · 2021-09-17T09:25:52Z

...server/src/main/java/org/apache/activemq/artemis/core/remoting/impl/netty/NettyAcceptor.java

@@ -401,6 +406,21 @@ public ActiveMQThreadFactory run() {
            acceptorType = KQUEUE_ACCEPTOR_TYPE;

            logger.debug("Acceptor using native kqueue");
+         } else if (useIoUring && CheckDependencies.isIoUringAvailable()) {


You can probably add some metadata to artemis-server-osgi let it know the io.netty.incubator.channel.uring package usages are optional.

clebertsuconic · 2021-09-21T14:14:38Z

I think we should run the extended suite using io_uring... if it's pass.. we should merge it upstream.

michaelandrepearce · 2021-09-24T22:29:23Z

@clebertsuconic i agree we should merge upstream asap, but we should be using the maven dependency and some docs are def needed before merge, @franz1981 my intent was to send you PR with changes and docs, I had a disaster work week, im hoping to get to it next week for you, actually just booked some hours out the calendar to try make sure i get the time.

@franz1981 - Just a note on machines with Solarflare cards with kernal by pass with ONLOAD, perf improvement not really noticeable, but i borrowed a machine without special network kit, and there was a quite noticeable latency improvement and drop in resource usage for the same loads, essentially matching what your own tests had already shown, but it does mean its reproducable outside your lab the improvement

michaelandrepearce · 2021-09-28T11:04:44Z

@franz1981 i sent to your branch which this is PR from, a PR with my proposed changes to address the comments i had raised in this PR, see: franz1981#15 , if you agree with those then feel free to merge to your branch, and i have no further comments.

Basically 3 key bits i addressed.

Logging order so entries are in asc id order so its easy for next development to know the next id to use
Use netty binaries via maven and remove reflection
Documentation - with clear note about incubator status

jbertram · 2022-12-16T04:28:29Z

@franz1981, @clebertsuconic, @michaelandrepearce, where are we on this? Is this something we still want to do? Netty's io_uring lib is still in their incubator, but it's up to 0.16 now.

michaelandrepearce · 2023-01-12T17:27:04Z

@jbertram i would like to still see this in, as long as we mark it experimental or in incubation in docs, i personally support this. @franz1981, its on your branch atm this PR and i sent you those changes still pending, if you don't have time anymore, i can send in PR from my branch (closing this one), and merge/rebase on current master.

jbertram · 2023-01-23T02:02:15Z

@michaelandrepearce, given the silence from @franz1981 I'd say you should go ahead and send a PR of your own with the changes you recommended (plus resolving the existing conflicts). Then we can move forward. Thanks!

franz1981 force-pushed the io_uring_incubator branch 2 times, most recently from 33f6084 to aacb59e Compare March 7, 2021 14:49

franz1981 added 2 commits March 7, 2021 17:05

ARTEMIS-3163 Experimental support for Netty IO_URING incubator

4128da6

ARTEMIS-3163 Bump Netty version to be compatible with IO_URING

b4307e8

franz1981 force-pushed the io_uring_incubator branch from aacb59e to b4307e8 Compare March 7, 2021 16:05

michaelandrepearce reviewed Mar 7, 2021

View reviewed changes

franz1981 mentioned this pull request Mar 12, 2021

Few adjustments on the Artemis Test softwaremill/mqperf#57

Closed

gemmellr reviewed Sep 17, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARTEMIS-3163 Experimental support for Netty IO_URING incubator #3479

ARTEMIS-3163 Experimental support for Netty IO_URING incubator #3479

franz1981 commented Mar 7, 2021

michaelandrepearce Mar 7, 2021

franz1981 Mar 7, 2021

franz1981 Mar 7, 2021

gemmellr Sep 17, 2021

franz1981 commented Mar 7, 2021 •

edited

michaelandrepearce Mar 7, 2021

franz1981 Mar 7, 2021

michaelandrepearce Mar 7, 2021 •

edited

michaelandrepearce Mar 7, 2021

michaelandrepearce Mar 7, 2021

franz1981 Mar 7, 2021

michaelandrepearce Mar 7, 2021

michaelandrepearce Mar 7, 2021

michaelandrepearce Mar 7, 2021

gemmellr Sep 17, 2021

michaelandrepearce commented Mar 7, 2021

michaelandrepearce commented Sep 6, 2021

franz1981 commented Sep 7, 2021 •

edited

jbertram commented Sep 17, 2021

franz1981 commented Sep 17, 2021 •

edited

gemmellr Sep 17, 2021

gemmellr Sep 17, 2021

michaelandrepearce Sep 24, 2021 •

edited

gemmellr Sep 24, 2021 •

edited

gemmellr Sep 17, 2021

clebertsuconic commented Sep 21, 2021

michaelandrepearce commented Sep 24, 2021 •

edited

michaelandrepearce commented Sep 28, 2021 •

edited

jbertram commented Dec 16, 2022

michaelandrepearce commented Jan 12, 2023 •

edited

jbertram commented Jan 23, 2023

		@@ -61,6 +61,8 @@

		public static final String USE_KQUEUE_PROP_NAME = "useKQueue";

		public static final String USE_IOURING_PROP_NAME = "useIoUring";

ARTEMIS-3163 Experimental support for Netty IO_URING incubator #3479

Are you sure you want to change the base?

ARTEMIS-3163 Experimental support for Netty IO_URING incubator #3479

Conversation

franz1981 commented Mar 7, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

franz1981 commented Mar 7, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelandrepearce Mar 7, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelandrepearce commented Mar 7, 2021

michaelandrepearce commented Sep 6, 2021

franz1981 commented Sep 7, 2021 • edited

jbertram commented Sep 17, 2021

franz1981 commented Sep 17, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelandrepearce Sep 24, 2021 • edited

Choose a reason for hiding this comment

gemmellr Sep 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clebertsuconic commented Sep 21, 2021

michaelandrepearce commented Sep 24, 2021 • edited

michaelandrepearce commented Sep 28, 2021 • edited

jbertram commented Dec 16, 2022

michaelandrepearce commented Jan 12, 2023 • edited

jbertram commented Jan 23, 2023

franz1981 commented Mar 7, 2021 •

edited

michaelandrepearce Mar 7, 2021 •

edited

franz1981 commented Sep 7, 2021 •

edited

franz1981 commented Sep 17, 2021 •

edited

michaelandrepearce Sep 24, 2021 •

edited

gemmellr Sep 24, 2021 •

edited

michaelandrepearce commented Sep 24, 2021 •

edited

michaelandrepearce commented Sep 28, 2021 •

edited

michaelandrepearce commented Jan 12, 2023 •

edited