Iterating Over a Bitset in Java

How fast can you iterate over a bitset? Daniel Lemire published a benchmark recently in support of a strategy using the number of trailing zeroes to skip over empty bits. I have used the same technique in Java several times in my hobby project SplitMap and this is something I am keen to optimise. I think that the best strategy depends on what you want to do with the set bits, and how sparse and uniformly distributed they are. I argue that the cost of iteration is less important than the constraints your API imposes on the caller, and whether the caller is free to exploit patterns in the data.

C2 Generates Good Code

If you think C++ is much faster than Java, you either don’t know much about Java or do lots of floating point arithmetic. This isn’t about benchmarking C++ against Java, but comparing the compilation outputs for a C++ implementation and a Java implementation shows that there won’t be much difference if your Java method gets hot. Only the time to performance will differ, and this is amortised over the lifetime of an application. The trailing zeroes implementation is probably the fastest technique in Java as well as in C++, but that is to ignore the optimisations you can’t apply to the callback if you use it too literally.

Compiling this C++ function with GCC yields the snippet of assembly taken from the loop kernel:


template <typename CALLBACK>
static void for_each(const long* bitmap, const int size, const CALLBACK& callback) {
    for (size_t k = 0; k < size; ++k) {
        long bitset = bitmap[k];
        while (bitset != 0) {
            callback((k * 64) + __builtin_ctzl(bitset));
            bitset ^= (bitset & -bitset);
        }
    }
}

The instruction tzcntl calculates the next set bit and blsr switches it off.


.L99:
  movq  %rdi, %rcx
  blsr  %ebx, %ebx
  call  _ZNSo3putEc
  movq  %rax, %rcx
  call  _ZNSo5flushEv
  testl  %ebx, %ebx
  je  .L96
.L100:
  xorl  %edx, %edx
  movq  %r12, %rcx
  tzcntl  %ebx, %edx
  addl  %ebp, %edx
  call  _ZNSolsEi
  movq  %rax, %rdi
  movq  (%rax), %rax
  movq  -24(%rax), %rax
  movq  240(%rdi,%rax), %rsi
  testq  %rsi, %rsi
  je  .L108
  cmpb  $0, 56(%rsi)
  jne  .L109
  movq  %rsi, %rcx
  call  _ZNKSt5ctypeIcE13_M_widen_initEv
  movq  (%rsi), %rax
  movl  $10, %edx
  movq  48(%rax), %rax
  cmpq  %r14, %rax
  je  .L99
  movq  %rsi, %rcx
  call  *%rax
  movsbl  %al, %edx
  jmp  .L99
  .p2align 4,,10

In Java, almost identical code is generated.


public void forEach(long[] bitmap, IntConsumer consumer) {
    for (int i = 0; i < bitmap.length; ++i) {
      long word = bitmap[i];
      while (word != 0) {
        consumer.accept(Long.SIZE * i + Long.numberOfTrailingZeros(word));
        word ^= Long.lowestOneBit(word);
      }
    }
  }

The key difference is that xor and blsi haven’t been fused into blsr, so the C++ code is probably slightly faster. A lambda function accumulating the contents of an array is inlined into this loop (the add comes from an inlined lambda, but notice how little time is spent adding compared to computing the bit to switch off in this sample produced by perfasm).


   .83%    0x000002d79d366a19: tzcnt   r9,rcx
  8.53%    0x000002d79d366a1e: add     r9d,ebx
  0.42%    0x000002d79d366a21: cmp     r9d,r8d
  0.00%    0x000002d79d366a24: jnb     2d79d366a4dh
  0.62%    0x000002d79d366a26: add     r10d,dword ptr [rdi+r9*4+10h]
 16.22%    0x000002d79d366a2b: vmovq   r11,xmm4
  6.68%    0x000002d79d366a30: mov     dword ptr [r11+10h],r10d
 27.92%    0x000002d79d366a34: blsi    r10,rcx
  0.55%    0x000002d79d366a39: xor     rcx,r10         
  0.10%    0x000002d79d366a3c: mov     r11,qword ptr [r15+70h]  

It’s this Java code, and its impact on which optimisations can be applied to the IntConsumer that this post focuses on. There are different principles, particularly related to inlining and vectorisation opportunities in C++, but this blog is about Java. Depending on what your callback does, you get different benchmark results and you should make different choices about how to do the iteration: you just can’t assess this in isolation.

Special Casing -1

Imagine you have an int[] containing data, and you are iterating over a mask or materialised predicate over that data. For each set bit, you want to add the corresponding entry in the array to a sum. In Java, that looks like this (you’ve already seen the generated assembly above):


  @Benchmark
  public int reduce() {
    int[] result = new int[1];
    forEach(bitmap, i -> result[0] += data[i]);
    return result[0];
  }

How fast can this get? It obviously depends on how full the bitset is. The worst case would be that it’s completely full, and it couldn’t get much better than if only one bit per word were set. The difference is noticeable, but scales by a factor less than the number of bits:

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: scenario
reduce thrpt 1 10 7.435909 0.017491 ops/ms FULL
reduce thrpt 1 10 260.305307 6.081961 ops/ms ONE_BIT_PER_WORD

But the important code here, the callback itself, is stuck at entry level compilation. There is no unrolling, no vectorisation, the adds can’t be pipelined because there is a data dependency on blsi and xor. We can do much better in some cases, and not much worse in others, just by treating -1 as a special case, profiting from optimisations that can now be applied inside the callback. Passing a different callback which consumes whole words costs a branch, but it’s often worth it. Here’s the iterator now:


  interface WordConsumer {
    void acceptWord(int wordIndex, long word);
  }

  public void forEach(long[] bitmap, IntConsumer intConsumer, WordConsumer wordConsumer) {
    for (int i = 0; i < bitmap.length; ++i) {
      long word = bitmap[i];
      if (word == -1L) {
        wordConsumer.acceptWord(i, word);
      } else {
        while (word != 0) {
          intConsumer.accept(Long.SIZE * i + Long.numberOfTrailingZeros(word));
          word ^= Long.lowestOneBit(word);
        }
      }
    }
  }

  @Benchmark
  public int reduceWithWordConsumer() {
    int[] result = new int[1];
    forEach(bitmap, i -> result[0] += data[i], (index, word) -> {
      if (word != -1L) {
        throw new IllegalStateException();
      }
      int sum = 0;
      for (int i = index * Long.SIZE; i < (index + 1) * Long.SIZE; ++i) {
        sum += data[i];
      }
      result[0] += sum;
    });
    return result[0];
  }

This really pays off when the bitset is full, but having that extra branch does seem to cost something even though it is never taken, whereas the full case improves 6x.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: scenario
reduce thrpt 1 10 7.401202 0.118648 ops/ms FULL
reduce thrpt 1 10 261.682016 4.155856 ops/ms ONE_BIT_PER_WORD
reduceWithWordConsumer thrpt 1 10 43.972759 0.993264 ops/ms FULL
reduceWithWordConsumer thrpt 1 10 222.824868 4.877147 ops/ms ONE_BIT_PER_WORD

We still don’t actually know the cost of the branch when it’s taken every now and then. To estimate it, we need a new scenario (or new scenarios) which mix full and sparse words. As you might expect, having the WordConsumer is great when one word in every few is full: the fast path is so much faster, it practically skips the word.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: scenario
reduce thrpt 1 10 157.358633 4.538679 ops/ms SPARSE_16_FULL_WORDS
reduceWithWordConsumer thrpt 1 10 257.041035 7.446404 ops/ms SPARSE_16_FULL_WORDS

So in this scenario, the branch has paid for itself. How? The data dependency has been removed with a countable loop. Here’s the perfasm output. Notice two things: long runs of add instructions, and the vastly reduced percentage against blsi. The time is now spent adding numbers up, not switching off least significant bits. This feels like progress.


....[Hottest Region 1]..............................................................................
c2, com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub, version 170 (804 bytes) 

                                                         ;   {optimized virtual_call}
           0x000001dd5b35acd4: mov     r9,qword ptr [rsp+0a0h]
           0x000001dd5b35acdc: movzx   r11d,byte ptr [r9+94h]  ;*getfield isDone {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@30 (line 144)
                                                         ; implicit exception: dispatches to 0x000001dd5b35b486
           0x000001dd5b35ace4: test    r11d,r11d
           0x000001dd5b35ace7: jne     1dd5b35b19dh      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@33 (line 144)
           0x000001dd5b35aced: mov     ebx,1h
           0x000001dd5b35acf2: jmp     1dd5b35ad31h
  0.00%    0x000001dd5b35acf4: vmovq   rdi,xmm0          ;*invokevirtual reduceWithWordConsumer {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35acf9: mov     r8d,dword ptr [rdi+10h]  ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@28 (line 85)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001dd5b35acfd: mov     rdx,qword ptr [rsp+30h]
  0.03%    0x000001dd5b35ad02: nop
  0.00%    0x000001dd5b35ad03: call    1dd5b2c7000h      ; ImmutableOopMap{[160]=Oop [168]=Oop [176]=Oop [40]=NarrowOop [48]=Oop }
                                                         ;*invokevirtual consume {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@20 (line 142)
                                                         ;   {optimized virtual_call}
  0.04%    0x000001dd5b35ad08: mov     r9,qword ptr [rsp+0a0h]
           0x000001dd5b35ad10: movzx   r11d,byte ptr [r9+94h]  ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.02%    0x000001dd5b35ad18: mov     r10,qword ptr [r15+70h]
  0.00%    0x000001dd5b35ad1c: mov     rbx,qword ptr [rsp+40h]
  0.03%    0x000001dd5b35ad21: add     rbx,1h            ; ImmutableOopMap{r9=Oop [160]=Oop [168]=Oop [176]=Oop [40]=NarrowOop [48]=Oop }
                                                         ;*ifeq {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@33 (line 144)
  0.00%    0x000001dd5b35ad25: test    dword ptr [r10],eax  ;   {poll}
  0.00%    0x000001dd5b35ad28: test    r11d,r11d
           0x000001dd5b35ad2b: jne     1dd5b35b1a2h
  0.00%    0x000001dd5b35ad31: mov     rdi,qword ptr [r15+80h]
  0.01%    0x000001dd5b35ad38: mov     r10,rdi
  0.00%    0x000001dd5b35ad3b: add     r10,18h
  0.01%    0x000001dd5b35ad3f: cmp     r10,qword ptr [r15+90h]
           0x000001dd5b35ad46: jnb     1dd5b35b119h      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ad4c: mov     qword ptr [r15+80h],r10
           0x000001dd5b35ad53: prefetchw byte ptr [r10+0c0h]
  0.00%    0x000001dd5b35ad5b: mov     qword ptr [rdi],1h
  0.01%    0x000001dd5b35ad62: prefetchw byte ptr [r10+100h]
           0x000001dd5b35ad6a: mov     dword ptr [rdi+8h],0f8000168h
                                                         ;   {metadata({type array int})}
  0.00%    0x000001dd5b35ad71: prefetchw byte ptr [r10+140h]
  0.01%    0x000001dd5b35ad79: mov     dword ptr [rdi+0ch],1h
  0.01%    0x000001dd5b35ad80: prefetchw byte ptr [r10+180h]
  0.00%    0x000001dd5b35ad88: mov     qword ptr [rdi+10h],r12  ;*newarray {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@1 (line 74)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ad8c: mov     rcx,qword ptr [rsp+0b0h]
  0.00%    0x000001dd5b35ad94: mov     ebp,dword ptr [rcx+0ch]  ;*getfield bitmap {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@6 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ad97: mov     dword ptr [rsp+70h],ebp
  0.00%    0x000001dd5b35ad9b: mov     rsi,qword ptr [r15+80h]
  0.00%    0x000001dd5b35ada2: mov     r10,rsi
           0x000001dd5b35ada5: add     r10,18h
  0.00%    0x000001dd5b35ada9: cmp     r10,qword ptr [r15+90h]
           0x000001dd5b35adb0: jnb     1dd5b35b14fh      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35adb6: mov     qword ptr [r15+80h],r10
           0x000001dd5b35adbd: prefetchw byte ptr [r10+0c0h]
  0.00%    0x000001dd5b35adc5: mov     r8,qword ptr [rsp+20h]
  0.01%    0x000001dd5b35adca: mov     r10,qword ptr [r8+0b0h]
  0.01%    0x000001dd5b35add1: mov     qword ptr [rsi],r10
  0.00%    0x000001dd5b35add4: mov     dword ptr [rsi+8h],0f8021906h
                                                         ;   {metadata(&apos;com/openkappa/simd/iterate/BitSetIterator$$Lambda$43&apos;)}
           0x000001dd5b35addb: mov     qword ptr [rsi+10h],r12  ;*new {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@0
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35addf: mov     r11d,dword ptr [rsp+28h]
  0.00%    0x000001dd5b35ade4: mov     dword ptr [rsi+0ch],r11d
                                                         ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ade8: mov     rax,qword ptr [r15+80h]
  0.00%    0x000001dd5b35adef: mov     rbp,rdi
  0.01%    0x000001dd5b35adf2: shr     rbp,3h
  0.00%    0x000001dd5b35adf6: mov     dword ptr [rsi+10h],ebp  ;*putfield arg$2 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@11
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35adf9: mov     r10,rax
  0.00%    0x000001dd5b35adfc: add     r10,18h
  0.00%    0x000001dd5b35ae00: cmp     r10,qword ptr [r15+90h]
           0x000001dd5b35ae07: jnb     1dd5b35b08eh      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ae0d: mov     qword ptr [r15+80h],r10
  0.00%    0x000001dd5b35ae14: prefetchw byte ptr [r10+0c0h]
  0.00%    0x000001dd5b35ae1c: mov     rdx,qword ptr [rsp+38h]
  0.00%    0x000001dd5b35ae21: mov     r10,qword ptr [rdx+0b0h]
  0.02%    0x000001dd5b35ae28: mov     qword ptr [rax],r10
  0.01%    0x000001dd5b35ae2b: mov     dword ptr [rax+8h],0f8021a86h
                                                         ;   {metadata(&apos;com/openkappa/simd/iterate/BitSetIterator$$Lambda$44&apos;)}
  0.00%    0x000001dd5b35ae32: mov     qword ptr [rax+10h],r12
  0.00%    0x000001dd5b35ae36: mov     dword ptr [rax+0ch],r11d
                                                         ;*new {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::get$Lambda@0
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@18 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ae3a: mov     dword ptr [rax+10h],ebp  ;*invokespecial &lt;init&gt; {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@18 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001dd5b35ae3d: mov     ebp,dword ptr [rsp+70h]
  0.00%    0x000001dd5b35ae41: mov     r13d,dword ptr [r12+rbp*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@6 (line 104)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001dd5b35b422
  0.03%    0x000001dd5b35ae46: test    r13d,r13d
           0x000001dd5b35ae49: jbe     1dd5b35b024h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@7 (line 104)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ae4f: mov     r10d,r13d
  0.00%    0x000001dd5b35ae52: dec     r10d
           0x000001dd5b35ae55: cmp     r10d,r13d
           0x000001dd5b35ae58: jnb     1dd5b35b24ah
  0.01%    0x000001dd5b35ae5e: lea     r14,[r12+rbp*8]   ;*getfield bitmap {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@6 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ae62: xor     r10d,r10d
  0.00%    0x000001dd5b35ae65: xor     ebp,ebp
           0x000001dd5b35ae67: mov     qword ptr [rsp+0b0h],rcx
  0.00%    0x000001dd5b35ae6f: mov     qword ptr [rsp+0a0h],r9
           0x000001dd5b35ae77: mov     qword ptr [rsp+20h],r8
  0.01%    0x000001dd5b35ae7c: mov     dword ptr [rsp+28h],r11d
  0.00%    0x000001dd5b35ae81: mov     qword ptr [rsp+38h],rdx
  0.00%    0x000001dd5b35ae86: mov     qword ptr [rsp+40h],rbx
  0.00%    0x000001dd5b35ae8b: vmovq   xmm0,rdi
  0.00%    0x000001dd5b35ae90: jmp     1dd5b35afafh
  0.03%    0x000001dd5b35ae95: mov     r8d,r10d
  0.03%    0x000001dd5b35ae98: add     r8d,40h           ;*imul {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@32 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.04%    0x000001dd5b35ae9c: cmp     r10d,r8d
           0x000001dd5b35ae9f: jnl     1dd5b35b01dh      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@33 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.03%    0x000001dd5b35aea5: mov     r11d,dword ptr [r12+rcx*8+0ch]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@44 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001dd5b35b1d3
  0.04%    0x000001dd5b35aeaa: cmp     r10d,r11d
           0x000001dd5b35aead: jnb     1dd5b35b1d3h
  0.04%    0x000001dd5b35aeb3: movsxd  r11,r11d
  0.04%    0x000001dd5b35aeb6: movsxd  r9,r8d
  0.02%    0x000001dd5b35aeb9: dec     r9
  0.05%    0x000001dd5b35aebc: cmp     r9,r11
  0.00%    0x000001dd5b35aebf: jnb     1dd5b35b1d3h
  0.04%    0x000001dd5b35aec5: lea     rdi,[r12+rcx*8]
  0.04%    0x000001dd5b35aec9: mov     ecx,r10d
  0.03%    0x000001dd5b35aecc: inc     ecx
  0.04%    0x000001dd5b35aece: mov     r9d,r10d
  0.04%    0x000001dd5b35aed1: xor     ebx,ebx           ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@36 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.03%    0x000001dd5b35aed3: add     ebx,dword ptr [rdi+r9*4+10h]
                                                         ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@45 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.20%    0x000001dd5b35aed8: inc     r9d               ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@48 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.04%    0x000001dd5b35aedb: cmp     r9d,ecx
           0x000001dd5b35aede: jl      1dd5b35aed3h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@33 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.04%    0x000001dd5b35aee0: add     r10d,31h
  0.03%    0x000001dd5b35aee4: cmp     r8d,r10d
  0.03%    0x000001dd5b35aee7: mov     r11d,80000000h
  0.04%    0x000001dd5b35aeed: cmovl   r10d,r11d
  0.04%    0x000001dd5b35aef1: cmp     r9d,r10d
           0x000001dd5b35aef4: jnl     1dd5b35af68h
  0.04%    0x000001dd5b35aef6: nop     word ptr [rax+rax+0h]  ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@36 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.07%    0x000001dd5b35af00: movsxd  r11,r9d
  0.05%    0x000001dd5b35af03: add     ebx,dword ptr [rdi+r9*4+10h]
  0.31%    0x000001dd5b35af08: add     ebx,dword ptr [rdi+r11*4+14h]
  0.32%    0x000001dd5b35af0d: add     ebx,dword ptr [rdi+r11*4+18h]
  0.33%    0x000001dd5b35af12: add     ebx,dword ptr [rdi+r11*4+1ch]
  0.37%    0x000001dd5b35af17: add     ebx,dword ptr [rdi+r11*4+20h]
  0.34%    0x000001dd5b35af1c: add     ebx,dword ptr [rdi+r11*4+24h]
  0.39%    0x000001dd5b35af21: add     ebx,dword ptr [rdi+r11*4+28h]
  0.36%    0x000001dd5b35af26: add     ebx,dword ptr [rdi+r11*4+2ch]
  0.34%    0x000001dd5b35af2b: add     ebx,dword ptr [rdi+r11*4+30h]
  0.35%    0x000001dd5b35af30: add     ebx,dword ptr [rdi+r11*4+34h]
  0.38%    0x000001dd5b35af35: add     ebx,dword ptr [rdi+r11*4+38h]
  0.36%    0x000001dd5b35af3a: add     ebx,dword ptr [rdi+r11*4+3ch]
  0.49%    0x000001dd5b35af3f: add     ebx,dword ptr [rdi+r11*4+40h]
  0.39%    0x000001dd5b35af44: add     ebx,dword ptr [rdi+r11*4+44h]
  0.42%    0x000001dd5b35af49: add     ebx,dword ptr [rdi+r11*4+48h]
  0.39%    0x000001dd5b35af4e: add     ebx,dword ptr [rdi+r11*4+4ch]
                                                         ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@45 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.41%    0x000001dd5b35af53: add     r9d,10h           ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@48 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001dd5b35af57: cmp     r9d,r10d
           0x000001dd5b35af5a: jl      1dd5b35af00h      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001dd5b35af5c: mov     r11,qword ptr [r15+70h]  ; ImmutableOopMap{rdi=Oop rsi=Oop rax=Oop r14=Oop xmm0=Oop [160]=Oop [168]=Oop [176]=Oop [40]=NarrowOop [48]=Oop [112]=NarrowOop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@51 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001dd5b35af60: test    dword ptr [r11],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@51 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  0.15%    0x000001dd5b35af63: cmp     r9d,r10d
           0x000001dd5b35af66: jl      1dd5b35af00h
           0x000001dd5b35af68: cmp     r9d,r8d
           0x000001dd5b35af6b: jnl     1dd5b35af7dh
           0x000001dd5b35af6d: nop                       ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@36 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.67%    0x000001dd5b35af70: add     ebx,dword ptr [rdi+r9*4+10h]
                                                         ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@45 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.59%    0x000001dd5b35af75: inc     r9d               ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@48 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.13%    0x000001dd5b35af78: cmp     r9d,r8d
           0x000001dd5b35af7b: jl      1dd5b35af70h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@33 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001dd5b35af7d: vmovq   r10,xmm0
  0.02%    0x000001dd5b35af82: add     dword ptr [r10+10h],ebx
  2.64%    0x000001dd5b35af86: mov     r8,rax
  2.45%    0x000001dd5b35af89: mov     r9,rsi            ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.70%    0x000001dd5b35af8c: mov     r10,qword ptr [r15+70h]
  2.16%    0x000001dd5b35af90: mov     r11d,ebp
  2.37%    0x000001dd5b35af93: inc     r11d              ; ImmutableOopMap{r8=Oop r9=Oop rsi=Oop rax=Oop r14=Oop xmm0=Oop [160]=Oop [168]=Oop [176]=Oop [40]=NarrowOop [48]=Oop [112]=NarrowOop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@78 (line 104)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.44%    0x000001dd5b35af96: test    dword ptr [r10],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@78 (line 104)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  2.74%    0x000001dd5b35af99: cmp     r11d,r13d
  0.00%    0x000001dd5b35af9c: jnl     1dd5b35acf4h      ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@10 (line 105)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.14%    0x000001dd5b35afa2: shl     ebp,6h
  2.37%    0x000001dd5b35afa5: mov     r10d,ebp
  2.41%    0x000001dd5b35afa8: add     r10d,40h          ;*imul {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@22 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.72%    0x000001dd5b35afac: mov     ebp,r11d          ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@10 (line 105)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.14%    0x000001dd5b35afaf: mov     rbx,qword ptr [r14+rbp*8+10h]
                                                         ;*laload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@13 (line 105)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.59%    0x000001dd5b35afb4: mov     r11,qword ptr [rsp+0b0h]
  2.30%    0x000001dd5b35afbc: mov     ecx,dword ptr [r11+10h]  ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$1@5 (line 75)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@57 (line 110)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.76%    0x000001dd5b35afc0: cmp     rbx,0ffffffffffffffffh
           0x000001dd5b35afc4: je      1dd5b35ae95h      ;*ifne {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@22 (line 106)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.09%    0x000001dd5b35afca: test    rbx,rbx
  0.00%    0x000001dd5b35afcd: je      1dd5b35af86h      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@42 (line 109)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.34%    0x000001dd5b35afcf: mov     r9d,dword ptr [r12+rcx*8+0ch]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$1@9 (line 75)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@57 (line 110)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001dd5b35b3f6
  2.50%    0x000001dd5b35afd4: lea     rdi,[r12+rcx*8]
  2.62%    0x000001dd5b35afd8: nop     dword ptr [rax+rax+0h]  ;*aload_2 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@45 (line 110)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.11%    0x000001dd5b35afe0: vmovq   r11,xmm0
  2.35%    0x000001dd5b35afe5: mov     r8d,dword ptr [r11+10h]  ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$1@3 (line 75)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@57 (line 110)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.39%    0x000001dd5b35afe9: tzcnt   r11,rbx
  2.65%    0x000001dd5b35afee: add     r11d,r10d         ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@56 (line 110)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.15%    0x000001dd5b35aff1: cmp     r11d,r9d
  0.00%    0x000001dd5b35aff4: jnb     1dd5b35b04dh
  2.29%    0x000001dd5b35aff6: add     r8d,dword ptr [rdi+r11*4+10h]
 11.03%    0x000001dd5b35affb: vmovq   r11,xmm0
  2.45%    0x000001dd5b35b000: mov     dword ptr [r11+10h],r8d  ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  3.14%    0x000001dd5b35b004: mov     r11,qword ptr [r15+70h]
  2.18%    0x000001dd5b35b008: blsi    r8,rbx
  2.23%    0x000001dd5b35b00d: xor     rbx,r8            ; ImmutableOopMap{rcx=NarrowOop rdi=Oop rsi=Oop rax=Oop r14=Oop xmm0=Oop [160]=Oop [168]=Oop [176]=Oop [40]=NarrowOop [48]=Oop [112]=NarrowOop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@72 (line 111)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.44%    0x000001dd5b35b010: test    dword ptr [r11],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@72 (line 111)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  2.71%    0x000001dd5b35b013: test    rbx,rbx
           0x000001dd5b35b016: jne     1dd5b35afe0h      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@42 (line 109)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.23%    0x000001dd5b35b018: jmp     1dd5b35af86h
           0x000001dd5b35b01d: xor     ebx,ebx
           0x000001dd5b35b01f: jmp     1dd5b35af7dh
           0x000001dd5b35b024: mov     qword ptr [rsp+0b0h],rcx
           0x000001dd5b35b02c: mov     qword ptr [rsp+0a0h],r9
           0x000001dd5b35b034: mov     qword ptr [rsp+20h],r8
           0x000001dd5b35b039: mov     dword ptr [rsp+28h],r11d
           0x000001dd5b35b03e: mov     qword ptr [rsp+38h],rdx
....................................................................................................
 98.92%  <total for region 1>

Heroically ploughing through the full words tells a different story: blsi is up at 11%. This indicates more time is spent iterating rather than evaluating the callback.


....[Hottest Region 1]..............................................................................
c2, com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub, version 164 (218 bytes) 

                                                         ; - java.lang.invoke.LambdaForm$MH/1035918105::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@11 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
           0x0000019f106c66e7: mov     qword ptr [r15+3d8h],rdi
                                                         ;*invokespecial &lt;init&gt; {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/1035918105::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@11 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  0.00%    0x0000019f106c66ee: mov     r10d,dword ptr [rsp+60h]
  0.00%    0x0000019f106c66f3: mov     edx,dword ptr [r12+r10*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@4 (line 89)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x0000019f106c6aba
  0.02%    0x0000019f106c66f8: test    edx,edx
           0x0000019f106c66fa: jbe     19f106c6826h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@5 (line 89)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
           0x0000019f106c6700: mov     r10d,edx
  0.00%    0x0000019f106c6703: dec     r10d
           0x0000019f106c6706: cmp     r10d,edx
           0x0000019f106c6709: jnb     19f106c6942h
  0.00%    0x0000019f106c670f: mov     r10d,dword ptr [rsp+60h]
           0x0000019f106c6714: lea     rax,[r12+r10*8]   ;*getfield bitmap {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@6 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  0.00%    0x0000019f106c6718: xor     ecx,ecx
           0x0000019f106c671a: jmp     19f106c674dh
  1.54%    0x0000019f106c671c: mov     r11,r13
  1.57%    0x0000019f106c671f: mov     r13,rbp
  1.69%    0x0000019f106c6722: mov     r8,r14
  1.46%    0x0000019f106c6725: vmovd   r9d,xmm1
  1.48%    0x0000019f106c672a: vmovq   rbx,xmm0
  1.61%    0x0000019f106c672f: vmovq   r14,xmm2
  1.63%    0x0000019f106c6734: vmovq   rbp,xmm3
  1.46%    0x0000019f106c6739: mov     rdi,rbp           ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/1035918105::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@11 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.49%    0x0000019f106c673c: mov     r10,qword ptr [r15+70h]
  1.54%    0x0000019f106c6740: inc     ecx               ; ImmutableOopMap{r11=Oop r9=NarrowOop rdi=Oop rax=Oop rbp=Oop r13=Oop r14=Oop [152]=Oop [48]=Oop [96]=NarrowOop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@52 (line 89)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.73%    0x0000019f106c6742: test    dword ptr [r10],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@52 (line 89)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  1.43%    0x0000019f106c6745: cmp     ecx,edx
  0.00%    0x0000019f106c6747: jnl     19f106c6480h      ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@8 (line 90)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.57%    0x0000019f106c674d: mov     rdi,qword ptr [rax+rcx*8+10h]
                                                         ;*laload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@10 (line 90)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.61%    0x0000019f106c6752: test    rdi,rdi
           0x0000019f106c6755: je      19f106c6739h      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@17 (line 91)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.68%    0x0000019f106c6757: vmovq   xmm3,rbp
  1.47%    0x0000019f106c675c: mov     esi,dword ptr [r11+10h]  ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduce$0@5 (line 67)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@31 (line 92)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.48%    0x0000019f106c6760: mov     r10d,dword ptr [r12+rsi*8+0ch]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduce$0@9 (line 67)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@31 (line 92)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x0000019f106c6a3e
  1.58%    0x0000019f106c6765: vmovq   xmm2,r14
  1.68%    0x0000019f106c676a: vmovq   xmm0,rbx
  1.46%    0x0000019f106c676f: vmovd   xmm1,r9d
  1.46%    0x0000019f106c6774: mov     r14,r8
  1.51%    0x0000019f106c6777: mov     rbp,r13
  1.68%    0x0000019f106c677a: mov     r13,r11
  1.41%    0x0000019f106c677d: lea     r8,[r12+rsi*8]
  1.40%    0x0000019f106c6781: mov     ebx,ecx
  1.55%    0x0000019f106c6783: shl     ebx,6h
  1.63%    0x0000019f106c6786: nop     word ptr [rax+rax+0h]  ;*aload_2 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@20 (line 92)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.68%    0x0000019f106c6790: vmovq   r11,xmm2
  1.70%    0x0000019f106c6795: mov     r11d,dword ptr [r11+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduce$0@3 (line 67)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@31 (line 92)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  6.98%    0x0000019f106c6799: tzcnt   r9,rdi
  3.47%    0x0000019f106c679e: add     r9d,ebx           ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 92)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.65%    0x0000019f106c67a1: cmp     r9d,r10d
           0x0000019f106c67a4: jnb     19f106c67cdh
  1.67%    0x0000019f106c67a6: add     r11d,dword ptr [r8+r9*4+10h]
 11.45%    0x0000019f106c67ab: vmovq   r9,xmm2
  3.20%    0x0000019f106c67b0: mov     dword ptr [r9+10h],r11d  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@46 (line 93)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
 11.31%    0x0000019f106c67b4: blsi    r11,rdi
  1.71%    0x0000019f106c67b9: xor     rdi,r11           ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/1035918105::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@11 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.72%    0x0000019f106c67bc: mov     r11,qword ptr [r15+70h]  ; ImmutableOopMap{r8=Oop r9=Oop rsi=NarrowOop rax=Oop rbp=Oop r13=Oop xmm1=NarrowOop xmm2=Oop xmm3=Oop [152]=Oop [48]=Oop [96]=NarrowOop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@46 (line 93)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.80%    0x0000019f106c67c0: test    dword ptr [r11],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@46 (line 93)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  6.99%    0x0000019f106c67c3: test    rdi,rdi
           0x0000019f106c67c6: jne     19f106c6790h      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@17 (line 91)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.39%    0x0000019f106c67c8: jmp     19f106c671ch
           0x0000019f106c67cd: mov     edx,0ffffffe4h
           0x0000019f106c67d2: mov     r10,qword ptr [rsp+98h]
           0x0000019f106c67da: mov     qword ptr [rsp+90h],r10
           0x0000019f106c67e2: mov     r10,qword ptr [rsp+30h]
           0x0000019f106c67e7: mov     qword ptr [rsp+0a0h],r13
           0x0000019f106c67ef: vmovsd  qword ptr [rsp+20h],xmm0
           0x0000019f106c67f5: mov     qword ptr [rsp+28h],r10
....................................................................................................
 98.59%  <total for region 1>

This shows the cost of a data dependency in a loop. The operation we want to perform is associative, so we could even vectorise this. In C++ that might happen automatically, or could be ensured with intrinsics, but C2 has various heuristics: it won’t try to vectorise a simple reduction, and 64 would probably be on the short side for most cases it would try to vectorise.

Acknowledging Runs

You might be tempted to transfer even more control to the callback, by accumulating runs and then calling the callback once per run. It simplifies the code to exclude incomplete start and end words from the run.


private interface RunConsumer {
    void acceptRun(int start, int end);
  }

  public void forEach(long[] bitmap, IntConsumer intConsumer, RunConsumer runConsumer) {
    int runStart = -1;
    for (int i = 0; i < bitmap.length; ++i) {
      long word = bitmap[i];
      if (word == -1L) {
        if (runStart == -1) {
          runStart = i;
        }
      } else {
        if (runStart != -1) {
          runConsumer.acceptRun(runStart * Long.SIZE, i * Long.SIZE);
          runStart = -1;
        }
        while (word != 0) {
          intConsumer.accept(Long.SIZE * i + Long.numberOfTrailingZeros(word));
          word ^= Long.lowestOneBit(word);
        }
      }
    }
    if (runStart != -1) {
      runConsumer.acceptRun(runStart * Long.SIZE, bitmap.length * Long.SIZE);
    }
  }

For a simple reduction, the extra complexity isn’t justified: you’re better off with the WordIterator.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: scenario
reduce thrpt 1 10 160.502749 2.960568 ops/ms SPARSE_16_FULL_WORDS
reduce thrpt 1 10 7.294747 0.186678 ops/ms FULL
reduce thrpt 1 10 258.064511 8.902233 ops/ms ONE_BIT_PER_WORD
reduce thrpt 1 10 159.613877 3.424432 ops/ms SPARSE_1_16_WORD_RUN
reduceWithRunConsumer thrpt 1 10 251.683131 6.799639 ops/ms SPARSE_16_FULL_WORDS
reduceWithRunConsumer thrpt 1 10 37.809154 0.723198 ops/ms FULL
reduceWithRunConsumer thrpt 1 10 218.133560 13.756779 ops/ms ONE_BIT_PER_WORD
reduceWithRunConsumer thrpt 1 10 140.896826 8.495777 ops/ms SPARSE_1_16_WORD_RUN
reduceWithWordConsumer thrpt 1 10 257.961783 5.892072 ops/ms SPARSE_16_FULL_WORDS
reduceWithWordConsumer thrpt 1 10 43.909471 0.601319 ops/ms FULL
reduceWithWordConsumer thrpt 1 10 213.731758 20.398077 ops/ms ONE_BIT_PER_WORD
reduceWithWordConsumer thrpt 1 10 258.280428 11.316647 ops/ms SPARSE_1_16_WORD_RUN

It’s simplistic to measure this and conclude that this is a bad approach though. There are several other dimensions to this problem:

  1. Vectorised callbacks
  2. Inlining failures preventing optimisations
  3. The number of runs and their lengths (i.e. your data and how you structure it)

Vectorisable Callbacks

There are real benefits to batching up callbacks if the workload in the callback can be vectorised. The code doesn’t need to get much more complicated to start benefitting from larger iteration batches. Mapping each bit to a scaled and squared value from the data array and storing it into an output array illustrates this.


  @Benchmark
  public void map(Blackhole bh) {
    forEach(bitmap, i -> output[i] = data[i] * data[i] * factor);
    bh.consume(output);
  }

  @Benchmark
  public void mapWithWordConsumer(Blackhole bh) {
    forEach(bitmap, i -> output[0] = data[i] * factor, (WordConsumer) (index, word) -> {
      if (word != -1L) {
        throw new IllegalStateException();
      }
      for (int i = index * Long.SIZE; i < (index + 1) * Long.SIZE; ++i) {
        output[i] = data[i] * data[i] * factor;
      }
    });
    bh.consume(output);
  }

  @Benchmark
  public void mapWithRunConsumer(Blackhole bh) {
    forEach(bitmap, i -> output[0] = data[i] * factor, (RunConsumer) (start, end) -> {
      for (int i = start; i < end; ++i) {
        output[i] = data[i] * data[i] * factor;
      }
    });
    bh.consume(output);
  }

The RunConsumer does much better in the full case, never much worse than the WordConsumer and always better than the basic strategy – even when there is only one run in the entire bitset, or when there are a few full words in an otherwise sparse bitset.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: scenario
map thrpt 1 10 127.876662 3.411741 ops/ms SPARSE_16_FULL_WORDS
map thrpt 1 10 10.598974 0.022404 ops/ms FULL
map thrpt 1 10 126.434666 18.608547 ops/ms ONE_BIT_PER_WORD
map thrpt 1 10 115.977840 20.449258 ops/ms SPARSE_1_16_WORD_RUN
mapWithRunConsumer thrpt 1 10 199.186167 8.138446 ops/ms SPARSE_16_FULL_WORDS
mapWithRunConsumer thrpt 1 10 64.230868 2.871434 ops/ms FULL
mapWithRunConsumer thrpt 1 10 219.963063 4.257561 ops/ms ONE_BIT_PER_WORD
mapWithRunConsumer thrpt 1 10 203.403804 6.907366 ops/ms SPARSE_1_16_WORD_RUN
mapWithWordConsumer thrpt 1 10 229.822235 5.276084 ops/ms SPARSE_16_FULL_WORDS
mapWithWordConsumer thrpt 1 10 48.381990 3.845642 ops/ms FULL
mapWithWordConsumer thrpt 1 10 218.907803 5.331011 ops/ms ONE_BIT_PER_WORD
mapWithWordConsumer thrpt 1 10 240.795280 10.204818 ops/ms SPARSE_1_16_WORD_RUN

This is simply because the callback was vectorised, and the style of the RunConsumer API allows this to be exploited. This can be seen with perfasm. Both the WordConsumer and RunConsumer are actually vectorised, but the thing to notice is that there are two hot regions in the WordConsumer benchmark: the iteration and the callback, this boundary is often crossed. On the other hand, the RunConsumer implementation spends most of its time in the callback.

WordConsumer


....[Hottest Region 1]..............................................................................
c2, com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub, version 172 (227 bytes) 

           0x000001c2aa13c788: ud2                       ;*athrow {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@21
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@47 (line 226)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001c2aa13c78a: mov     r8d,ecx
  0.23%    0x000001c2aa13c78d: mov     ecx,r8d           ;*aload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@62 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.55%    0x000001c2aa13c790: vmovdqu ymm1,ymmword ptr [r9+r10*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@69 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.15%    0x000001c2aa13c797: vpmulld ymm1,ymm1,ymm1
  3.72%    0x000001c2aa13c79c: vpmulld ymm1,ymm1,ymm2
 16.02%    0x000001c2aa13c7a1: vmovdqu ymmword ptr [rdx+r10*4+10h],ymm1
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@78 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.69%    0x000001c2aa13c7a8: movsxd  r8,r10d
  1.55%    0x000001c2aa13c7ab: vmovdqu ymm1,ymmword ptr [r9+r8*4+30h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@69 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.46%    0x000001c2aa13c7b2: vpmulld ymm1,ymm1,ymm1
  1.71%    0x000001c2aa13c7b7: vpmulld ymm1,ymm1,ymm2
  3.20%    0x000001c2aa13c7bc: vmovdqu ymmword ptr [rdx+r8*4+30h],ymm1
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@78 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.07%    0x000001c2aa13c7c3: add     r10d,10h          ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@79 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.70%    0x000001c2aa13c7c7: cmp     r10d,r11d
           0x000001c2aa13c7ca: jl      1c2aa13c790h      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1453625478::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1453625478::get$Lambda@5
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@10
                                                         ; - java.lang.invoke.LambdaForm$MH/1166726978::linkToTargetMethod@5
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@6 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.02%    0x000001c2aa13c7cc: mov     r8,qword ptr [r15+70h]  ; ImmutableOopMap{r9=Oop rdx=Oop rax=Oop r13=NarrowOop xmm0=Oop [144]=Oop [152]=Oop [48]=Oop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@82 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.50%    0x000001c2aa13c7d0: test    dword ptr [r8],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@82 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  0.04%    0x000001c2aa13c7d3: cmp     r10d,r11d
           0x000001c2aa13c7d6: jl      1c2aa13c78ah
  0.05%    0x000001c2aa13c7d8: mov     r11d,dword ptr [rsp+5ch]
  0.02%    0x000001c2aa13c7dd: add     r11d,39h
  1.57%    0x000001c2aa13c7e1: mov     r8d,ecx
  0.02%    0x000001c2aa13c7e4: cmp     r8d,r11d
  0.06%    0x000001c2aa13c7e7: mov     ecx,80000000h
  0.02%    0x000001c2aa13c7ec: cmovl   r11d,ecx
  1.50%    0x000001c2aa13c7f0: cmp     r10d,r11d
           0x000001c2aa13c7f3: jnl     1c2aa13c819h
  0.02%    0x000001c2aa13c7f5: nop                       ;*aload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@62 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.06%    0x000001c2aa13c7f8: vmovdqu ymm1,ymmword ptr [r9+r10*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@69 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.21%    0x000001c2aa13c7ff: vpmulld ymm1,ymm1,ymm1
  2.16%    0x000001c2aa13c804: vpmulld ymm1,ymm1,ymm2
  1.80%    0x000001c2aa13c809: vmovdqu ymmword ptr [rdx+r10*4+10h],ymm1
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@78 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.56%    0x000001c2aa13c810: add     r10d,8h           ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@79 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c814: cmp     r10d,r11d
           0x000001c2aa13c817: jl      1c2aa13c7f8h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@59 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c819: cmp     r10d,r8d
           0x000001c2aa13c81c: jnl     1c2aa13c83ah
  0.01%    0x000001c2aa13c81e: nop                       ;*aload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@62 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.58%    0x000001c2aa13c820: mov     ecx,dword ptr [r9+r10*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@69 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.03%    0x000001c2aa13c825: imul    ecx,ecx
  1.58%    0x000001c2aa13c828: imul    ecx,dword ptr [rsp+58h]
  0.03%    0x000001c2aa13c82d: mov     dword ptr [rdx+r10*4+10h],ecx
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@78 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.56%    0x000001c2aa13c832: inc     r10d              ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@79 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c835: cmp     r10d,r8d
           0x000001c2aa13c838: jl      1c2aa13c820h      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1453625478::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1453625478::get$Lambda@5
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@10
                                                         ; - java.lang.invoke.LambdaForm$MH/1166726978::linkToTargetMethod@5
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@6 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.55%    0x000001c2aa13c83a: mov     r10,qword ptr [r15+70h]
  0.01%    0x000001c2aa13c83e: inc     esi               ; ImmutableOopMap{rax=Oop r13=NarrowOop xmm0=Oop [144]=Oop [152]=Oop [48]=Oop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@78 (line 168)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.02%    0x000001c2aa13c840: test    dword ptr [r10],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@78 (line 168)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  0.02%    0x000001c2aa13c843: cmp     esi,dword ptr [rsp+0a0h]
           0x000001c2aa13c84a: jnl     1c2aa13cb2ah      ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@10 (line 169)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.54%    0x000001c2aa13c850: mov     r8,rax
  0.01%    0x000001c2aa13c853: mov     r14,qword ptr [rsp+90h]
  0.01%    0x000001c2aa13c85b: mov     rbx,qword ptr [rsp+50h]
  0.01%    0x000001c2aa13c860: mov     r10d,r13d
  1.54%    0x000001c2aa13c863: vmovq   rax,xmm0
  0.01%    0x000001c2aa13c868: mov     r9d,dword ptr [rsp+0a0h]
  0.02%    0x000001c2aa13c870: jmp     1c2aa13c904h
           0x000001c2aa13c875: mov     r13,qword ptr [rsp+98h]
           0x000001c2aa13c87d: mov     r8,qword ptr [rsp+68h]  ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@7 (line 168)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001c2aa13c882: mov     ebp,dword ptr [r8+18h]  ;*getfield output {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@22 (line 141)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001c2aa13c886: mov     r10,qword ptr [rsp+30h]
           0x000001c2aa13c88b: test    r10,r10
....................................................................................................
 53.26%  <total for region 1>

....[Hottest Region 2]..............................................................................
c2, com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub, version 172 (391 bytes) 

                                                         ; implicit exception: dispatches to 0x000001c2aa13cdd6
           0x000001c2aa13c8c8: test    r11d,r11d
           0x000001c2aa13c8cb: jne     1c2aa13cc5eh      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@30 (line 144)
           0x000001c2aa13c8d1: mov     ebx,1h            ;*invokespecial &lt;init&gt; {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::get$Lambda@5
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@10
                                                         ; - java.lang.invoke.LambdaForm$MH/1166726978::linkToTargetMethod@5
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@12 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c8d6: mov     r8,qword ptr [rsp+0a0h]
  0.01%    0x000001c2aa13c8de: mov     r10d,dword ptr [r8+10h]  ;*getfield bitmap {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@2 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001c2aa13c8e2: mov     r9d,dword ptr [r12+r10*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@6 (line 168)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001c2aa13cdc6
  0.04%    0x000001c2aa13c8e7: test    r9d,r9d
           0x000001c2aa13c8ea: jbe     1c2aa13cb2fh      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@7 (line 168)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001c2aa13c8f0: mov     ecx,r9d
           0x000001c2aa13c8f3: dec     ecx
           0x000001c2aa13c8f5: cmp     ecx,r9d
           0x000001c2aa13c8f8: jnb     1c2aa13cd1ah
           0x000001c2aa13c8fe: lea     rax,[r12+r10*8]   ;*getfield bitmap {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@2 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001c2aa13c902: xor     esi,esi           ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@10 (line 169)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c904: mov     r11,qword ptr [rax+rsi*8+10h]
                                                         ;*laload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@13 (line 169)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.57%    0x000001c2aa13c909: cmp     r11,0ffffffffffffffffh
           0x000001c2aa13c90d: jne     1c2aa13cccah      ;*ifne {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@22 (line 170)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c913: mov     edx,dword ptr [r8+14h]  ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@3 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c917: mov     edi,dword ptr [r12+rdx*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@35 (line 225)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001c2aa13cda6
  0.01%    0x000001c2aa13c91c: mov     dword ptr [rsp+0a0h],r9d
  1.56%    0x000001c2aa13c924: mov     r13d,dword ptr [r8+0ch]  ;*getfield factor {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@11 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c928: mov     r11d,esi
  0.02%    0x000001c2aa13c92b: shl     r11d,6h           ;*imul {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@19 (line 223)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c92f: lea     r9,[r12+rdx*8]    ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@3 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.55%    0x000001c2aa13c933: mov     ecx,r11d
  0.01%    0x000001c2aa13c936: add     ecx,40h           ;*imul {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@27 (line 224)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c939: mov     ebp,dword ptr [r8+18h]  ;*getfield output {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@7 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c93d: test    r11d,r11d
  1.55%    0x000001c2aa13c940: jl      1c2aa13ca62h      ;*iflt {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@1
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@36 (line 225)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c946: cmp     r11d,ecx
           0x000001c2aa13c949: jnle    1c2aa13ca62h      ;*if_icmpgt {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@6
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@36 (line 225)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c94f: cmp     ecx,edi
           0x000001c2aa13c951: jnle    1c2aa13ca62h      ;*if_icmple {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@11
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@36 (line 225)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001c2aa13c957: mov     dword ptr [rsp+60h],ecx
  1.51%    0x000001c2aa13c95b: mov     dword ptr [rsp+5ch],r11d
  0.02%    0x000001c2aa13c960: mov     dword ptr [rsp+58h],r13d
  0.01%    0x000001c2aa13c965: mov     ecx,edi
  0.01%    0x000001c2aa13c967: vmovd   xmm1,edx
  1.48%    0x000001c2aa13c96b: vmovq   xmm0,rax
  0.02%    0x000001c2aa13c970: mov     r13d,r10d
  0.02%    0x000001c2aa13c973: mov     qword ptr [rsp+50h],rbx
  0.01%    0x000001c2aa13c978: mov     qword ptr [rsp+90h],r14
  1.52%    0x000001c2aa13c980: mov     rax,r8
  0.01%    0x000001c2aa13c983: mov     edi,dword ptr [r12+rbp*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@46 (line 226)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001c2aa13cdb6
  0.01%    0x000001c2aa13c988: lea     rdx,[r12+rbp*8]   ;*getfield output {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@7 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.02%    0x000001c2aa13c98c: mov     r10d,dword ptr [rsp+60h]
  1.50%    0x000001c2aa13c991: cmp     r10d,edi
           0x000001c2aa13c994: jnle    1c2aa13cac2h      ;*if_icmple {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@11
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@47 (line 226)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c99a: cmp     r11d,r10d
           0x000001c2aa13c99d: jnl     1c2aa13c83ah      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@59 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.02%    0x000001c2aa13c9a3: cmp     r11d,ecx
           0x000001c2aa13c9a6: jnb     1c2aa13cc9ah
  0.01%    0x000001c2aa13c9ac: movsxd  r10,dword ptr [rsp+60h]
  1.58%    0x000001c2aa13c9b1: movsxd  r11,ecx
  0.01%    0x000001c2aa13c9b4: dec     r10
  0.02%    0x000001c2aa13c9b7: cmp     r10,r11
           0x000001c2aa13c9ba: jnb     1c2aa13cc9ah
  0.01%    0x000001c2aa13c9c0: mov     r11d,dword ptr [rsp+5ch]
  1.50%    0x000001c2aa13c9c5: cmp     r11d,edi
           0x000001c2aa13c9c8: jnb     1c2aa13cc9ah
  0.02%    0x000001c2aa13c9ce: movsxd  r11,edi
  0.02%    0x000001c2aa13c9d1: cmp     r10,r11
           0x000001c2aa13c9d4: jnb     1c2aa13cc9ah
  0.02%    0x000001c2aa13c9da: mov     r11d,edx
  1.62%    0x000001c2aa13c9dd: shr     r11d,2h
  0.02%    0x000001c2aa13c9e1: and     r11d,7h
  0.01%    0x000001c2aa13c9e5: add     r11d,dword ptr [rsp+5ch]
  0.02%    0x000001c2aa13c9ea: mov     r8d,3h
  1.55%    0x000001c2aa13c9f0: sub     r8d,r11d
  0.01%    0x000001c2aa13c9f3: and     r8d,7h
  0.01%    0x000001c2aa13c9f7: add     r8d,dword ptr [rsp+5ch]
  0.01%    0x000001c2aa13c9fc: inc     r8d
  1.58%    0x000001c2aa13c9ff: cmp     r8d,dword ptr [rsp+60h]
  0.02%    0x000001c2aa13ca04: mov     ecx,dword ptr [rsp+60h]
  0.02%    0x000001c2aa13ca08: cmovnle r8d,ecx
  0.01%    0x000001c2aa13ca0c: mov     r10d,dword ptr [rsp+5ch]
  1.54%    0x000001c2aa13ca11: mov     ebx,ecx           ;*aload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@62 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.11%    0x000001c2aa13ca13: mov     r11d,dword ptr [r9+r10*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@69 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  4.67%    0x000001c2aa13ca18: imul    r11d,r11d
  0.14%    0x000001c2aa13ca1c: imul    r11d,dword ptr [rsp+58h]
  4.67%    0x000001c2aa13ca22: mov     dword ptr [rdx+r10*4+10h],r11d
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@78 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.16%    0x000001c2aa13ca27: inc     r10d              ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@79 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  4.63%    0x000001c2aa13ca2a: cmp     r10d,r8d
           0x000001c2aa13ca2d: jl      1c2aa13ca13h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@59 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13ca2f: mov     r11d,dword ptr [rsp+5ch]
  0.01%    0x000001c2aa13ca34: add     r11d,31h
  0.02%    0x000001c2aa13ca38: mov     r8d,ebx
  1.48%    0x000001c2aa13ca3b: cmp     r8d,r11d
  0.01%    0x000001c2aa13ca3e: mov     ebx,80000000h
  0.02%    0x000001c2aa13ca43: cmovl   r11d,ebx
  0.04%    0x000001c2aa13ca47: cmp     r10d,r11d
           0x000001c2aa13ca4a: jnl     1c2aa13c819h
  2.48%    0x000001c2aa13ca50: vpshufd xmm2,xmmword ptr [rsp+58h],0h
  2.82%    0x000001c2aa13ca57: vinserti128 ymm2,ymm2,xmm2,1h
  2.88%    0x000001c2aa13ca5d: jmp     1c2aa13c78dh      ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@14
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@36 (line 225)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001c2aa13ca62: mov     qword ptr [rsp+90h],r14
           0x000001c2aa13ca6a: mov     qword ptr [rsp+28h],r8
....................................................................................................
 46.39%  <total for region 2>

RunConsumer


....[Hottest Region 1]..............................................................................
c2, com.openkappa.simd.iterate.BitSetIterator$$Lambda$44.1209658195::acceptRun, version 166 (816 bytes) 

           # parm1:    r9        = int
           #           [sp+0x50]  (sp of caller)
           0x0000016658954740: mov     r10d,dword ptr [rdx+8h]
           0x0000016658954744: shl     r10,3h
           0x0000016658954748: cmp     rax,r10
           0x000001665895474b: jne     166588c6d80h      ;   {runtime_call ic_miss_stub}
           0x0000016658954751: nop
           0x0000016658954754: nop     dword ptr [rax+rax+0h]
           0x000001665895475c: nop
         [Verified Entry Point]
  0.00%    0x0000016658954760: mov     dword ptr [rsp+0ffffffffffff9000h],eax
  0.00%    0x0000016658954767: push    rbp
           0x0000016658954768: sub     rsp,40h           ;*synchronization entry
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@-1
           0x000001665895476c: mov     ecx,dword ptr [rdx+0ch]  ;*getfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@1
  0.00%    0x000001665895476f: mov     ebx,dword ptr [r12+rcx*8+14h]
                                                         ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@3 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
                                                         ; implicit exception: dispatches to 0x0000016658954b76
           0x0000016658954774: mov     eax,dword ptr [r12+rbx*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@3 (line 211)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
                                                         ; implicit exception: dispatches to 0x0000016658954b8e
  0.06%    0x0000016658954779: mov     r11d,dword ptr [r12+rcx*8+0ch]
                                                         ;*getfield factor {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@11 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x000001665895477e: mov     edi,dword ptr [r12+rcx*8+18h]
                                                         ;*getfield output {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@7 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954783: test    r8d,r8d
           0x0000016658954786: jl      16658954ab6h      ;*iflt {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@1
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@4 (line 211)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x000001665895478c: cmp     r8d,r9d
           0x000001665895478f: jnle    16658954ae6h      ;*if_icmpgt {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@6
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@4 (line 211)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954795: cmp     r9d,eax
           0x0000016658954798: jnle    16658954b1ah      ;*if_icmple {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@11
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@4 (line 211)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x000001665895479e: mov     ebp,dword ptr [r12+rdi*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@11 (line 212)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
                                                         ; implicit exception: dispatches to 0x0000016658954b9e
  0.05%    0x00000166589547a3: cmp     r9d,ebp
           0x00000166589547a6: jnle    16658954b4ah      ;*if_icmple {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@11
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@12 (line 212)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x00000166589547ac: cmp     r8d,r9d
           0x00000166589547af: jnl     16658954a81h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@22 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x00000166589547b5: cmp     r8d,eax
           0x00000166589547b8: jnb     16658954a91h
  0.00%    0x00000166589547be: movsxd  r10,r9d
           0x00000166589547c1: movsxd  rcx,eax
           0x00000166589547c4: dec     r10
           0x00000166589547c7: cmp     r10,rcx
           0x00000166589547ca: jnb     16658954a91h
           0x00000166589547d0: cmp     r8d,ebp
           0x00000166589547d3: jnb     16658954a91h
           0x00000166589547d9: movsxd  rcx,ebp
  0.01%    0x00000166589547dc: cmp     r10,rcx
           0x00000166589547df: jnb     16658954a91h
  0.00%    0x00000166589547e5: shl     rdi,3h            ;*getfield output {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@7 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x00000166589547e9: lea     rdx,[r12+rbx*8]   ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@3 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x00000166589547ed: mov     r10d,edi
  0.00%    0x00000166589547f0: shr     r10d,2h
  0.00%    0x00000166589547f4: and     r10d,7h
           0x00000166589547f8: add     r10d,r8d
           0x00000166589547fb: mov     ecx,3h
  0.00%    0x0000016658954800: sub     ecx,r10d
  0.00%    0x0000016658954803: and     ecx,7h
           0x0000016658954806: add     ecx,r8d
           0x0000016658954809: inc     ecx
  0.00%    0x000001665895480b: cmp     ecx,r9d
  0.00%    0x000001665895480e: cmovnle ecx,r9d           ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@25 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954812: mov     r10d,dword ptr [rdx+r8*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954817: imul    r10d,r10d
  0.01%    0x000001665895481b: imul    r10d,r11d
  0.00%    0x000001665895481f: mov     dword ptr [rdi+r8*4+10h],r10d
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954824: inc     r8d               ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@41 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954827: cmp     r8d,ecx
           0x000001665895482a: jl      16658954812h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@22 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x000001665895482c: mov     r10d,r9d
           0x000001665895482f: add     r10d,0ffffff81h
  0.00%    0x0000016658954833: mov     ebx,80000000h
  0.00%    0x0000016658954838: cmp     r9d,r10d
           0x000001665895483b: cmovl   r10d,ebx
           0x000001665895483f: cmp     r8d,r10d
           0x0000016658954842: jnl     16658954a61h
  0.00%    0x0000016658954848: vmovd   xmm1,r11d
           0x000001665895484d: vpshufd xmm1,xmm1,0h
           0x0000016658954852: vinserti128 ymm1,ymm1,xmm1,1h  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@44 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954858: nop     dword ptr [rax+rax+0h]  ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@25 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.92%    0x0000016658954860: vmovdqu ymm0,ymmword ptr [rdx+r8*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.31%    0x0000016658954867: vpmulld ymm0,ymm0,ymm0
  1.74%    0x000001665895486c: vpmulld ymm0,ymm0,ymm1
  4.55%    0x0000016658954871: vmovdqu ymmword ptr [rdi+r8*4+10h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.69%    0x0000016658954878: movsxd  rcx,r8d
  0.01%    0x000001665895487b: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+30h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.41%    0x0000016658954881: vpmulld ymm0,ymm0,ymm0
  0.78%    0x0000016658954886: vpmulld ymm0,ymm0,ymm1
  0.83%    0x000001665895488b: vmovdqu ymmword ptr [rdi+rcx*4+30h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.25%    0x0000016658954891: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+50h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.29%    0x0000016658954897: vpmulld ymm0,ymm0,ymm0
  1.51%    0x000001665895489c: vpmulld ymm0,ymm0,ymm1
  3.65%    0x00000166589548a1: vmovdqu ymmword ptr [rdi+rcx*4+50h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.54%    0x00000166589548a7: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+70h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.31%    0x00000166589548ad: vpmulld ymm0,ymm0,ymm0
  0.47%    0x00000166589548b2: vpmulld ymm0,ymm0,ymm1
  1.11%    0x00000166589548b7: vmovdqu ymmword ptr [rdi+rcx*4+70h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.28%    0x00000166589548bd: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+90h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.17%    0x00000166589548c6: vpmulld ymm0,ymm0,ymm0
  1.89%    0x00000166589548cb: vpmulld ymm0,ymm0,ymm1
  3.56%    0x00000166589548d0: vmovdqu ymmword ptr [rdi+rcx*4+90h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.73%    0x00000166589548d9: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+0b0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.21%    0x00000166589548e2: vpmulld ymm0,ymm0,ymm0
  0.34%    0x00000166589548e7: vpmulld ymm0,ymm0,ymm1
  1.29%    0x00000166589548ec: vmovdqu ymmword ptr [rdi+rcx*4+0b0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.33%    0x00000166589548f5: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+0d0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.97%    0x00000166589548fe: vpmulld ymm0,ymm0,ymm0
  1.90%    0x0000016658954903: vpmulld ymm0,ymm0,ymm1
  3.59%    0x0000016658954908: vmovdqu ymmword ptr [rdi+rcx*4+0d0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.82%    0x0000016658954911: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+0f0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.18%    0x000001665895491a: vpmulld ymm0,ymm0,ymm0
  0.29%    0x000001665895491f: vpmulld ymm0,ymm0,ymm1
  1.25%    0x0000016658954924: vmovdqu ymmword ptr [rdi+rcx*4+0f0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.33%    0x000001665895492d: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+110h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.10%    0x0000016658954936: vpmulld ymm0,ymm0,ymm0
  2.11%    0x000001665895493b: vpmulld ymm0,ymm0,ymm1
  3.67%    0x0000016658954940: vmovdqu ymmword ptr [rdi+rcx*4+110h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.93%    0x0000016658954949: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+130h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.13%    0x0000016658954952: vpmulld ymm0,ymm0,ymm0
  0.25%    0x0000016658954957: vpmulld ymm0,ymm0,ymm1
  1.35%    0x000001665895495c: vmovdqu ymmword ptr [rdi+rcx*4+130h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.32%    0x0000016658954965: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+150h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.93%    0x000001665895496e: vpmulld ymm0,ymm0,ymm0
  2.16%    0x0000016658954973: vpmulld ymm0,ymm0,ymm1
  3.73%    0x0000016658954978: vmovdqu ymmword ptr [rdi+rcx*4+150h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.95%    0x0000016658954981: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+170h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.14%    0x000001665895498a: vpmulld ymm0,ymm0,ymm0
  0.21%    0x000001665895498f: vpmulld ymm0,ymm0,ymm1
  1.39%    0x0000016658954994: vmovdqu ymmword ptr [rdi+rcx*4+170h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.29%    0x000001665895499d: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+190h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.42%    0x00000166589549a6: vpmulld ymm0,ymm0,ymm0
  2.61%    0x00000166589549ab: vpmulld ymm0,ymm0,ymm1
  4.42%    0x00000166589549b0: vmovdqu ymmword ptr [rdi+rcx*4+190h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.01%    0x00000166589549b9: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+1b0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.10%    0x00000166589549c2: vpmulld ymm0,ymm0,ymm0
  0.17%    0x00000166589549c7: vpmulld ymm0,ymm0,ymm1
  1.46%    0x00000166589549cc: vmovdqu ymmword ptr [rdi+rcx*4+1b0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.27%    0x00000166589549d5: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+1d0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
 13.60%    0x00000166589549de: vpmulld ymm0,ymm0,ymm0
  3.51%    0x00000166589549e3: vpmulld ymm0,ymm0,ymm1
  4.69%    0x00000166589549e8: vmovdqu ymmword ptr [rdi+rcx*4+1d0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.00%    0x00000166589549f1: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+1f0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.11%    0x00000166589549fa: vpmulld ymm0,ymm0,ymm0
  0.15%    0x00000166589549ff: vpmulld ymm0,ymm0,ymm1
  1.46%    0x0000016658954a04: vmovdqu ymmword ptr [rdi+rcx*4+1f0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.26%    0x0000016658954a0d: add     r8d,80h           ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@41 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.01%    0x0000016658954a14: cmp     r8d,r10d
           0x0000016658954a17: jl      16658954860h      ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@44 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954a1d: mov     r14,qword ptr [r15+70h]  ; ImmutableOopMap{rdi=Oop rdx=Oop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@44 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.06%    0x0000016658954a21: test    dword ptr [r14],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@44 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
                                                         ;   {poll}
  0.17%    0x0000016658954a24: cmp     r8d,r10d
           0x0000016658954a27: jl      16658954860h
           0x0000016658954a2d: mov     r10d,r9d
           0x0000016658954a30: add     r10d,0fffffff9h
           0x0000016658954a34: cmp     r9d,r10d
  0.00%    0x0000016658954a37: cmovl   r10d,ebx
           0x0000016658954a3b: cmp     r8d,r10d
           0x0000016658954a3e: jnl     16658954a61h      ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@25 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954a40: vmovdqu ymm0,ymmword ptr [rdx+r8*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.14%    0x0000016658954a47: vpmulld ymm0,ymm0,ymm0
  0.05%    0x0000016658954a4c: vpmulld ymm0,ymm0,ymm1
  0.03%    0x0000016658954a51: vmovdqu ymmword ptr [rdi+r8*4+10h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.04%    0x0000016658954a58: add     r8d,8h            ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@41 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954a5c: cmp     r8d,r10d
           0x0000016658954a5f: jl      16658954a40h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@22 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954a61: cmp     r8d,r9d
           0x0000016658954a64: jnl     16658954a81h
           0x0000016658954a66: nop                       ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@25 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954a68: mov     ecx,dword ptr [rdx+r8*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954a6d: imul    ecx,ecx
           0x0000016658954a70: imul    ecx,r11d
           0x0000016658954a74: mov     dword ptr [rdi+r8*4+10h],ecx
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954a79: inc     r8d               ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@41 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954a7c: cmp     r8d,r9d
  0.00%    0x0000016658954a7f: jl      16658954a68h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@22 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954a81: vzeroupper
  0.00%    0x0000016658954a84: add     rsp,40h
           0x0000016658954a88: pop     rbp
  0.00%    0x0000016658954a89: mov     r10,qword ptr [r15+70h]
           0x0000016658954a8d: test    dword ptr [r10],eax  ;   {poll_return}
  0.00%    0x0000016658954a90: ret
           0x0000016658954a91: mov     edx,0ffffff86h
           0x0000016658954a96: mov     ebp,r9d
           0x0000016658954a99: mov     dword ptr [rsp],r11d
           0x0000016658954a9d: mov     dword ptr [rsp+4h],r8d
           0x0000016658954aa2: mov     dword ptr [rsp+8h],ebx
           0x0000016658954aa6: mov     dword ptr [rsp+0ch],edi
           0x0000016658954aaa: nop
           0x0000016658954aac: vzeroupper
           0x0000016658954aaf: call    166588c8a00h      ; ImmutableOopMap{[8]=NarrowOop [12]=NarrowOop }
                                                         ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
....................................................................................................
 96.10%  <total for region 1>

Inlining

So far, everything has been inlined. Java optimistically assumes you only have one implementation and aggressively inlines at first, deoptimising to add a branch when it sees a second implementation, deoptimising again and replacing with a virtual call if it sees a third implementation. This doesn’t matter much usually, but the cost of this not only dwarfs any savings in an iteration strategy; it also prevents various optimisations which can be applied if the code is inlined. Once again, passing a batch of work into the callback completely ameliorates this, because even if the call is virtual, the callback itself might be hot and aggressively optimised. I haven’t benchmarked this because I think the point is self-evident to anyone who would read this far.

Number of runs

It’s clear to see from the benchmark results that the best choice of iteration strategy is sensitive to what you want to do with the data, but also how it is arranged. It is well documented in database literature that real data sets tend to contain runs. If you are building a bitmap index on some attribute of your data, and you sort your data by that attribute, you will have as many bitmaps as you have attribute values, and each attribute value bitmap will contain a single run. This is almost true for any index on attributes correlated with the attribute chosen for the sort order, and is completely untrue for uncorellated attributes. There are a range of iteration strategies to choose from, and the best iteration strategy for one index may not be the best for another.

My benchmarks are available at github.

Faster Floating Point Reductions

At the moment, I am working on a hobby project called SplitMap, which aims to evaluate aggregations over complex boolean expressions as fast as possible using the same high level constructs of the streams API. It’s already capable of performing logic that takes vanilla parallel streams 20ms in under 300μs, but I think sub 100μs is possible for these calculations. I have reached the stage where the bottleneck is floating point reductions: Java won’t vectorise these because the result would be numerically unstable. This is a bit limiting, because it often doesn’t matter very much: nobody represents money with floating point numbers, and if you’re solving stiff differential equations it won’t be numerical stability that stops you from using a more suitable language. The reality is, somebody somewhere probably really cares about this, and that’s probably why there’s no fastfp semantics in the language. This ancient proposal lay stagnant before being withdrawn, and several optimisations just can’t be implemented by the JIT compiler without violating language guarantees. An intrinsic for FMA only arrived in Java 9, 15 years after JSR 84 was withdrawn, and vectorised FMA is only available in JDK10, 18 years after JSR 84 was proposed. Computing an average is hardly numerical computing, but Java just isn’t a friendly language for this sort of thing.

Back to SplitMap. At this point, I’ve already maxed out the parallelism I can get from the fork join pool, so I want the code below to vectorise to squeeze out more performance:

  private double reduce(double[] data) {
    double reduced = 0D;
    for (int i = 0; i < data.length; ++i) {
      reduced += data[i];
    }
    return reduced;
  }

Looking at this with perfasm, it’s clear that unrolled scalar code is generated:


  0.00%    0x000001db0c6f5730: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+10h]
  6.17%    0x000001db0c6f5736: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+18h]
  6.15%    0x000001db0c6f573c: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+20h]
  6.23%    0x000001db0c6f5742: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+28h]
  6.16%    0x000001db0c6f5748: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+30h]
  6.37%    0x000001db0c6f574e: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+38h]
  6.22%    0x000001db0c6f5754: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+40h]
  6.21%    0x000001db0c6f575a: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+48h]
  6.11%    0x000001db0c6f5760: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+50h]
  6.18%    0x000001db0c6f5766: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+58h]
  6.18%    0x000001db0c6f576c: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+60h]
  6.30%    0x000001db0c6f5772: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+68h]
  6.23%    0x000001db0c6f5778: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+70h]
  6.33%    0x000001db0c6f577e: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+78h]
  6.25%    0x000001db0c6f5784: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+80h]
  6.31%    0x000001db0c6f578d: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+88h]

Let’s do something really dumb – allocate an array! Then I’ll reduce vertically onto it, and do a small horizontal reduction at the end.


  @Benchmark
  public double reduceBuffered() {
    double[] buffer = new double[1024];
    for (int i = 0; i < data.length; ++i) {
      buffer[i & 1023] += data[i];
    }
    return reduce(buffer);
  }

I benchmarked this against reduce. Using size 1024 as a sanity check, it’s clear the work is just being done twice, which is reassuring. Once the array gets a bit bigger, the gains of (what I think should be) the faster vertical reduction prior to the horizontal reduction pays for the array allocation.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
reduceBuffered thrpt 1 10 333.990079 3.542656 ops/ms 1024
reduceBuffered thrpt 1 10 18.639314 0.488300 ops/ms 65536
reduceBuffered thrpt 1 10 9.313916 0.343261 ops/ms 131072
reduceSimple thrpt 1 10 656.408971 1.771530 ops/ms 1024
reduceSimple thrpt 1 10 9.840417 0.032713 ops/ms 65536
reduceSimple thrpt 1 10 4.881353 0.076707 ops/ms 131072

The code in reduceBuffered produces a slightly different result because the elements are summated in a different order, though you’re hardly likely to notice. While, by any pragmatic definition, the function performs the same operation, it is semantically different. C2 actually doesn’t vectorise this code, and I have no idea why it’s so much faster! I won’t dwell on this because this is a dead end. In any case, here’s the perfasm output:


  0.20%    0x000001eaf2e49f50: vmovsd  xmm0,qword ptr [rdx+r9*8+10h]
  0.27%    0x000001eaf2e49f57: mov     ebx,r9d
  1.95%    0x000001eaf2e49f5a: add     ebx,0fh
  0.36%    0x000001eaf2e49f5d: and     ebx,3ffh
  0.21%    0x000001eaf2e49f63: mov     ecx,r9d
  0.30%    0x000001eaf2e49f66: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  2.96%    0x000001eaf2e49f6c: vaddsd  xmm0,xmm0,mmword ptr [r8+rcx*8+10h]
  0.61%    0x000001eaf2e49f73: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.50%    0x000001eaf2e49f7a: mov     ecx,r9d
  1.78%    0x000001eaf2e49f7d: inc     ecx
  0.37%    0x000001eaf2e49f7f: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.17%    0x000001eaf2e49f85: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  0.49%    0x000001eaf2e49f8c: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+18h]
  2.51%    0x000001eaf2e49f93: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.21%    0x000001eaf2e49f9a: mov     ecx,r9d
  0.34%    0x000001eaf2e49f9d: add     ecx,2h
  0.90%    0x000001eaf2e49fa0: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.32%    0x000001eaf2e49fa6: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.28%    0x000001eaf2e49fad: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+20h]
  1.43%    0x000001eaf2e49fb4: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.38%    0x000001eaf2e49fbb: vmovsd  xmm0,qword ptr [rdx+r9*8+28h]
  0.47%    0x000001eaf2e49fc2: mov     ecx,r9d
  0.28%    0x000001eaf2e49fc5: add     ecx,3h
  0.77%    0x000001eaf2e49fc8: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.34%    0x000001eaf2e49fce: vaddsd  xmm0,xmm0,mmword ptr [r8+rcx*8+10h]
  0.93%    0x000001eaf2e49fd5: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.36%    0x000001eaf2e49fdc: mov     ecx,r9d
  0.75%    0x000001eaf2e49fdf: add     ecx,4h
  0.43%    0x000001eaf2e49fe2: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.32%    0x000001eaf2e49fe8: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.32%    0x000001eaf2e49fef: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+30h]
  1.32%    0x000001eaf2e49ff6: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.24%    0x000001eaf2e49ffd: mov     ecx,r9d
  0.40%    0x000001eaf2e4a000: add     ecx,5h
  0.79%    0x000001eaf2e4a003: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.46%    0x000001eaf2e4a009: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.22%    0x000001eaf2e4a010: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+38h]
  4.18%    0x000001eaf2e4a017: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.61%    0x000001eaf2e4a01e: mov     ecx,r9d
  0.24%    0x000001eaf2e4a021: add     ecx,6h
  0.34%    0x000001eaf2e4a024: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.69%    0x000001eaf2e4a02a: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.55%    0x000001eaf2e4a031: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+40h]
  0.95%    0x000001eaf2e4a038: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.97%    0x000001eaf2e4a03f: mov     ecx,r9d
  0.37%    0x000001eaf2e4a042: add     ecx,7h
  0.20%    0x000001eaf2e4a045: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.32%    0x000001eaf2e4a04b: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.95%    0x000001eaf2e4a052: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+48h]
  0.92%    0x000001eaf2e4a059: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.95%    0x000001eaf2e4a060: mov     ecx,r9d
  0.42%    0x000001eaf2e4a063: add     ecx,8h
  0.35%    0x000001eaf2e4a066: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.20%    0x000001eaf2e4a06c: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.97%    0x000001eaf2e4a073: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+50h]
  1.22%    0x000001eaf2e4a07a: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.67%    0x000001eaf2e4a081: mov     ecx,r9d
  0.48%    0x000001eaf2e4a084: add     ecx,9h
  0.39%    0x000001eaf2e4a087: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.32%    0x000001eaf2e4a08d: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.66%    0x000001eaf2e4a094: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+58h]
  1.14%    0x000001eaf2e4a09b: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.45%    0x000001eaf2e4a0a2: mov     ecx,r9d
  0.65%    0x000001eaf2e4a0a5: add     ecx,0ah
  0.49%    0x000001eaf2e4a0a8: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.39%    0x000001eaf2e4a0ae: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.45%    0x000001eaf2e4a0b5: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+60h]
  1.32%    0x000001eaf2e4a0bc: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.39%    0x000001eaf2e4a0c3: mov     ecx,r9d
  0.39%    0x000001eaf2e4a0c6: add     ecx,0bh
  0.65%    0x000001eaf2e4a0c9: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.53%    0x000001eaf2e4a0cf: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.37%    0x000001eaf2e4a0d6: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+68h]
  1.22%    0x000001eaf2e4a0dd: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.46%    0x000001eaf2e4a0e4: mov     ecx,r9d
  0.42%    0x000001eaf2e4a0e7: add     ecx,0ch
  0.40%    0x000001eaf2e4a0ea: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.60%    0x000001eaf2e4a0f0: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.47%    0x000001eaf2e4a0f7: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+70h]
  1.04%    0x000001eaf2e4a0fe: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.74%    0x000001eaf2e4a105: mov     ecx,r9d
  0.37%    0x000001eaf2e4a108: add     ecx,0dh
  0.43%    0x000001eaf2e4a10b: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.36%    0x000001eaf2e4a111: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.68%    0x000001eaf2e4a118: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+78h]
  2.82%    0x000001eaf2e4a11f: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  2.04%    0x000001eaf2e4a126: mov     ecx,r9d
  0.19%    0x000001eaf2e4a129: add     ecx,0eh
  0.29%    0x000001eaf2e4a12c: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.36%    0x000001eaf2e4a132: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  2.02%    0x000001eaf2e4a139: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+80h]
  0.76%    0x000001eaf2e4a143: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
  1.99%    0x000001eaf2e4a14a: vmovsd  xmm0,qword ptr [rdx+r9*8+88h]
  0.39%    0x000001eaf2e4a154: vaddsd  xmm0,xmm0,mmword ptr [r8+rbx*8+10h]
  0.56%    0x000001eaf2e4a15b: vmovsd  qword ptr [r8+rbx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)

Using the same idea, but employing a trick I’ve used before for matrix multiplication, the code gets a lot faster!

  @Benchmark
  public double reduceVectorised() {
    double[] buffer = new double[1024];
    double[] temp = new double[1024];
    for (int i = 0; i < data.length >>> 10; ++i) {
      System.arraycopy(data, i * 1024, temp, 0,  temp.length);
      for (int j = 0; j < 1024; ++j) {
        buffer[j] += temp[j];
      }
    }
    return reduce(buffer);
  }

This generates a vectorised main loop:


  0.05%    0x000001e3fc0d71e0: vmovdqu ymm0,ymmword ptr [r9+r10*8+10h]
  0.16%    0x000001e3fc0d71e7: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+10h]
  1.26%    0x000001e3fc0d71ee: vmovdqu ymmword ptr [r13+r10*8+10h],ymm0
  0.39%    0x000001e3fc0d71f5: vmovdqu ymm0,ymmword ptr [r9+r10*8+30h]
  0.18%    0x000001e3fc0d71fc: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+30h]
  0.93%    0x000001e3fc0d7203: vmovdqu ymmword ptr [r13+r10*8+30h],ymm0
  0.68%    0x000001e3fc0d720a: vmovdqu ymm0,ymmword ptr [r9+r10*8+50h]
  0.17%    0x000001e3fc0d7211: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+50h]
  0.86%    0x000001e3fc0d7218: vmovdqu ymmword ptr [r13+r10*8+50h],ymm0
  0.73%    0x000001e3fc0d721f: vmovdqu ymm0,ymmword ptr [r9+r10*8+70h]
  0.19%    0x000001e3fc0d7226: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+70h]
  0.86%    0x000001e3fc0d722d: vmovdqu ymmword ptr [r13+r10*8+70h],ymm0
  0.78%    0x000001e3fc0d7234: vmovdqu ymm0,ymmword ptr [r9+r10*8+90h]
  0.17%    0x000001e3fc0d723e: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+90h]
  0.75%    0x000001e3fc0d7248: vmovdqu ymmword ptr [r13+r10*8+90h],ymm0
  0.84%    0x000001e3fc0d7252: vmovdqu ymm0,ymmword ptr [r9+r10*8+0b0h]
  0.15%    0x000001e3fc0d725c: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+0b0h]
  0.64%    0x000001e3fc0d7266: vmovdqu ymmword ptr [r13+r10*8+0b0h],ymm0
  0.92%    0x000001e3fc0d7270: vmovdqu ymm0,ymmword ptr [r9+r10*8+0d0h]
  0.15%    0x000001e3fc0d727a: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+0d0h]
  0.71%    0x000001e3fc0d7284: vmovdqu ymmword ptr [r13+r10*8+0d0h],ymm0
  0.91%    0x000001e3fc0d728e: vmovdqu ymm0,ymmword ptr [r9+r10*8+0f0h]
  0.15%    0x000001e3fc0d7298: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+0f0h]
  0.74%    0x000001e3fc0d72a2: vmovdqu ymmword ptr [r13+r10*8+0f0h],ymm0
  0.96%    0x000001e3fc0d72ac: vmovdqu ymm0,ymmword ptr [r9+r10*8+110h]
  0.12%    0x000001e3fc0d72b6: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+110h]
  0.70%    0x000001e3fc0d72c0: vmovdqu ymmword ptr [r13+r10*8+110h],ymm0
  0.99%    0x000001e3fc0d72ca: vmovdqu ymm0,ymmword ptr [r9+r10*8+130h]
  0.13%    0x000001e3fc0d72d4: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+130h]
  0.71%    0x000001e3fc0d72de: vmovdqu ymmword ptr [r13+r10*8+130h],ymm0
  0.94%    0x000001e3fc0d72e8: vmovdqu ymm0,ymmword ptr [r9+r10*8+150h]
  0.12%    0x000001e3fc0d72f2: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+150h]
  0.70%    0x000001e3fc0d72fc: vmovdqu ymmword ptr [r13+r10*8+150h],ymm0
  1.01%    0x000001e3fc0d7306: vmovdqu ymm0,ymmword ptr [r9+r10*8+170h]
  0.14%    0x000001e3fc0d7310: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+170h]
  0.75%    0x000001e3fc0d731a: vmovdqu ymmword ptr [r13+r10*8+170h],ymm0
  1.00%    0x000001e3fc0d7324: vmovdqu ymm0,ymmword ptr [r9+r10*8+190h]
  0.13%    0x000001e3fc0d732e: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+190h]
  0.67%    0x000001e3fc0d7338: vmovdqu ymmword ptr [r13+r10*8+190h],ymm0
  0.97%    0x000001e3fc0d7342: vmovdqu ymm0,ymmword ptr [r9+r10*8+1b0h]
  0.14%    0x000001e3fc0d734c: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+1b0h]
  0.72%    0x000001e3fc0d7356: vmovdqu ymmword ptr [r13+r10*8+1b0h],ymm0
  0.99%    0x000001e3fc0d7360: vmovdqu ymm0,ymmword ptr [r9+r10*8+1d0h]
  0.12%    0x000001e3fc0d736a: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+1d0h]
  0.69%    0x000001e3fc0d7374: vmovdqu ymmword ptr [r13+r10*8+1d0h],ymm0
  0.96%    0x000001e3fc0d737e: vmovdqu ymm0,ymmword ptr [r9+r10*8+1f0h]
  0.14%    0x000001e3fc0d7388: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+1f0h]
  0.70%    0x000001e3fc0d7392: vmovdqu ymmword ptr [r13+r10*8+1f0h],ymm0

This implementation is more than 3x faster than the original code, which includes the fuss of the buffers and copies. Wouldn’t it be nice to be able to opt in to fast floating point semantics somehow? Here are the final results (as usual, this is a throughput benchmark and higher is better):

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
reduceBuffered thrpt 1 10 329.935462 6.295871 ops/ms 1024
reduceBuffered thrpt 1 10 18.380467 0.455724 ops/ms 65536
reduceBuffered thrpt 1 10 9.257122 0.402876 ops/ms 131072
reduceSimple thrpt 1 10 654.809021 5.812795 ops/ms 1024
reduceSimple thrpt 1 10 9.694730 0.325011 ops/ms 65536
reduceSimple thrpt 1 10 4.772520 0.265691 ops/ms 131072
reduceVectorised thrpt 1 10 287.712794 27.492846 ops/ms 1024
reduceVectorised thrpt 1 10 34.454235 1.293985 ops/ms 65536
reduceVectorised thrpt 1 10 17.867701 0.813367 ops/ms 131072

The benchmark is on github.

L’esprit de l’escalier

L’esprit de l’escalier is the inability to respond until it’s too late to bother. Perhaps not until you’ve reached the bottom of a staircase, having had an altercation at the top. You’re most likely to have experienced this if you are at all introverted and have been to a job interview, having only managed to find solutions to simple problems once you find yourself alone on a train full of strangers. The more I ponder the psychology of this phenomenon, the more it fascinates me.

Some programmers dread brain teasers, claiming they are irrelevant to the job. I think this is wrong. In reality, so-called brain teasers are valid tests for application developers because they tend to be questions about programming translated into some other – admittedly, often fanciful – domain. They measure your ability to interpret requirements. I think if you try to assess this in person, the outcome often depends on personality types.

Unwrapping the Riddle

Here’s an odd puzzle with a programming question hiding somewhere inside it.

There are 64 bottles of wine, and one is poisoned. In three days’ time there will be a banquet and you must make sure none of the guests are poisoned. The poison takes between two and two-and-a-half days to kill a man, but you can risk the lives of as many of 32 prisoners as you like to find the poisoned wine bottle. What’s the smallest number of prisoners you can use to find the poisoned bottle?

You may have seen this puzzle before and get out of jail free, and many websites exist to allow people to research these problems for a reason. But, phrased another way, you are actually being asked a simple programming question:

A 64-bit integer has a value equal to a power of two. Find its logarithm in as few steps as possible.

OK! Simple! Only one bit is set, it just needs to be found. The logarithm is the number of zeroes less significant than the bit. The set bit is either in the top half or the bottom half, so test this recursively. We can find the index of the bit by using a sequence of independent bit masks, incrementing a counter differently for each mask which intersects with the value. The code is very simple:


  private static int indexOfFirstBit(long value) {
    int index = 0;
    int temp;
    if ((value & 0xFFFFFFFF00000000L) != 0) { 
      index += 32;
      temp = (int)(value >>> 32);
    } else {
      temp = (int)value;
    }
    if ((temp & 0xFFFF0000) != 0) { 
      index += 16;
      temp >>>= 16;
    }
    if ((temp & 0xFF00) != 0) { 
      index += 8;
      temp >>>= 8;
    }
    if ((temp & 0xF0) != 0) {
      index += 4;
      temp >>>= 4;
    }
    if ((temp & 0xC) != 0) {
      index += 2;
      temp >>>= 2;
    }
    if ((temp & 0x2) != 0) {
      index += 1;
    }
    return index;
  }

The idea is that you keep halving the size of the problem, and add the width of the zero part of each mask that passes (counting from the right). This can be rearranged but the code above is most obvious. You may even recognise that this is equivalent to the x86 instruction TZCNT, which in Java is accessible via the intrinsic Long.numberOfTrailingZeros.

So, going back to la-la land, you only need to risk the lives of six prisoners. All you have to do is ask each of the six prisoners to drink half of the bottles, the first one drinks from the first 32 bottles, the second from the first 16 and the third 16, and so on until the sixth prisoner who drinks from every other bottle. Before three days are up, up to six prisoners will die. If prisoner one dies, you know the poisoned bottle is in the second half. If prisoner two also dies, you know it’s in the third quarter. If the sixth prisoner doesn’t die, you know the poisoned bottle is one of the odd numbers.

Psychology and Interactive Problem Solving

Some people have seen these problems before, others enjoy solving them collaboratively, others instantly recognise solutions. Why does this only come to mind on the train home for some people? Well, it could be stupidity, but I think it’s purely psychological. You may be the sort of person who doesn’t want to solve a problem, even a simple one like this, interactively. Perhaps it feels invasive to reveal your precise thought processes to someone you just met, and aren’t sure you like yet. You might find you take your work with you on walks, or come up with your best ideas in the shower. In short, you may just be an introvert.

There’s a darker side to this psychological cause of failure. You may find that your interviewer is actually an interrogator, is trying to insult your ego, and you’ll feel drawn to defend yourself. The more you defend yourself, the less you can commit to a logical thought process; you become emotional. Your thoughts become jumbled as your emotional defence takes over. You focus on how you are being judged, that you were being judged five seconds ago and still haven’t solved the problem, that you are being judged yet more harshly now. You aren’t actually thinking about the problem at all any more. You aren’t being tested on your ability to solve the problem, but on your ability to solve the problem and defend your self esteem at the same time. Some people fail at this.

I actually think this is fair game in many cases. For instance, if you want to be an Army Officer, you can’t get flustered and be allowed to be thrown off your game like this: people might be shooting at you! For programmers, however, I’m not sure. Most programmers are used to the feeling of being in “flow” – the feeling of being immersed in the task at hand, knowledge instantly accessible. There’s never another person there when you’re in flow. I suppose a programmer in flow is a lot like Schrödinger’s Cat – you destroy it just by interacting with it.

Building RoaringBitmaps from Streams

RoaringBitmap is a fast compressed bitset format. In the Java implementation of Roaring, it was until recently preferential to build a bitset in one go from sorted data; there were performance penalties of varying magnitude for incremental or unordered insertions. In a recent pull request, I wanted to improve incremental monotonic insertion so I could build bitmaps from streams, but sped up unsorted batch creation significantly by accident.

Incremental Ordered Insertion

If you want to build a bitmap, you can do so efficiently with the RoaringBitmap.bitmapOf factory method.


int[] data = ...
RoaringBitmap bitmap = RoaringBitmap.bitmapOf(data);

However, I often find that I want to stream integers into a bitmap. Given that the integers being inserted into a bitmap often represent indices into an array, such a stream is likely to be monotonic. You might implement this like so:


IntStream stream = ...
RoaringBitmap bitmap = new RoaringBitmap();
stream.forEach(bitmap::add);

While this is OK, it has a few inefficiencies compared to the batch creation method.

  • Indirection: the container being written to must be located on each insertion
  • Eagerness: the cardinality must be kept up to date on each insertion
  • Allocation pressure: the best container type can’t be known in advance. Choice of container, and therefore the instance, may change as data is inserted.

You could also collect the stream into an int[] and use the batch method, but it could be a large temporary object with obvious drawbacks.

OrderedWriter

The solution I proposed is to create a writer object (OrderedWriter) which allocates a small buffer of 8KB, to use as a bitmap large enough to cover 16 bits. The stream to bitmap code becomes:


IntStream stream = ...
RoaringBitmap bitmap = new RoaringBitmap();
OrderedWriter writer = new OrderedWriter(bitmap);
stream.forEach(writer::add);
writer.flush(); // clear the buffer out

This is implemented so that changes in key (where the most significant 16 bits of each integer is stored) trigger a flush of the buffer.


  public void add(int value) {
    short key = Util.highbits(value);
    short low = Util.lowbits(value);
    if (key != currentKey) {
      if (Util.compareUnsigned(key, currentKey) < 0) {
        throw new IllegalStateException("Must write in ascending key order");
      }
      flush();
    }
    int ulow = low & 0xFFFF;
    bitmap[(ulow >>> 6)] |= (1L << ulow);
    currentKey = key;
    dirty = true;
  }

When a flush occurs, a container type is chosen and appended to the bitmap’s prefix index.


  public void flush() {
    if (dirty) {
      RoaringArray highLowContainer = underlying.highLowContainer;
      // we check that it's safe to append since RoaringArray.append does no validation
      if (highLowContainer.size > 0) {
        short key = highLowContainer.getKeyAtIndex(highLowContainer.size - 1);
        if (Util.compareUnsigned(currentKey, key) <= 0) {
          throw new IllegalStateException("Cannot write " + currentKey + " after " + key);
        }
      }
      highLowContainer.append(currentKey, chooseBestContainer());
      clearBitmap();
      dirty = false;
    }
  }

There are significant performance advantages in this approach. There is no indirection cost, and no searches in the prefix index for containers: the writes are just buffered. The buffer is small enough to fit in cache, and containers only need to be created when the writer is flushed, which happens whenever a new key is seen, or when flush is called manually. During a flush, the cardinality can be computed in one go, the best container can be chosen, and run optimisation only has to happen once. Computing the cardinality is the only bottleneck – it requires 1024 calls to Long.bitCount which can’t be vectorised in a language like Java. It can’t be incremented on insertion without either sacrificing idempotence or incurring the cost of a membership check. After the flush, the buffer needs to be cleared, using a call to Arrays.fill which is vectorised. So, despite the cost of the buffer, this can be quite efficient.

This approach isn’t universally applicable. For instance, you must write data in ascending order of the most significant 16 bits. You must also remember to flush the writer when you’re finished: until you’ve called flush, the data in the last container may not be in the bitmap. For my particular use case, this is reasonable. However, there are times when this is not fit for purpose, such as if you are occasionally inserting values and expect them to be available to queries immediately. In general, if you don’t know when you’ll stop adding data to the bitmap, this isn’t a good fit because you won’t know when to call flush.

Benchmark

The RoaringBitmap project is a big user of JMH, and most pull requests require benchmarks as evidence of performance improvement. I benchmarked the two approaches, varying bitmap sizes and randomness (likelihood of there not being a compressible run), and was amazed to find that this approach actually beats having a sorted array and using RoaringBitmap.bitmapOf. Less surprising was beating the existing API for incremental adds (this was the goal in the first place). Lower is better:

Benchmark (randomness) (size) Mode Cnt Score Error Units
buildRoaringBitmap 0.1 10000 avgt 5 54.263 3.393 us/op
buildRoaringBitmap 0.1 100000 avgt 5 355.188 15.234 us/op
buildRoaringBitmap 0.1 1000000 avgt 5 3567.839 135.149 us/op
buildRoaringBitmap 0.1 10000000 avgt 5 31982.046 1227.325 us/op
buildRoaringBitmap 0.5 10000 avgt 5 53.855 0.887 us/op
buildRoaringBitmap 0.5 100000 avgt 5 357.671 14.111 us/op
buildRoaringBitmap 0.5 1000000 avgt 5 3556.152 243.671 us/op
buildRoaringBitmap 0.5 10000000 avgt 5 34385.971 3864.143 us/op
buildRoaringBitmap 0.9 10000 avgt 5 59.354 10.385 us/op
buildRoaringBitmap 0.9 100000 avgt 5 374.245 54.485 us/op
buildRoaringBitmap 0.9 1000000 avgt 5 3712.684 657.964 us/op
buildRoaringBitmap 0.9 10000000 avgt 5 37223.976 4691.297 us/op
incrementalNativeAdd 0.1 10000 avgt 5 115.213 31.909 us/op
incrementalNativeAdd 0.1 100000 avgt 5 911.925 127.922 us/op
incrementalNativeAdd 0.1 1000000 avgt 5 8889.49 320.821 us/op
incrementalNativeAdd 0.1 10000000 avgt 5 102819.877 14247.868 us/op
incrementalNativeAdd 0.5 10000 avgt 5 116.878 28.232 us/op
incrementalNativeAdd 0.5 100000 avgt 5 947.076 128.255 us/op
incrementalNativeAdd 0.5 1000000 avgt 5 7190.443 202.012 us/op
incrementalNativeAdd 0.5 10000000 avgt 5 98843.303 4325.924 us/op
incrementalNativeAdd 0.9 10000 avgt 5 101.694 6.579 us/op
incrementalNativeAdd 0.9 100000 avgt 5 816.411 65.678 us/op
incrementalNativeAdd 0.9 1000000 avgt 5 9114.624 412.152 us/op
incrementalNativeAdd 0.9 10000000 avgt 5 108793.694 22562.527 us/op
incrementalUseOrderedWriter 0.1 10000 avgt 5 23.573 5.962 us/op
incrementalUseOrderedWriter 0.1 100000 avgt 5 289.588 36.814 us/op
incrementalUseOrderedWriter 0.1 1000000 avgt 5 2785.659 49.385 us/op
incrementalUseOrderedWriter 0.1 10000000 avgt 5 29489.758 2601.39 us/op
incrementalUseOrderedWriter 0.5 10000 avgt 5 23.57 1.536 us/op
incrementalUseOrderedWriter 0.5 100000 avgt 5 276.488 9.662 us/op
incrementalUseOrderedWriter 0.5 1000000 avgt 5 2799.408 198.77 us/op
incrementalUseOrderedWriter 0.5 10000000 avgt 5 28313.626 1976.042 us/op
incrementalUseOrderedWriter 0.9 10000 avgt 5 22.345 1.574 us/op
incrementalUseOrderedWriter 0.9 100000 avgt 5 280.205 36.987 us/op
incrementalUseOrderedWriter 0.9 1000000 avgt 5 2779.732 93.456 us/op
incrementalUseOrderedWriter 0.9 10000000 avgt 5 30568.591 2140.826 us/op

These benchmarks don’t go far enough to support replacing RoaringBitmap.bitmapOf.

Unsorted Input Data

In the cases benchmarked, this approach seems to be worthwhile. I can’t actually think of a case where someone would want to build a bitmap from unsorted data, but it occurred to me that this approach might be fast enough to cover the cost of a sort. OrderedWriter is also relaxed enough that it only needs the most significant 16 bits to be monotonic, so a full sort isn’t necessary. Implementing a radix sort on the most significant 16 bits (stable in the least significant 16 bits), prior to incremental insertion via an OrderedWriter, leads to huge increases in performance over RoaringBitmap.bitmapOf. The implementation is as follows:


  public static RoaringBitmap bitmapOfUnordered(final int... data) {
    partialRadixSort(data);
    RoaringBitmap bitmap = new RoaringBitmap();
    OrderedWriter writer = new OrderedWriter(bitmap);
    for (int i : data) {
      writer.add(i);
    }
    writer.flush();
    return bitmap;
  }

It did very well, according to benchmarks, even against various implementations of sort prior to RoaringBitmap.bitmapOf. Lower is better:

Benchmark (randomness) (size) Mode Cnt Score Error Units
bitmapOf 0.1 10000 avgt 5 1058.106 76.013 us/op
bitmapOf 0.1 100000 avgt 5 12323.905 976.68 us/op
bitmapOf 0.1 1000000 avgt 5 171812.526 9593.879 us/op
bitmapOf 0.1 10000000 avgt 5 3376296.157 170362.195 us/op
bitmapOf 0.5 10000 avgt 5 1096.663 477.795 us/op
bitmapOf 0.5 100000 avgt 5 12836.177 1674.54 us/op
bitmapOf 0.5 1000000 avgt 5 171998.126 4176 us/op
bitmapOf 0.5 10000000 avgt 5 3707804.439 974532.361 us/op
bitmapOf 0.9 10000 avgt 5 1124.881 65.673 us/op
bitmapOf 0.9 100000 avgt 5 14585.589 1894.788 us/op
bitmapOf 0.9 1000000 avgt 5 198506.813 8552.218 us/op
bitmapOf 0.9 10000000 avgt 5 3723942.934 423704.363 us/op
bitmapOfUnordered 0.1 10000 avgt 5 174.583 17.475 us/op
bitmapOfUnordered 0.1 100000 avgt 5 1768.613 86.543 us/op
bitmapOfUnordered 0.1 1000000 avgt 5 17889.705 135.714 us/op
bitmapOfUnordered 0.1 10000000 avgt 5 192645.352 6482.726 us/op
bitmapOfUnordered 0.5 10000 avgt 5 157.351 3.254 us/op
bitmapOfUnordered 0.5 100000 avgt 5 1674.919 90.138 us/op
bitmapOfUnordered 0.5 1000000 avgt 5 16900.458 778.999 us/op
bitmapOfUnordered 0.5 10000000 avgt 5 185399.32 4383.485 us/op
bitmapOfUnordered 0.9 10000 avgt 5 145.642 1.257 us/op
bitmapOfUnordered 0.9 100000 avgt 5 1515.845 82.914 us/op
bitmapOfUnordered 0.9 1000000 avgt 5 15807.597 811.048 us/op
bitmapOfUnordered 0.9 10000000 avgt 5 167863.49 3501.132 us/op
partialSortThenBitmapOf 0.1 10000 avgt 5 1060.152 168.802 us/op
partialSortThenBitmapOf 0.1 100000 avgt 5 10942.731 347.583 us/op
partialSortThenBitmapOf 0.1 1000000 avgt 5 100606.506 24705.341 us/op
partialSortThenBitmapOf 0.1 10000000 avgt 5 1035448.545 157383.713 us/op
partialSortThenBitmapOf 0.5 10000 avgt 5 1029.883 100.291 us/op
partialSortThenBitmapOf 0.5 100000 avgt 5 10472.509 832.719 us/op
partialSortThenBitmapOf 0.5 1000000 avgt 5 101144.032 16908.087 us/op
partialSortThenBitmapOf 0.5 10000000 avgt 5 958242.087 39650.946 us/op
partialSortThenBitmapOf 0.9 10000 avgt 5 1008.413 70.999 us/op
partialSortThenBitmapOf 0.9 100000 avgt 5 10458.34 600.416 us/op
partialSortThenBitmapOf 0.9 1000000 avgt 5 103945.644 2026.26 us/op
partialSortThenBitmapOf 0.9 10000000 avgt 5 1065638.269 102257.059 us/op
setupCost 0.1 10000 avgt 5 6.577 0.121 us/op
setupCost 0.1 100000 avgt 5 61.378 24.113 us/op
setupCost 0.1 1000000 avgt 5 1021.588 536.68 us/op
setupCost 0.1 10000000 avgt 5 13182.341 196.773 us/op
setupCost 0.5 10000 avgt 5 7.139 2.216 us/op
setupCost 0.5 100000 avgt 5 60.847 23.395 us/op
setupCost 0.5 1000000 avgt 5 800.888 14.711 us/op
setupCost 0.5 10000000 avgt 5 13431.625 553.44 us/op
setupCost 0.9 10000 avgt 5 6.599 0.09 us/op
setupCost 0.9 100000 avgt 5 60.946 22.511 us/op
setupCost 0.9 1000000 avgt 5 813.445 4.896 us/op
setupCost 0.9 10000000 avgt 5 13374.943 349.314 us/op
sortThenBitmapOf 0.1 10000 avgt 5 636.23 13.423 us/op
sortThenBitmapOf 0.1 100000 avgt 5 7411.756 174.264 us/op
sortThenBitmapOf 0.1 1000000 avgt 5 92299.305 3651.161 us/op
sortThenBitmapOf 0.1 10000000 avgt 5 1096374.443 162575.234 us/op
sortThenBitmapOf 0.5 10000 avgt 5 634.957 47.447 us/op
sortThenBitmapOf 0.5 100000 avgt 5 7939.074 409.328 us/op
sortThenBitmapOf 0.5 1000000 avgt 5 93505.427 5409.749 us/op
sortThenBitmapOf 0.5 10000000 avgt 5 1147933.592 57485.51 us/op
sortThenBitmapOf 0.9 10000 avgt 5 661.072 6.717 us/op
sortThenBitmapOf 0.9 100000 avgt 5 7915.506 356.148 us/op
sortThenBitmapOf 0.9 1000000 avgt 5 93403.343 5454.583 us/op
sortThenBitmapOf 0.9 10000000 avgt 5 1095960.734 85753.917 us/op

It looks like there are good performance gains available here, but these things tend to depend on particular data sets. I would be interested in hearing from anyone who has tried to use this class in a real application.

Matrix Multiplication Revisited

In a recent post, I took a look at matrix multiplication in pure Java, to see if it can go faster than reported in SIMD Intrinsics on Managed Language Runtimes. I found faster implementations than the paper’s benchmarks implied was possible. Nevertheless, I found that there were some limitations in Hotspot’s autovectoriser that I didn’t expect to see, even in JDK10. Were these limitations somehow fundamental, or can other compilers do better with essentially the same input?

I took a look at the code generated by GCC’s autovectoriser to see what’s possible in C/C++ without resorting to complicated intrinsics. For a bit of fun, I went over to the dark side to squeeze out some a lot of extra performance, which gave inspiration to a simple vectorised Java implementation which can maintain intensity as matrix size increases.

Background

The paper reports a 5x improvement in matrix multiplication throughput as a result of using LMS generated intrinsics. Using GCC as LMS’s backend, I easily reproduced very good throughput, but I found two Java implementations better than the paper’s baseline. The best performing Java implementation proposed in the paper was blocked. This post is not about the LMS benchmarks, but this code is this post’s inspiration.


public void blocked(float[] a, float[] b, float[] c, int n) {
  int BLOCK_SIZE = 8; 
  // GOOD: attempts to do as much work in submatrices
  // GOOD: tries to avoid bringing data through cache multiple times
  for (int kk = 0; kk < n; kk += BLOCK_SIZE) {
    for (int jj = 0; jj < n; jj += BLOCK_SIZE) {
      for (int i = 0; i < n; i++) {
        for (int j = jj; j < jj + BLOCK_SIZE; ++j) {
          // BAD: manual unrolling, bypasses optimisations
          float sum = c[i * n + j]; 
          for (int k = kk; k < kk + BLOCK_SIZE; ++k) {
            // BAD: second read (k * n) requires a gather - bad for cache, bad for dTLB
            // BAD: horizontal sums are inefficient
            sum += a[i * n + k] * b[k * n + j]; 
          }
          c[i * n + j] = sum;
         }
       }
    }
  }
}

I proposed the following implementation for improved cache efficiency and expected it to vectorise automatically.

public void fast(float[] a, float[] b, float[] c, int n) {
   // GOOD: 2x faster than "blocked" - why?
   int in = 0;
   for (int i = 0; i < n; ++i) {
       int kn = 0;
       for (int k = 0; k < n; ++k) {
           float aik = a[in + k];
           // MIXED: passes over c[in:in+n] multiple times per k-value, "free" if n is small
           // MIXED: reloads b[kn:kn+n] repeatedly for each i, bad if n is large, "free" if n is small
           // BAD: doesn't vectorise but should
           for (int j = 0; j < n; ++j) {
               c[in + j] += aik * b[kn + j]; // sequential writes and reads, cache and vectoriser friendly
           }
           kn += n;
       }
       in += n;
    }
}

My code actually doesn’t vectorise, even in JDK10, which really surprised me because the inner loop vectorises if the offsets are always zero. In any case, there is a simple hack involving the use of buffers, which unfortunately thrashes the cache, but narrows the field significantly.

  
  public void fastBuffered(float[] a, float[] b, float[] c, int n) {
    float[] bBuffer = new float[n];
    float[] cBuffer = new float[n];
    int in = 0;
    for (int i = 0; i < n; ++i) {
      int kn = 0;
      for (int k = 0; k < n; ++k) {
        float aik = a[in + k];
        System.arraycopy(b, kn, bBuffer, 0, n);
        saxpy(n, aik, bBuffer, cBuffer);
        kn += n;
      }
      System.arraycopy(cBuffer, 0, c, in, n);
      Arrays.fill(cBuffer, 0f);
      in += n;
    }
  }

I left the problem looking like this, with the “JDKX vectorised” lines using the algorithm above with a buffer hack:

GCC Autovectorisation

The Java code is very easy to translate into C/C++. Before looking at performance I want to get an idea of what GCC’s autovectoriser does. I want to see the code generated at GCC optimisation level 3, with unrolled loops, FMA, and AVX2, which can be seen as follows:

g++ -mavx2 -mfma -march=native -funroll-loops -O3 -S mmul.cpp

The generated assembly code can be seen in full context here. Let’s look at the mmul_saxpy routine first:

static void mmul_saxpy(const int n, const float* left, const float* right, float* result) {
    int in = 0;
    for (int i = 0; i < n; ++i) {
        int kn = 0;
        for (int k = 0; k < n; ++k) {
            float aik = left[in + k];
            for (int j = 0; j < n; ++j) {
                result[in + j] += aik * right[kn + j];
            }
            kn += n;
        }
        in += n;
    }
}

This routine uses SIMD instructions, which means in principle any other compiler could do this too. The inner loop has been unrolled, but this is only by virtue of the -funroll-loops flag. C2 does this sort of thing as standard, but only for hot loops. In general you might not want to unroll loops because of the impact on code size, and it’s great that a JIT compiler can decide only to do this when it’s profitable.

.L9:
  vmovups  (%rdx,%rax), %ymm4
  vfmadd213ps  (%rbx,%rax), %ymm3, %ymm4
  addl  $8, %r10d
  vmovaps  %ymm4, (%r11,%rax)
  vmovups  32(%rdx,%rax), %ymm5
  vfmadd213ps  32(%rbx,%rax), %ymm3, %ymm5
  vmovaps  %ymm5, 32(%r11,%rax)
  vmovups  64(%rdx,%rax), %ymm1
  vfmadd213ps  64(%rbx,%rax), %ymm3, %ymm1
  vmovaps  %ymm1, 64(%r11,%rax)
  vmovups  96(%rdx,%rax), %ymm2
  vfmadd213ps  96(%rbx,%rax), %ymm3, %ymm2
  vmovaps  %ymm2, 96(%r11,%rax)
  vmovups  128(%rdx,%rax), %ymm4
  vfmadd213ps  128(%rbx,%rax), %ymm3, %ymm4
  vmovaps  %ymm4, 128(%r11,%rax)
  vmovups  160(%rdx,%rax), %ymm5
  vfmadd213ps  160(%rbx,%rax), %ymm3, %ymm5
  vmovaps  %ymm5, 160(%r11,%rax)
  vmovups  192(%rdx,%rax), %ymm1
  vfmadd213ps  192(%rbx,%rax), %ymm3, %ymm1
  vmovaps  %ymm1, 192(%r11,%rax)
  vmovups  224(%rdx,%rax), %ymm2
  vfmadd213ps  224(%rbx,%rax), %ymm3, %ymm2
  vmovaps  %ymm2, 224(%r11,%rax)
  addq  $256, %rax
  cmpl  %r10d, 24(%rsp)
  ja  .L9

The mmul_blocked routine is compiled to quite convoluted assembly. It has a huge problem with the expression right[k * n + j], which requires a gather and is almost guaranteed to create 8 cache misses per block for large matrices. Moreover, this inefficiency gets much worse with problem size.

static void mmul_blocked(const int n, const float* left, const float* right, float* result) {
    int BLOCK_SIZE = 8;
    for (int kk = 0; kk < n; kk += BLOCK_SIZE) {
        for (int jj = 0; jj < n; jj += BLOCK_SIZE) {
            for (int i = 0; i < n; i++) {
                for (int j = jj; j < jj + BLOCK_SIZE; ++j) {
                    float sum = result[i * n + j];
                    for (int k = kk; k < kk + BLOCK_SIZE; ++k) {
                        sum += left[i * n + k] * right[k * n + j]; // second read here requires a gather
                    }
                    result[i * n + j] = sum;
                }
            }
        }
    }
}

This compiles to assembly with the unrolled vectorised loop below:

.L114:
  cmpq  %r10, %r9
  setbe  %cl
  cmpq  56(%rsp), %r8
  setnb  %dl
  orl  %ecx, %edx
  cmpq  %r14, %r9
  setbe  %cl
  cmpq  64(%rsp), %r8
  setnb  %r15b
  orl  %ecx, %r15d
  andl  %edx, %r15d
  cmpq  %r11, %r9
  setbe  %cl
  cmpq  48(%rsp), %r8
  setnb  %dl
  orl  %ecx, %edx
  andl  %r15d, %edx
  cmpq  %rbx, %r9
  setbe  %cl
  cmpq  40(%rsp), %r8
  setnb  %r15b
  orl  %ecx, %r15d
  andl  %edx, %r15d
  cmpq  %rsi, %r9
  setbe  %cl
  cmpq  32(%rsp), %r8
  setnb  %dl
  orl  %ecx, %edx
  andl  %r15d, %edx
  cmpq  %rdi, %r9
  setbe  %cl
  cmpq  24(%rsp), %r8
  setnb  %r15b
  orl  %ecx, %r15d
  andl  %edx, %r15d
  cmpq  %rbp, %r9
  setbe  %cl
  cmpq  16(%rsp), %r8
  setnb  %dl
  orl  %ecx, %edx
  andl  %r15d, %edx
  cmpq  %r12, %r9
  setbe  %cl
  cmpq  8(%rsp), %r8
  setnb  %r15b
  orl  %r15d, %ecx
  testb  %cl, %dl
  je  .L111
  leaq  32(%rax), %rdx
  cmpq  %rdx, %r8
  setnb  %cl
  cmpq  %rax, %r9
  setbe  %r15b
  orb  %r15b, %cl
  je  .L111
  vmovups  (%r8), %ymm2
  vbroadcastss  (%rax), %ymm0
  vfmadd132ps  (%r14), %ymm2, %ymm0
  vbroadcastss  4(%rax), %ymm1
  vfmadd231ps  (%r10), %ymm1, %ymm0
  vbroadcastss  8(%rax), %ymm3
  vfmadd231ps  (%r11), %ymm3, %ymm0
  vbroadcastss  12(%rax), %ymm4
  vfmadd231ps  (%rbx), %ymm4, %ymm0
  vbroadcastss  16(%rax), %ymm5
  vfmadd231ps  (%rsi), %ymm5, %ymm0
  vbroadcastss  20(%rax), %ymm2
  vfmadd231ps  (%rdi), %ymm2, %ymm0
  vbroadcastss  24(%rax), %ymm1
  vfmadd231ps  0(%rbp), %ymm1, %ymm0
  vbroadcastss  28(%rax), %ymm3
  vfmadd231ps  (%r12), %ymm3, %ymm0
  vmovups  %ymm0, (%r8)

Benchmarks

I implemented a suite of benchmarks to compare the implementations. You can run them, but since they measure throughput and intensity averaged over hundreds of iterations per matrix size, the full run will take several hours.

g++ -mavx2 -mfma -march=native -funroll-loops -O3 mmul.cpp -o mmul.exe && ./mmul.exe > results.csv

The saxpy routine wins, with blocked fading fast after a middling start.

name size throughput (ops/s) flops/cycle
blocked 64 22770.2 4.5916
saxpy 64 25638.4 5.16997
blocked 128 2736.9 4.41515
saxpy 128 4108.52 6.62783
blocked 192 788.132 4.29101
saxpy 192 1262.45 6.87346
blocked 256 291.728 3.76492
saxpy 256 521.515 6.73044
blocked 320 147.979 3.72997
saxpy 320 244.528 6.16362
blocked 384 76.986 3.35322
saxpy 384 150.441 6.55264
blocked 448 50.4686 3.4907
saxpy 448 95.0752 6.57594
blocked 512 30.0085 3.09821
saxpy 512 65.1842 6.72991
blocked 576 22.8301 3.35608
saxpy 576 44.871 6.59614
blocked 640 15.5007 3.12571
saxpy 640 32.3709 6.52757
blocked 704 12.2478 3.28726
saxpy 704 25.3047 6.79166
blocked 768 8.69277 3.02899
saxpy 768 19.8011 6.8997
blocked 832 7.29356 3.23122
saxpy 832 15.3437 6.7976
blocked 896 4.95207 2.74011
saxpy 896 11.9611 6.61836
blocked 960 3.4467 2.34571
saxpy 960 9.25535 6.29888
blocked 1024 2.02289 1.67082
saxpy 1024 6.87039 5.67463

With GCC autovectorisation, saxpy performs well, maintaining intensity as size increases, albeit well below the theoretical capacity. It would be nice if similar code could be JIT compiled in Java.

Intel Intrinsics

To understand the problem space a bit better, I find out how fast matrix multiplication can get without domain expertise by handcrafting an algorithm with intrinsics. My laptop’s Skylake chip (turbo boost and hyperthreading disabled) is capable of 32 SP flops per cycle per core – Java and the LMS implementation previously fell a long way short of that. It was difficult getting beyond 4f/c with Java, and LMS peaked at almost 6f/c before quickly tailing off. GCC autovectorisation achieved and maintained 7f/c.

To start, I’ll take full advantage of the facility to align the matrices on 64 byte intervals, since I have 64B cache lines, though this might just be voodoo. I take the saxpy routine and replace its kernel with intrinsics. Because of the -funroll-loops option, this will get unrolled without effort.

static void mmul_saxpy_avx(const int n, const float* left, const float* right, float* result) {
    int in = 0;
    for (int i = 0; i < n; ++i) {
        int kn = 0;
        for (int k = 0; k < n; ++k) {
            __m256 aik = _mm256_set1_ps(left[in + k]);
            int j = 0;
            for (; j < n; j += 8) {
                _mm256_store_ps(result + in + j, _mm256_fmadd_ps(aik, _mm256_load_ps(right + kn + j), _mm256_load_ps(result + in + j)));
            }
            for (; j < n; ++j) {
                result[in + j] += left[in + k] * right[kn + j];
            }
            kn += n;
        }
        in += n;
    }
}

This code is actually not a lot faster, if at all, than the basic saxpy above: a lot of aggressive optimisations have already been applied.

Combining Blocked and SAXPY

What makes blocked so poor is the gather and the cache miss, not the concept of blocking itself. A limiting factor for saxpy performance is that the ratio of loads to floating point operations is too high. With this in mind, I tried combining the blocking idea with saxpy, by implementing saxpy multiplications for smaller sub-matrices. This results in a different algorithm with fewer loads per floating point operation, and the inner two loops are swapped. It avoids the gather and the cache miss in blocked. Because the matrices are in row major format, I make the width of the blocks much larger than the height. Also, different heights and widths make sense depending on the size of the matrix, so I choose them dynamically. The design constraints are to avoid gathers and horizontal reduction.

static void mmul_tiled_avx(const int n, const float *left, const float *right, float *result) {
    const int block_width = n >= 256 ? 512 : 256;
    const int block_height = n >= 512 ? 8 : n >= 256 ? 16 : 32;
    for (int row_offset = 0; row_offset < n; row_offset += block_height) {
        for (int column_offset = 0; column_offset < n; column_offset += block_width) {
            for (int i = 0; i < n; ++i) {
                for (int j = column_offset; j < column_offset + block_width && j < n; j += 8) {
                    __m256 sum = _mm256_load_ps(result + i * n + j);
                    for (int k = row_offset; k < row_offset + block_height && k < n; ++k) {
                        sum = _mm256_fmadd_ps(_mm256_set1_ps(left[i * n + k]), _mm256_load_ps(right + k * n + j), sum);
                    }
                    _mm256_store_ps(result + i * n + j, sum);
                }
            }
        }
    }
}

You will see in the benchmark results that this routine really doesn’t do very well compared to saxpy. Finally, I unroll it, which is profitable despite setting -funroll-loops because there is slightly more to this than an unroll. This is a sequence of vertical reductions which have no data dependencies.

static void mmul_tiled_avx_unrolled(const int n, const float *left, const float *right, float *result) {
    const int block_width = n >= 256 ? 512 : 256;
    const int block_height = n >= 512 ? 8 : n >= 256 ? 16 : 32;
    for (int column_offset = 0; column_offset < n; column_offset += block_width) {
        for (int row_offset = 0; row_offset < n; row_offset += block_height) {
            for (int i = 0; i < n; ++i) {
                for (int j = column_offset; j < column_offset + block_width && j < n; j += 64) {
                    __m256 sum1 = _mm256_load_ps(result + i * n + j);
                    __m256 sum2 = _mm256_load_ps(result + i * n + j + 8);
                    __m256 sum3 = _mm256_load_ps(result + i * n + j + 16);
                    __m256 sum4 = _mm256_load_ps(result + i * n + j + 24);
                    __m256 sum5 = _mm256_load_ps(result + i * n + j + 32);
                    __m256 sum6 = _mm256_load_ps(result + i * n + j + 40);
                    __m256 sum7 = _mm256_load_ps(result + i * n + j + 48);
                    __m256 sum8 = _mm256_load_ps(result + i * n + j + 56);
                    for (int k = row_offset; k < row_offset + block_height && k < n; ++k) {
                        __m256 multiplier = _mm256_set1_ps(left[i * n + k]);
                        sum1 = _mm256_fmadd_ps(multiplier, _mm256_load_ps(right + k * n + j), sum1);
                        sum2 = _mm256_fmadd_ps(multiplier, _mm256_load_ps(right + k * n + j + 8), sum2);
                        sum3 = _mm256_fmadd_ps(multiplier, _mm256_load_ps(right + k * n + j + 16), sum3);
                        sum4 = _mm256_fmadd_ps(multiplier, _mm256_load_ps(right + k * n + j + 24), sum4);
                        sum5 = _mm256_fmadd_ps(multiplier, _mm256_load_ps(right + k * n + j + 32), sum5);
                        sum6 = _mm256_fmadd_ps(multiplier, _mm256_load_ps(right + k * n + j + 40), sum6);
                        sum7 = _mm256_fmadd_ps(multiplier, _mm256_load_ps(right + k * n + j + 48), sum7);
                        sum8 = _mm256_fmadd_ps(multiplier, _mm256_load_ps(right + k * n + j + 56), sum8);
                    }
                    _mm256_store_ps(result + i * n + j, sum1);
                    _mm256_store_ps(result + i * n + j + 8, sum2);
                    _mm256_store_ps(result + i * n + j + 16, sum3);
                    _mm256_store_ps(result + i * n + j + 24, sum4);
                    _mm256_store_ps(result + i * n + j + 32, sum5);
                    _mm256_store_ps(result + i * n + j + 40, sum6);
                    _mm256_store_ps(result + i * n + j + 48, sum7);
                    _mm256_store_ps(result + i * n + j + 56, sum8);
                }
            }
        }
    }
}

This final implementation is fast, and is probably as good as I am going to manage, without reading papers. This should be a CPU bound problem because the algorithm is O(n^3) whereas the problem size is O(n^2). But the flops/cycle decreases with problem size in all of these implementations. It’s possible that this could be amelioarated by a better dynamic tiling policy. I’m unlikely to be able to fix that.

It does make a huge difference being able to go very low level – handwritten intrinsics with GCC unlock awesome throughput – but it’s quite hard to actually get to the point where you can beat a good optimising compiler. Mind you, there are harder problems to solve this, and you may well be a domain expert.

The benchmark results summarise this best:

name size throughput (ops/s) flops/cycle
saxpy_avx 64 49225.7 9.92632
tiled_avx 64 33680.5 6.79165
tiled_avx_unrolled 64 127936 25.7981
saxpy_avx 128 5871.02 9.47109
tiled_avx 128 4210.07 6.79166
tiled_avx_unrolled 128 15997.6 25.8072
saxpy_avx 192 1603.84 8.73214
tiled_avx 192 1203.33 6.55159
tiled_avx_unrolled 192 4383.09 23.8638
saxpy_avx 256 633.595 8.17689
tiled_avx 256 626.157 8.0809
tiled_avx_unrolled 256 1792.52 23.1335
saxpy_avx 320 284.161 7.1626
tiled_avx 320 323.197 8.14656
tiled_avx_unrolled 320 935.571 23.5822
saxpy_avx 384 161.517 7.03508
tiled_avx 384 188.215 8.19794
tiled_avx_unrolled 384 543.235 23.6613
saxpy_avx 448 99.1987 6.86115
tiled_avx 448 118.588 8.2022
tiled_avx_unrolled 448 314 21.718
saxpy_avx 512 70.0296 7.23017
tiled_avx 512 73.2019 7.55769
tiled_avx_unrolled 512 197.815 20.4233
saxpy_avx 576 46.1944 6.79068
tiled_avx 576 50.6315 7.44294
tiled_avx_unrolled 576 126.045 18.5289
saxpy_avx 640 33.8209 6.81996
tiled_avx 640 37.0288 7.46682
tiled_avx_unrolled 640 92.784 18.7098
saxpy_avx 704 24.9096 6.68561
tiled_avx 704 27.7543 7.44912
tiled_avx_unrolled 704 69.0399 18.53
saxpy_avx 768 19.5158 6.80027
tiled_avx 768 21.532 7.50282
tiled_avx_unrolled 768 54.1763 18.8777
saxpy_avx 832 12.8635 5.69882
tiled_avx 832 14.6666 6.49766
tiled_avx_unrolled 832 37.9592 16.8168
saxpy_avx 896 12.0526 6.66899
tiled_avx 896 13.3799 7.40346
tiled_avx_unrolled 896 34.0838 18.8595
saxpy_avx 960 8.97193 6.10599
tiled_avx 960 10.1052 6.87725
tiled_avx_unrolled 960 21.0263 14.3098
saxpy_avx 1024 6.73081 5.55935
tiled_avx 1024 7.21214 5.9569
tiled_avx_unrolled 1024 12.7768 10.5531

Can we do better in Java?

Writing genuinely fast code gives an indication of how little of the processor Java actually utilises, but is it possible to bring this knowledge over to Java? The saxpy based implementations in my previous post performed well for small to medium sized matrices. Once the matrices grow, however, they become too big to be allowed to pass through cache multiple times: we need hot, small cached data to be replenished from the larger matrix. Ideally we wouldn’t need to make any copies, but it seems that the autovectoriser doesn’t like offsets: System.arraycopy is a reasonably fast compromise. The basic sequential read pattern is validated: even native code requiring a gather does not perform well for this problem. The best effort C++ code translates almost verbatim into this Java code, which is quite fast for large matrices.


public void tiled(float[] a, float[] b, float[] c, int n) {
    final int bufferSize = 512;
    final int width = Math.min(n, bufferSize);
    final int height = Math.min(n, n >= 512 ? 8 : n >= 256 ? 16 : 32);
    float[] sum = new float[bufferSize];
    float[] vector = new float[bufferSize];
    for (int rowOffset = 0; rowOffset < n; rowOffset += height) {
      for (int columnOffset = 0; columnOffset < n; columnOffset += width) {
        for (int i = 0; i < n; ++i) {
          for (int j = columnOffset; j < columnOffset + width && j < n; j += width) {
            int stride = Math.min(n - columnOffset, bufferSize);
            // copy to give autovectorisation a hint
            System.arraycopy(c, i * n + j, sum, 0, stride);
            for (int k = rowOffset; k < rowOffset + height && k < n; ++k) {
              float multiplier = a[i * n + k];
              System.arraycopy(b, k * n  + j, vector, 0, stride);
              for (int l = 0; l < stride; ++l) {
                sum[l] = Math.fma(multiplier, vector[l], sum[l]);
              }
            }
            System.arraycopy(sum, 0, c, i * n + j, stride);
          }
        }
      }
    }
  }

Benchmarking it using the same harness used in the previous post, the performance is ~10% higher for large arrays than my previous best effort. Still, the reality is that this is too slow to be useful. If you need to do linear algebra, use C/C++ for the time being!

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size flops/cycle
fastBuffered thrpt 1 10 53.331195 0.270526 ops/s 448 3.688688696
fastBuffered thrpt 1 10 34.365765 0.16641 ops/s 512 3.548072999
fastBuffered thrpt 1 10 26.128264 0.239719 ops/s 576 3.840914622
fastBuffered thrpt 1 10 19.044509 0.139197 ops/s 640 3.84031059
fastBuffered thrpt 1 10 14.312154 1.045093 ops/s 704 3.841312378
fastBuffered thrpt 1 10 7.772745 0.074598 ops/s 768 2.708411991
fastBuffered thrpt 1 10 6.276182 0.067338 ops/s 832 2.780495238
fastBuffered thrpt 1 10 4.8784 0.067368 ops/s 896 2.699343067
fastBuffered thrpt 1 10 4.068907 0.038677 ops/s 960 2.769160387
fastBuffered thrpt 1 10 2.568101 0.108612 ops/s 1024 2.121136502
tiled thrpt 1 10 56.495366 0.584872 ops/s 448 3.907540754
tiled thrpt 1 10 30.884954 3.221017 ops/s 512 3.188698735
tiled thrpt 1 10 15.580581 0.412654 ops/s 576 2.290381075
tiled thrpt 1 10 9.178969 0.841178 ops/s 640 1.850932038
tiled thrpt 1 10 12.229763 0.350233 ops/s 704 3.282408783
tiled thrpt 1 10 9.371032 0.330742 ops/s 768 3.265334889
tiled thrpt 1 10 7.727068 0.277969 ops/s 832 3.423271628
tiled thrpt 1 10 6.076451 0.30305 ops/s 896 3.362255222
tiled thrpt 1 10 4.916811 0.2823 ops/s 960 3.346215151
tiled thrpt 1 10 3.722623 0.26486 ops/s 1024 3.074720008