Floating Point: Manual Unrolling or Autovectorisation?

Java is very strict about floating point arithmetic. There’s even a keyword, strictfp, which allows you to make it stricter, ensuring you’ll get a potentially less precise but identical result wherever you run your program. There’s actually a JEP to make this the only behaviour. JLS 15.18.2 states clearly that floating point addition is not associative in Java, which means that JIT compilers have to respect the order of double[]s and float[]s when compiling code, even if it turns out the order is actually arbitrary to the application. This means they can’t vectorise or even pipeline loops containing interdependent floating point addition, can’t distribute multiplications over additions, can’t telescopically collapse multiplications: the list goes on. If you want this code to go faster you must work around the JIT compiler somehow. In C++, it’s possible to choose to treat floating point numbers as if they had the algebraic properties of the Reals. Using this option is often maligned, perhaps because it assumes much more than associativity, and applies to entire compilation units. A proposal for fastfp semantics at a class and method scope was withdrawn a long time ago.

Prior to the arrival of the Vector API, I’m interested in which vectorisation transformations are possible automatically. This is because I’ve found that many bottlenecks in applications are related to low single threaded performance, and others come from the premature usage of threads to solve these performance problems. Imagine how powerful multithreading could be if used only after saturating the intensity available on each core?

There are certain things you just can’t do in Java because of the language specification, and one of these is getting C2 to pipeline two or more dependent vpaddd instructions, so the maximum achievable floating point intensity is quite low. In fact, you can be better off giving up on autovectorisation and unrolling the loop yourself, allowing the pipelining of scalar additions. This depends intimately on your microarchitecture.

Double Precision Sum Product

C2 can automatically vectorise a sum product between two double[]s. It does this by issuing eight vmovdqu loads at once, then four vpmulpd multiplications at once, and then a long in-order scalar reduction. It does this with the simplest possible code, potentially rewarding simplicity with decent performance:


  @Benchmark
  public double vectorisedDoubleSumProduct() {
    double sp = 0D;
    for (int i = 0; i < xd.length && i < yd.length; ++i) {
      sp += xd[i] * yd[i];
    }
    return sp;
  }

Perfasm shows the problematic scalar reduction quite clearly:


....[Hottest Region 1]..............................................................................
c2, com.openkappa.simd.sumproduct.generated.SumProduct_vectorisedDoubleSumProduct_jmhTest::vectorisedDoubleSumProduct_thrpt_jmhStub, version 164 (238 bytes) 

           0x00000144f724f8e9: mov     r8d,r9d
           0x00000144f724f8ec: add     r8d,0fffffff1h
           0x00000144f724f8f0: cmp     r9d,r8d
           0x00000144f724f8f3: mov     edx,80000000h
           0x00000144f724f8f8: cmovl   r8d,edx
           0x00000144f724f8fc: cmp     r11d,r8d
           0x00000144f724f8ff: jnl     144f724f7dfh
           0x00000144f724f905: nop     word ptr [rax+rax+0h]  ;*iload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::vectorisedDoubleSumProduct@13 (line 43)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_vectorisedDoubleSumProduct_jmhTest::vectorisedDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  0.00%    0x00000144f724f910: vmovdqu ymm1,ymmword ptr [rsi+r11*8+70h]
  1.07%    0x00000144f724f917: vmovdqu ymm2,ymmword ptr [rax+r11*8+70h]
  2.49%    0x00000144f724f91e: vmovdqu ymm3,ymmword ptr [rsi+r11*8+50h]
  0.67%    0x00000144f724f925: vmovdqu ymm4,ymmword ptr [rax+r11*8+50h]
  0.75%    0x00000144f724f92c: vmovdqu ymm5,ymmword ptr [rsi+r11*8+30h]
  0.01%    0x00000144f724f933: vmovdqu ymm6,ymmword ptr [rax+r11*8+30h]
  1.52%    0x00000144f724f93a: vmovdqu ymm7,ymmword ptr [rsi+r11*8+10h]
                                                         ;*daload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::vectorisedDoubleSumProduct@34 (line 44)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_vectorisedDoubleSumProduct_jmhTest::vectorisedDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  0.01%    0x00000144f724f941: vmovdqu ymm8,ymmword ptr [rax+r11*8+10h]
           0x00000144f724f948: vmulpd  ymm9,ymm2,ymm1
  0.02%    0x00000144f724f94c: vmulpd  ymm7,ymm8,ymm7
  1.50%    0x00000144f724f950: vmulpd  ymm8,ymm4,ymm3
  0.02%    0x00000144f724f954: vmulpd  ymm10,ymm6,ymm5
  0.00%    0x00000144f724f958: vaddsd  xmm0,xmm0,xmm7
  1.51%    0x00000144f724f95c: vpshufd xmm1,xmm7,0eh
  0.00%    0x00000144f724f961: vaddsd  xmm0,xmm0,xmm1
  5.91%    0x00000144f724f965: vextractf128 xmm4,ymm7,1h
  0.01%    0x00000144f724f96b: vaddsd  xmm0,xmm0,xmm4
  6.56%    0x00000144f724f96f: vpshufd xmm1,xmm4,0eh
  0.01%    0x00000144f724f974: vaddsd  xmm0,xmm0,xmm1
  5.75%    0x00000144f724f978: vaddsd  xmm0,xmm0,xmm10
  5.71%    0x00000144f724f97d: vpshufd xmm4,xmm10,0eh
  0.00%    0x00000144f724f983: vaddsd  xmm0,xmm0,xmm4
  5.89%    0x00000144f724f987: vextractf128 xmm6,ymm10,1h
  0.01%    0x00000144f724f98d: vaddsd  xmm0,xmm0,xmm6
  6.01%    0x00000144f724f991: vpshufd xmm4,xmm6,0eh
  0.01%    0x00000144f724f996: vaddsd  xmm0,xmm0,xmm4
  5.95%    0x00000144f724f99a: vaddsd  xmm0,xmm0,xmm8
  5.98%    0x00000144f724f99f: vpshufd xmm1,xmm8,0eh
  0.01%    0x00000144f724f9a5: vaddsd  xmm0,xmm0,xmm1
  5.99%    0x00000144f724f9a9: vextractf128 xmm5,ymm8,1h
  0.00%    0x00000144f724f9af: vaddsd  xmm0,xmm0,xmm5
  5.93%    0x00000144f724f9b3: vpshufd xmm1,xmm5,0eh
  0.00%    0x00000144f724f9b8: vaddsd  xmm0,xmm0,xmm1
  6.05%    0x00000144f724f9bc: vaddsd  xmm0,xmm0,xmm9
  5.92%    0x00000144f724f9c1: vpshufd xmm3,xmm9,0eh
  0.00%    0x00000144f724f9c7: vaddsd  xmm0,xmm0,xmm3
  6.05%    0x00000144f724f9cb: vextractf128 xmm2,ymm9,1h
  0.00%    0x00000144f724f9d1: vaddsd  xmm0,xmm0,xmm2
  6.05%    0x00000144f724f9d5: vpshufd xmm3,xmm2,0eh
  0.00%    0x00000144f724f9da: vaddsd  xmm0,xmm0,xmm3    ;*dadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::vectorisedDoubleSumProduct@36 (line 44)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_vectorisedDoubleSumProduct_jmhTest::vectorisedDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  6.05%    0x00000144f724f9de: add     r11d,10h          ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::vectorisedDoubleSumProduct@38 (line 43)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_vectorisedDoubleSumProduct_jmhTest::vectorisedDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  0.00%    0x00000144f724f9e2: cmp     r11d,r8d
           0x00000144f724f9e5: jl      144f724f910h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::vectorisedDoubleSumProduct@10 (line 43)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_vectorisedDoubleSumProduct_jmhTest::vectorisedDoubleSumProduct_thrpt_jmhStub@17 (line 119)
           0x00000144f724f9eb: mov     edx,r9d
           0x00000144f724f9ee: add     edx,0fffffffdh
           0x00000144f724f9f1: cmp     r9d,edx
           0x00000144f724f9f4: mov     r9d,80000000h
           0x00000144f724f9fa: cmovl   edx,r9d
  0.00%    0x00000144f724f9fe: cmp     r11d,edx
           0x00000144f724fa01: jl      144f724f7a4h
           0x00000144f724fa07: jmp     144f724f7dfh
           0x00000144f724fa0c: vxorpd  xmm0,xmm0,xmm0
           0x00000144f724fa10: jmp     144f724f807h
           0x00000144f724fa15: mov     edx,0ffffff86h
           0x00000144f724fa1a: mov     qword ptr [rsp+70h],rcx
           0x00000144f724fa1f: push    qword ptr [rsp+80h]
           0x00000144f724fa27: pop     qword ptr [rsp+78h]
           0x00000144f724fa2c: push    qword ptr [rsp+30h]
           0x00000144f724fa31: pop     qword ptr [rsp+28h]
....................................................................................................
 99.44%  <total for region 1>

Unrolling this will disable autovectorisation, but my Skylake chip can do 8 scalar floating point operations at once.


  @Benchmark
  public double unrolledDoubleSumProduct() {
    double sp1 = 0D;
    double sp2 = 0D;
    double sp3 = 0D;
    double sp4 = 0D;
    for (int i = 0; i < xd.length && i < yd.length; i += 4) {
      sp1 += xd[i] * yd[i];
      sp2 += xd[i + 1] * yd[i + 1];
      sp3 += xd[i + 2] * yd[i + 2];
      sp4 += xd[i + 3] * yd[i + 3];
    }
    return sp1 + sp2 + sp3 + sp4;
  }

Looking at the perfasm output, you can see this code is scalar but the additions are interleaved without interdependencies. This is enough to take down a crippled target.


....[Hottest Region 1]..............................................................................
c2, com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub, version 162 (79 bytes) 

                                                         ;   {section_word}
           0x0000024719311ddb: lea     r9,[r12+rcx*8]
           0x0000024719311ddf: lea     rcx,[r12+rbx*8]
           0x0000024719311de3: cmp     r11d,4h
           0x0000024719311de7: jle     24719311ee1h
           0x0000024719311ded: mov     r8d,4h
           0x0000024719311df3: nop     word ptr [rax+rax+0h]
           0x0000024719311dfc: nop                       ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@23 (line 55)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  4.20%    0x0000024719311e00: vmovsd  xmm0,qword ptr [rcx+r8*8+28h]
                                                         ;*daload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@116 (line 59)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  8.93%    0x0000024719311e07: vmulsd  xmm0,xmm0,mmword ptr [r9+r8*8+28h]
 15.10%    0x0000024719311e0e: vmovsd  xmm1,qword ptr [rcx+r8*8+18h]
                                                         ;*daload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@69 (line 57)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  3.95%    0x0000024719311e15: vmulsd  xmm1,xmm1,mmword ptr [r9+r8*8+18h]
  9.91%    0x0000024719311e1c: vmovsd  xmm2,qword ptr [rcx+r8*8+20h]
                                                         ;*daload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@92 (line 58)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  3.97%    0x0000024719311e23: vmulsd  xmm2,xmm2,mmword ptr [r9+r8*8+20h]
 11.04%    0x0000024719311e2a: vmovsd  xmm3,qword ptr [rcx+r8*8+10h]
                                                         ;*daload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@47 (line 56)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  3.86%    0x0000024719311e31: vmulsd  xmm3,xmm3,mmword ptr [r9+r8*8+10h]
  9.04%    0x0000024719311e38: vaddsd  xmm4,xmm4,xmm0    ;*dadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@118 (line 59)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  7.11%    0x0000024719311e3c: vaddsd  xmm7,xmm7,xmm3    ;*dadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@49 (line 56)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  7.03%    0x0000024719311e40: vaddsd  xmm5,xmm5,xmm2    ;*dadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@94 (line 58)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  7.29%    0x0000024719311e44: vaddsd  xmm6,xmm6,xmm1    ;*dadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@71 (line 57)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  3.58%    0x0000024719311e48: add     r8d,4h            ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@121 (line 55)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
  4.39%    0x0000024719311e4c: cmp     r8d,r11d
  0.00%    0x0000024719311e4f: jl      24719311e00h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@20 (line 55)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
           0x0000024719311e51: cmp     r8d,edi
           0x0000024719311e54: jnl     24719311c92h
           0x0000024719311e5a: nop                       ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@23 (line 55)
                                                         ; - com.openkappa.simd.sumproduct.generated.SumProduct_unrolledDoubleSumProduct_jmhTest::unrolledDoubleSumProduct_thrpt_jmhStub@17 (line 119)
           0x0000024719311e5c: cmp     r8d,r10d
           0x0000024719311e5f: jnl     24719311f15h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.sumproduct.SumProduct::unrolledDoubleSumProduct@30 (line 55)
....................................................................................................
 99.39%  <total for region 1>

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
unrolledDoubleSumProduct thrpt 1 10 1912.140438 48.445308 ops/ms 1024
unrolledDoubleSumProduct thrpt 1 10 24.677122 0.510459 ops/ms 65536
vectorisedDoubleSumProduct thrpt 1 10 647.848021 10.508824 ops/ms 1024
vectorisedDoubleSumProduct thrpt 1 10 9.474097 0.479281 ops/ms 65536

So, if you realise that in your application the order of your array is irrelevant, you can write a tiny bit of extra code and get multiplicatively better performance. These results were produced with JDK9. When I tried with JDK10, the sum product was not vectorised, presumably because it has been noticed that it is unprofitable. This benchmark can be seen in full context at github.

Vertical Sum

I was motivated to write this post after Ioannis Tsakpinis shared a gist of a benchmark after reading a recent post about coaxing vectorisation into action for a simple floating point sum. The post was intended to be a prelude to a post about the wonders of paginated arrays. With a paginated array, autovectorisation pays off and is preferable to a manual unroll. The non-associativity of the operation is of course still violated, but I am working on the premise that this virtually never matters. I revisited this benchmark, with a paginated array this time.


  @Benchmark // inspired by Ioannis' code
  public double reduceUnrolledPaginated() {
    double a0 = 0.0;
    double a1 = 0.0;
    double a2 = 0.0;
    double a3 = 0.0;
    for (int i = 0; i < paginated.length; ++i) {
      double[] page = paginated[i];
      for (int j = 0; j < paginated[0].length; j += 4) {
        a0 += page[j + 0];
        a1 += page[j + 1];
        a2 += page[j + 2];
        a3 += page[j + 3];
      }
    }
    return a0 + a1 + a2 + a3;
  }

  @Benchmark
  public double reducePaginated() {
    double[] buffer = Arrays.copyOf(paginated[0], paginated[0].length);
    for (int i = 1; i < paginated.length; ++i) {
      double[] page = paginated[i];
      for (int j = 0; j < page.length && j < buffer.length; ++j) {
        buffer[j] += page[j];
      }
    }
    return reduceUnrolled(buffer);
  }

The array being paginated, requiring no offset calculations, is the perfect case for a vectorised loop here. Which is one reason why Java arrays should be paginated in application code.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
reducePaginated thrpt 1 10 597.046492 23.080803 ops/ms 1024
reducePaginated thrpt 1 10 42.801021 0.831318 ops/ms 65536
reducePaginated thrpt 1 10 1.503510 0.187167 ops/ms 1048576
reduceUnrolledPaginated thrpt 1 10 1311.433592 9.063721 ops/ms 1024
reduceUnrolledPaginated thrpt 1 10 19.448202 0.503753 ops/ms 65536
reduceUnrolledPaginated thrpt 1 10 1.052183 0.086555 ops/ms 1048576

Nevertheless, loop unrolling can be a significant boon for floating point arithmetic in Java. It feels dirty to me – it’s one of those things that people did before my time. Compilers know how to do it, if they are allowed to do so. If there is no place for fastfp in Java, I imagine they are here to stay.

Posted on

Iterating Over a Bitset in Java

How fast can you iterate over a bitset? Daniel Lemire published a benchmark recently in support of a strategy using the number of trailing zeroes to skip over empty bits. I have used the same technique in Java several times in my hobby project SplitMap and this is something I am keen to optimise. I think that the best strategy depends on what you want to do with the set bits, and how sparse and uniformly distributed they are. I argue that the cost of iteration is less important than the constraints your API imposes on the caller, and whether the caller is free to exploit patterns in the data.

C2 Generates Good Code

If you think C++ is much faster than Java, you either don’t know much about Java or do lots of floating point arithmetic. This isn’t about benchmarking C++ against Java, but comparing the compilation outputs for a C++ implementation and a Java implementation shows that there won’t be much difference if your Java method gets hot. Only the time to performance will differ, and this is amortised over the lifetime of an application. The trailing zeroes implementation is probably the fastest technique in Java as well as in C++, but that is to ignore the optimisations you can’t apply to the callback if you use it too literally.

Compiling this C++ function with GCC yields the snippet of assembly taken from the loop kernel:


template <typename CALLBACK>
static void for_each(const long* bitmap, const int size, const CALLBACK& callback) {
    for (size_t k = 0; k < size; ++k) {
        long bitset = bitmap[k];
        while (bitset != 0) {
            callback((k * 64) + __builtin_ctzl(bitset));
            bitset ^= (bitset & -bitset);
        }
    }
}

The instruction tzcntl calculates the next set bit and blsr switches it off.


.L99:
  movq  %rdi, %rcx
  blsr  %ebx, %ebx
  call  _ZNSo3putEc
  movq  %rax, %rcx
  call  _ZNSo5flushEv
  testl  %ebx, %ebx
  je  .L96
.L100:
  xorl  %edx, %edx
  movq  %r12, %rcx
  tzcntl  %ebx, %edx
  addl  %ebp, %edx
  call  _ZNSolsEi
  movq  %rax, %rdi
  movq  (%rax), %rax
  movq  -24(%rax), %rax
  movq  240(%rdi,%rax), %rsi
  testq  %rsi, %rsi
  je  .L108
  cmpb  $0, 56(%rsi)
  jne  .L109
  movq  %rsi, %rcx
  call  _ZNKSt5ctypeIcE13_M_widen_initEv
  movq  (%rsi), %rax
  movl  $10, %edx
  movq  48(%rax), %rax
  cmpq  %r14, %rax
  je  .L99
  movq  %rsi, %rcx
  call  *%rax
  movsbl  %al, %edx
  jmp  .L99
  .p2align 4,,10

In Java, almost identical code is generated.


public void forEach(long[] bitmap, IntConsumer consumer) {
    for (int i = 0; i < bitmap.length; ++i) {
      long word = bitmap[i];
      while (word != 0) {
        consumer.accept(Long.SIZE * i + Long.numberOfTrailingZeros(word));
        word ^= Long.lowestOneBit(word);
      }
    }
  }

The key difference is that xor and blsi haven’t been fused into blsr, so the C++ code is probably slightly faster. A lambda function accumulating the contents of an array is inlined into this loop (the add comes from an inlined lambda, but notice how little time is spent adding compared to computing the bit to switch off in this sample produced by perfasm).


   .83%    0x000002d79d366a19: tzcnt   r9,rcx
  8.53%    0x000002d79d366a1e: add     r9d,ebx
  0.42%    0x000002d79d366a21: cmp     r9d,r8d
  0.00%    0x000002d79d366a24: jnb     2d79d366a4dh
  0.62%    0x000002d79d366a26: add     r10d,dword ptr [rdi+r9*4+10h]
 16.22%    0x000002d79d366a2b: vmovq   r11,xmm4
  6.68%    0x000002d79d366a30: mov     dword ptr [r11+10h],r10d
 27.92%    0x000002d79d366a34: blsi    r10,rcx
  0.55%    0x000002d79d366a39: xor     rcx,r10         
  0.10%    0x000002d79d366a3c: mov     r11,qword ptr [r15+70h]  

It’s this Java code, and its impact on which optimisations can be applied to the IntConsumer that this post focuses on. There are different principles, particularly related to inlining and vectorisation opportunities in C++, but this blog is about Java. Depending on what your callback does, you get different benchmark results and you should make different choices about how to do the iteration: you just can’t assess this in isolation.

Special Casing -1

Imagine you have an int[] containing data, and you are iterating over a mask or materialised predicate over that data. For each set bit, you want to add the corresponding entry in the array to a sum. In Java, that looks like this (you’ve already seen the generated assembly above):


  @Benchmark
  public int reduce() {
    int[] result = new int[1];
    forEach(bitmap, i -> result[0] += data[i]);
    return result[0];
  }

How fast can this get? It obviously depends on how full the bitset is. The worst case would be that it’s completely full, and it couldn’t get much better than if only one bit per word were set. The difference is noticeable, but scales by a factor less than the number of bits:

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: scenario
reduce thrpt 1 10 7.435909 0.017491 ops/ms FULL
reduce thrpt 1 10 260.305307 6.081961 ops/ms ONE_BIT_PER_WORD

But the important code here, the callback itself, is stuck at entry level compilation. There is no unrolling, no vectorisation, the adds can’t be pipelined because there is a data dependency on blsi and xor. We can do much better in some cases, and not much worse in others, just by treating -1 as a special case, profiting from optimisations that can now be applied inside the callback. Passing a different callback which consumes whole words costs a branch, but it’s often worth it. Here’s the iterator now:


  interface WordConsumer {
    void acceptWord(int wordIndex, long word);
  }

  public void forEach(long[] bitmap, IntConsumer intConsumer, WordConsumer wordConsumer) {
    for (int i = 0; i < bitmap.length; ++i) {
      long word = bitmap[i];
      if (word == -1L) {
        wordConsumer.acceptWord(i, word);
      } else {
        while (word != 0) {
          intConsumer.accept(Long.SIZE * i + Long.numberOfTrailingZeros(word));
          word ^= Long.lowestOneBit(word);
        }
      }
    }
  }

  @Benchmark
  public int reduceWithWordConsumer() {
    int[] result = new int[1];
    forEach(bitmap, i -> result[0] += data[i], (index, word) -> {
      if (word != -1L) {
        throw new IllegalStateException();
      }
      int sum = 0;
      for (int i = index * Long.SIZE; i < (index + 1) * Long.SIZE; ++i) {
        sum += data[i];
      }
      result[0] += sum;
    });
    return result[0];
  }

This really pays off when the bitset is full, but having that extra branch does seem to cost something even though it is never taken, whereas the full case improves 6x.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: scenario
reduce thrpt 1 10 7.401202 0.118648 ops/ms FULL
reduce thrpt 1 10 261.682016 4.155856 ops/ms ONE_BIT_PER_WORD
reduceWithWordConsumer thrpt 1 10 43.972759 0.993264 ops/ms FULL
reduceWithWordConsumer thrpt 1 10 222.824868 4.877147 ops/ms ONE_BIT_PER_WORD

We still don’t actually know the cost of the branch when it’s taken every now and then. To estimate it, we need a new scenario (or new scenarios) which mix full and sparse words. As you might expect, having the WordConsumer is great when one word in every few is full: the fast path is so much faster, it practically skips the word.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: scenario
reduce thrpt 1 10 157.358633 4.538679 ops/ms SPARSE_16_FULL_WORDS
reduceWithWordConsumer thrpt 1 10 257.041035 7.446404 ops/ms SPARSE_16_FULL_WORDS

So in this scenario, the branch has paid for itself. How? The data dependency has been removed with a countable loop. Here’s the perfasm output. Notice two things: long runs of add instructions, and the vastly reduced percentage against blsi. The time is now spent adding numbers up, not switching off least significant bits. This feels like progress.


....[Hottest Region 1]..............................................................................
c2, com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub, version 170 (804 bytes) 

                                                         ;   {optimized virtual_call}
           0x000001dd5b35acd4: mov     r9,qword ptr [rsp+0a0h]
           0x000001dd5b35acdc: movzx   r11d,byte ptr [r9+94h]  ;*getfield isDone {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@30 (line 144)
                                                         ; implicit exception: dispatches to 0x000001dd5b35b486
           0x000001dd5b35ace4: test    r11d,r11d
           0x000001dd5b35ace7: jne     1dd5b35b19dh      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@33 (line 144)
           0x000001dd5b35aced: mov     ebx,1h
           0x000001dd5b35acf2: jmp     1dd5b35ad31h
  0.00%    0x000001dd5b35acf4: vmovq   rdi,xmm0          ;*invokevirtual reduceWithWordConsumer {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35acf9: mov     r8d,dword ptr [rdi+10h]  ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@28 (line 85)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001dd5b35acfd: mov     rdx,qword ptr [rsp+30h]
  0.03%    0x000001dd5b35ad02: nop
  0.00%    0x000001dd5b35ad03: call    1dd5b2c7000h      ; ImmutableOopMap{[160]=Oop [168]=Oop [176]=Oop [40]=NarrowOop [48]=Oop }
                                                         ;*invokevirtual consume {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@20 (line 142)
                                                         ;   {optimized virtual_call}
  0.04%    0x000001dd5b35ad08: mov     r9,qword ptr [rsp+0a0h]
           0x000001dd5b35ad10: movzx   r11d,byte ptr [r9+94h]  ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.02%    0x000001dd5b35ad18: mov     r10,qword ptr [r15+70h]
  0.00%    0x000001dd5b35ad1c: mov     rbx,qword ptr [rsp+40h]
  0.03%    0x000001dd5b35ad21: add     rbx,1h            ; ImmutableOopMap{r9=Oop [160]=Oop [168]=Oop [176]=Oop [40]=NarrowOop [48]=Oop }
                                                         ;*ifeq {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@33 (line 144)
  0.00%    0x000001dd5b35ad25: test    dword ptr [r10],eax  ;   {poll}
  0.00%    0x000001dd5b35ad28: test    r11d,r11d
           0x000001dd5b35ad2b: jne     1dd5b35b1a2h
  0.00%    0x000001dd5b35ad31: mov     rdi,qword ptr [r15+80h]
  0.01%    0x000001dd5b35ad38: mov     r10,rdi
  0.00%    0x000001dd5b35ad3b: add     r10,18h
  0.01%    0x000001dd5b35ad3f: cmp     r10,qword ptr [r15+90h]
           0x000001dd5b35ad46: jnb     1dd5b35b119h      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ad4c: mov     qword ptr [r15+80h],r10
           0x000001dd5b35ad53: prefetchw byte ptr [r10+0c0h]
  0.00%    0x000001dd5b35ad5b: mov     qword ptr [rdi],1h
  0.01%    0x000001dd5b35ad62: prefetchw byte ptr [r10+100h]
           0x000001dd5b35ad6a: mov     dword ptr [rdi+8h],0f8000168h
                                                         ;   {metadata({type array int})}
  0.00%    0x000001dd5b35ad71: prefetchw byte ptr [r10+140h]
  0.01%    0x000001dd5b35ad79: mov     dword ptr [rdi+0ch],1h
  0.01%    0x000001dd5b35ad80: prefetchw byte ptr [r10+180h]
  0.00%    0x000001dd5b35ad88: mov     qword ptr [rdi+10h],r12  ;*newarray {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@1 (line 74)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ad8c: mov     rcx,qword ptr [rsp+0b0h]
  0.00%    0x000001dd5b35ad94: mov     ebp,dword ptr [rcx+0ch]  ;*getfield bitmap {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@6 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ad97: mov     dword ptr [rsp+70h],ebp
  0.00%    0x000001dd5b35ad9b: mov     rsi,qword ptr [r15+80h]
  0.00%    0x000001dd5b35ada2: mov     r10,rsi
           0x000001dd5b35ada5: add     r10,18h
  0.00%    0x000001dd5b35ada9: cmp     r10,qword ptr [r15+90h]
           0x000001dd5b35adb0: jnb     1dd5b35b14fh      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35adb6: mov     qword ptr [r15+80h],r10
           0x000001dd5b35adbd: prefetchw byte ptr [r10+0c0h]
  0.00%    0x000001dd5b35adc5: mov     r8,qword ptr [rsp+20h]
  0.01%    0x000001dd5b35adca: mov     r10,qword ptr [r8+0b0h]
  0.01%    0x000001dd5b35add1: mov     qword ptr [rsi],r10
  0.00%    0x000001dd5b35add4: mov     dword ptr [rsi+8h],0f8021906h
                                                         ;   {metadata(&apos;com/openkappa/simd/iterate/BitSetIterator$$Lambda$43&apos;)}
           0x000001dd5b35addb: mov     qword ptr [rsi+10h],r12  ;*new {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@0
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35addf: mov     r11d,dword ptr [rsp+28h]
  0.00%    0x000001dd5b35ade4: mov     dword ptr [rsi+0ch],r11d
                                                         ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ade8: mov     rax,qword ptr [r15+80h]
  0.00%    0x000001dd5b35adef: mov     rbp,rdi
  0.01%    0x000001dd5b35adf2: shr     rbp,3h
  0.00%    0x000001dd5b35adf6: mov     dword ptr [rsi+10h],ebp  ;*putfield arg$2 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@11
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35adf9: mov     r10,rax
  0.00%    0x000001dd5b35adfc: add     r10,18h
  0.00%    0x000001dd5b35ae00: cmp     r10,qword ptr [r15+90h]
           0x000001dd5b35ae07: jnb     1dd5b35b08eh      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ae0d: mov     qword ptr [r15+80h],r10
  0.00%    0x000001dd5b35ae14: prefetchw byte ptr [r10+0c0h]
  0.00%    0x000001dd5b35ae1c: mov     rdx,qword ptr [rsp+38h]
  0.00%    0x000001dd5b35ae21: mov     r10,qword ptr [rdx+0b0h]
  0.02%    0x000001dd5b35ae28: mov     qword ptr [rax],r10
  0.01%    0x000001dd5b35ae2b: mov     dword ptr [rax+8h],0f8021a86h
                                                         ;   {metadata(&apos;com/openkappa/simd/iterate/BitSetIterator$$Lambda$44&apos;)}
  0.00%    0x000001dd5b35ae32: mov     qword ptr [rax+10h],r12
  0.00%    0x000001dd5b35ae36: mov     dword ptr [rax+0ch],r11d
                                                         ;*new {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::get$Lambda@0
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@18 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ae3a: mov     dword ptr [rax+10h],ebp  ;*invokespecial &lt;init&gt; {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@18 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001dd5b35ae3d: mov     ebp,dword ptr [rsp+70h]
  0.00%    0x000001dd5b35ae41: mov     r13d,dword ptr [r12+rbp*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@6 (line 104)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001dd5b35b422
  0.03%    0x000001dd5b35ae46: test    r13d,r13d
           0x000001dd5b35ae49: jbe     1dd5b35b024h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@7 (line 104)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ae4f: mov     r10d,r13d
  0.00%    0x000001dd5b35ae52: dec     r10d
           0x000001dd5b35ae55: cmp     r10d,r13d
           0x000001dd5b35ae58: jnb     1dd5b35b24ah
  0.01%    0x000001dd5b35ae5e: lea     r14,[r12+rbp*8]   ;*getfield bitmap {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@6 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001dd5b35ae62: xor     r10d,r10d
  0.00%    0x000001dd5b35ae65: xor     ebp,ebp
           0x000001dd5b35ae67: mov     qword ptr [rsp+0b0h],rcx
  0.00%    0x000001dd5b35ae6f: mov     qword ptr [rsp+0a0h],r9
           0x000001dd5b35ae77: mov     qword ptr [rsp+20h],r8
  0.01%    0x000001dd5b35ae7c: mov     dword ptr [rsp+28h],r11d
  0.00%    0x000001dd5b35ae81: mov     qword ptr [rsp+38h],rdx
  0.00%    0x000001dd5b35ae86: mov     qword ptr [rsp+40h],rbx
  0.00%    0x000001dd5b35ae8b: vmovq   xmm0,rdi
  0.00%    0x000001dd5b35ae90: jmp     1dd5b35afafh
  0.03%    0x000001dd5b35ae95: mov     r8d,r10d
  0.03%    0x000001dd5b35ae98: add     r8d,40h           ;*imul {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@32 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.04%    0x000001dd5b35ae9c: cmp     r10d,r8d
           0x000001dd5b35ae9f: jnl     1dd5b35b01dh      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@33 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.03%    0x000001dd5b35aea5: mov     r11d,dword ptr [r12+rcx*8+0ch]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@44 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001dd5b35b1d3
  0.04%    0x000001dd5b35aeaa: cmp     r10d,r11d
           0x000001dd5b35aead: jnb     1dd5b35b1d3h
  0.04%    0x000001dd5b35aeb3: movsxd  r11,r11d
  0.04%    0x000001dd5b35aeb6: movsxd  r9,r8d
  0.02%    0x000001dd5b35aeb9: dec     r9
  0.05%    0x000001dd5b35aebc: cmp     r9,r11
  0.00%    0x000001dd5b35aebf: jnb     1dd5b35b1d3h
  0.04%    0x000001dd5b35aec5: lea     rdi,[r12+rcx*8]
  0.04%    0x000001dd5b35aec9: mov     ecx,r10d
  0.03%    0x000001dd5b35aecc: inc     ecx
  0.04%    0x000001dd5b35aece: mov     r9d,r10d
  0.04%    0x000001dd5b35aed1: xor     ebx,ebx           ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@36 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.03%    0x000001dd5b35aed3: add     ebx,dword ptr [rdi+r9*4+10h]
                                                         ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@45 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.20%    0x000001dd5b35aed8: inc     r9d               ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@48 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.04%    0x000001dd5b35aedb: cmp     r9d,ecx
           0x000001dd5b35aede: jl      1dd5b35aed3h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@33 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.04%    0x000001dd5b35aee0: add     r10d,31h
  0.03%    0x000001dd5b35aee4: cmp     r8d,r10d
  0.03%    0x000001dd5b35aee7: mov     r11d,80000000h
  0.04%    0x000001dd5b35aeed: cmovl   r10d,r11d
  0.04%    0x000001dd5b35aef1: cmp     r9d,r10d
           0x000001dd5b35aef4: jnl     1dd5b35af68h
  0.04%    0x000001dd5b35aef6: nop     word ptr [rax+rax+0h]  ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@36 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.07%    0x000001dd5b35af00: movsxd  r11,r9d
  0.05%    0x000001dd5b35af03: add     ebx,dword ptr [rdi+r9*4+10h]
  0.31%    0x000001dd5b35af08: add     ebx,dword ptr [rdi+r11*4+14h]
  0.32%    0x000001dd5b35af0d: add     ebx,dword ptr [rdi+r11*4+18h]
  0.33%    0x000001dd5b35af12: add     ebx,dword ptr [rdi+r11*4+1ch]
  0.37%    0x000001dd5b35af17: add     ebx,dword ptr [rdi+r11*4+20h]
  0.34%    0x000001dd5b35af1c: add     ebx,dword ptr [rdi+r11*4+24h]
  0.39%    0x000001dd5b35af21: add     ebx,dword ptr [rdi+r11*4+28h]
  0.36%    0x000001dd5b35af26: add     ebx,dword ptr [rdi+r11*4+2ch]
  0.34%    0x000001dd5b35af2b: add     ebx,dword ptr [rdi+r11*4+30h]
  0.35%    0x000001dd5b35af30: add     ebx,dword ptr [rdi+r11*4+34h]
  0.38%    0x000001dd5b35af35: add     ebx,dword ptr [rdi+r11*4+38h]
  0.36%    0x000001dd5b35af3a: add     ebx,dword ptr [rdi+r11*4+3ch]
  0.49%    0x000001dd5b35af3f: add     ebx,dword ptr [rdi+r11*4+40h]
  0.39%    0x000001dd5b35af44: add     ebx,dword ptr [rdi+r11*4+44h]
  0.42%    0x000001dd5b35af49: add     ebx,dword ptr [rdi+r11*4+48h]
  0.39%    0x000001dd5b35af4e: add     ebx,dword ptr [rdi+r11*4+4ch]
                                                         ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@45 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.41%    0x000001dd5b35af53: add     r9d,10h           ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@48 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001dd5b35af57: cmp     r9d,r10d
           0x000001dd5b35af5a: jl      1dd5b35af00h      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001dd5b35af5c: mov     r11,qword ptr [r15+70h]  ; ImmutableOopMap{rdi=Oop rsi=Oop rax=Oop r14=Oop xmm0=Oop [160]=Oop [168]=Oop [176]=Oop [40]=NarrowOop [48]=Oop [112]=NarrowOop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@51 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001dd5b35af60: test    dword ptr [r11],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@51 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  0.15%    0x000001dd5b35af63: cmp     r9d,r10d
           0x000001dd5b35af66: jl      1dd5b35af00h
           0x000001dd5b35af68: cmp     r9d,r8d
           0x000001dd5b35af6b: jnl     1dd5b35af7dh
           0x000001dd5b35af6d: nop                       ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@36 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.67%    0x000001dd5b35af70: add     ebx,dword ptr [rdi+r9*4+10h]
                                                         ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@45 (line 81)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.59%    0x000001dd5b35af75: inc     r9d               ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@48 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.13%    0x000001dd5b35af78: cmp     r9d,r8d
           0x000001dd5b35af7b: jl      1dd5b35af70h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@33 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001dd5b35af7d: vmovq   r10,xmm0
  0.02%    0x000001dd5b35af82: add     dword ptr [r10+10h],ebx
  2.64%    0x000001dd5b35af86: mov     r8,rax
  2.45%    0x000001dd5b35af89: mov     r9,rsi            ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.70%    0x000001dd5b35af8c: mov     r10,qword ptr [r15+70h]
  2.16%    0x000001dd5b35af90: mov     r11d,ebp
  2.37%    0x000001dd5b35af93: inc     r11d              ; ImmutableOopMap{r8=Oop r9=Oop rsi=Oop rax=Oop r14=Oop xmm0=Oop [160]=Oop [168]=Oop [176]=Oop [40]=NarrowOop [48]=Oop [112]=NarrowOop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@78 (line 104)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.44%    0x000001dd5b35af96: test    dword ptr [r10],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@78 (line 104)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  2.74%    0x000001dd5b35af99: cmp     r11d,r13d
  0.00%    0x000001dd5b35af9c: jnl     1dd5b35acf4h      ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@10 (line 105)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.14%    0x000001dd5b35afa2: shl     ebp,6h
  2.37%    0x000001dd5b35afa5: mov     r10d,ebp
  2.41%    0x000001dd5b35afa8: add     r10d,40h          ;*imul {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$2@22 (line 80)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/778368703::acceptWord@10
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 107)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.72%    0x000001dd5b35afac: mov     ebp,r11d          ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@10 (line 105)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.14%    0x000001dd5b35afaf: mov     rbx,qword ptr [r14+rbp*8+10h]
                                                         ;*laload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@13 (line 105)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.59%    0x000001dd5b35afb4: mov     r11,qword ptr [rsp+0b0h]
  2.30%    0x000001dd5b35afbc: mov     ecx,dword ptr [r11+10h]  ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$1@5 (line 75)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@57 (line 110)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.76%    0x000001dd5b35afc0: cmp     rbx,0ffffffffffffffffh
           0x000001dd5b35afc4: je      1dd5b35ae95h      ;*ifne {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@22 (line 106)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.09%    0x000001dd5b35afca: test    rbx,rbx
  0.00%    0x000001dd5b35afcd: je      1dd5b35af86h      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@42 (line 109)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.34%    0x000001dd5b35afcf: mov     r9d,dword ptr [r12+rcx*8+0ch]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$1@9 (line 75)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@57 (line 110)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001dd5b35b3f6
  2.50%    0x000001dd5b35afd4: lea     rdi,[r12+rcx*8]
  2.62%    0x000001dd5b35afd8: nop     dword ptr [rax+rax+0h]  ;*aload_2 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@45 (line 110)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.11%    0x000001dd5b35afe0: vmovq   r11,xmm0
  2.35%    0x000001dd5b35afe5: mov     r8d,dword ptr [r11+10h]  ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduceWithWordConsumer$1@3 (line 75)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@57 (line 110)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.39%    0x000001dd5b35afe9: tzcnt   r11,rbx
  2.65%    0x000001dd5b35afee: add     r11d,r10d         ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@56 (line 110)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.15%    0x000001dd5b35aff1: cmp     r11d,r9d
  0.00%    0x000001dd5b35aff4: jnb     1dd5b35b04dh
  2.29%    0x000001dd5b35aff6: add     r8d,dword ptr [rdi+r11*4+10h]
 11.03%    0x000001dd5b35affb: vmovq   r11,xmm0
  2.45%    0x000001dd5b35b000: mov     dword ptr [r11+10h],r8d  ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/513487367::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/592450429::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@11 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  3.14%    0x000001dd5b35b004: mov     r11,qword ptr [r15+70h]
  2.18%    0x000001dd5b35b008: blsi    r8,rbx
  2.23%    0x000001dd5b35b00d: xor     rbx,r8            ; ImmutableOopMap{rcx=NarrowOop rdi=Oop rsi=Oop rax=Oop r14=Oop xmm0=Oop [160]=Oop [168]=Oop [176]=Oop [40]=NarrowOop [48]=Oop [112]=NarrowOop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@72 (line 111)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.44%    0x000001dd5b35b010: test    dword ptr [r11],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@72 (line 111)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  2.71%    0x000001dd5b35b013: test    rbx,rbx
           0x000001dd5b35b016: jne     1dd5b35afe0h      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@42 (line 109)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduceWithWordConsumer@23 (line 75)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduceWithWordConsumer_jmhTest::reduceWithWordConsumer_thrpt_jmhStub@17 (line 142)
  2.23%    0x000001dd5b35b018: jmp     1dd5b35af86h
           0x000001dd5b35b01d: xor     ebx,ebx
           0x000001dd5b35b01f: jmp     1dd5b35af7dh
           0x000001dd5b35b024: mov     qword ptr [rsp+0b0h],rcx
           0x000001dd5b35b02c: mov     qword ptr [rsp+0a0h],r9
           0x000001dd5b35b034: mov     qword ptr [rsp+20h],r8
           0x000001dd5b35b039: mov     dword ptr [rsp+28h],r11d
           0x000001dd5b35b03e: mov     qword ptr [rsp+38h],rdx
....................................................................................................
 98.92%  <total for region 1>

Heroically ploughing through the full words tells a different story: blsi is up at 11%. This indicates more time is spent iterating rather than evaluating the callback.


....[Hottest Region 1]..............................................................................
c2, com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub, version 164 (218 bytes) 

                                                         ; - java.lang.invoke.LambdaForm$MH/1035918105::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@11 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
           0x0000019f106c66e7: mov     qword ptr [r15+3d8h],rdi
                                                         ;*invokespecial &lt;init&gt; {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/1035918105::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@11 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  0.00%    0x0000019f106c66ee: mov     r10d,dword ptr [rsp+60h]
  0.00%    0x0000019f106c66f3: mov     edx,dword ptr [r12+r10*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@4 (line 89)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x0000019f106c6aba
  0.02%    0x0000019f106c66f8: test    edx,edx
           0x0000019f106c66fa: jbe     19f106c6826h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@5 (line 89)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
           0x0000019f106c6700: mov     r10d,edx
  0.00%    0x0000019f106c6703: dec     r10d
           0x0000019f106c6706: cmp     r10d,edx
           0x0000019f106c6709: jnb     19f106c6942h
  0.00%    0x0000019f106c670f: mov     r10d,dword ptr [rsp+60h]
           0x0000019f106c6714: lea     rax,[r12+r10*8]   ;*getfield bitmap {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@6 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  0.00%    0x0000019f106c6718: xor     ecx,ecx
           0x0000019f106c671a: jmp     19f106c674dh
  1.54%    0x0000019f106c671c: mov     r11,r13
  1.57%    0x0000019f106c671f: mov     r13,rbp
  1.69%    0x0000019f106c6722: mov     r8,r14
  1.46%    0x0000019f106c6725: vmovd   r9d,xmm1
  1.48%    0x0000019f106c672a: vmovq   rbx,xmm0
  1.61%    0x0000019f106c672f: vmovq   r14,xmm2
  1.63%    0x0000019f106c6734: vmovq   rbp,xmm3
  1.46%    0x0000019f106c6739: mov     rdi,rbp           ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/1035918105::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@11 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.49%    0x0000019f106c673c: mov     r10,qword ptr [r15+70h]
  1.54%    0x0000019f106c6740: inc     ecx               ; ImmutableOopMap{r11=Oop r9=NarrowOop rdi=Oop rax=Oop rbp=Oop r13=Oop r14=Oop [152]=Oop [48]=Oop [96]=NarrowOop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@52 (line 89)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.73%    0x0000019f106c6742: test    dword ptr [r10],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@52 (line 89)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  1.43%    0x0000019f106c6745: cmp     ecx,edx
  0.00%    0x0000019f106c6747: jnl     19f106c6480h      ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@8 (line 90)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.57%    0x0000019f106c674d: mov     rdi,qword ptr [rax+rcx*8+10h]
                                                         ;*laload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@10 (line 90)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.61%    0x0000019f106c6752: test    rdi,rdi
           0x0000019f106c6755: je      19f106c6739h      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@17 (line 91)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.68%    0x0000019f106c6757: vmovq   xmm3,rbp
  1.47%    0x0000019f106c675c: mov     esi,dword ptr [r11+10h]  ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduce$0@5 (line 67)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@31 (line 92)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.48%    0x0000019f106c6760: mov     r10d,dword ptr [r12+rsi*8+0ch]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduce$0@9 (line 67)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@31 (line 92)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x0000019f106c6a3e
  1.58%    0x0000019f106c6765: vmovq   xmm2,r14
  1.68%    0x0000019f106c676a: vmovq   xmm0,rbx
  1.46%    0x0000019f106c676f: vmovd   xmm1,r9d
  1.46%    0x0000019f106c6774: mov     r14,r8
  1.51%    0x0000019f106c6777: mov     rbp,r13
  1.68%    0x0000019f106c677a: mov     r13,r11
  1.41%    0x0000019f106c677d: lea     r8,[r12+rsi*8]
  1.40%    0x0000019f106c6781: mov     ebx,ecx
  1.55%    0x0000019f106c6783: shl     ebx,6h
  1.63%    0x0000019f106c6786: nop     word ptr [rax+rax+0h]  ;*aload_2 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@20 (line 92)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.68%    0x0000019f106c6790: vmovq   r11,xmm2
  1.70%    0x0000019f106c6795: mov     r11d,dword ptr [r11+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$reduce$0@3 (line 67)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::accept@9
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@31 (line 92)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  6.98%    0x0000019f106c6799: tzcnt   r9,rdi
  3.47%    0x0000019f106c679e: add     r9d,ebx           ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 92)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.65%    0x0000019f106c67a1: cmp     r9d,r10d
           0x0000019f106c67a4: jnb     19f106c67cdh
  1.67%    0x0000019f106c67a6: add     r11d,dword ptr [r8+r9*4+10h]
 11.45%    0x0000019f106c67ab: vmovq   r9,xmm2
  3.20%    0x0000019f106c67b0: mov     dword ptr [r9+10h],r11d  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@46 (line 93)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
 11.31%    0x0000019f106c67b4: blsi    r11,rdi
  1.71%    0x0000019f106c67b9: xor     rdi,r11           ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1830914914::get$Lambda@6
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
                                                         ; - java.lang.invoke.LambdaForm$MH/1035918105::linkToTargetMethod@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@11 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.72%    0x0000019f106c67bc: mov     r11,qword ptr [r15+70h]  ; ImmutableOopMap{r8=Oop r9=Oop rsi=NarrowOop rax=Oop rbp=Oop r13=Oop xmm1=NarrowOop xmm2=Oop xmm3=Oop [152]=Oop [48]=Oop [96]=NarrowOop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@46 (line 93)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.80%    0x0000019f106c67c0: test    dword ptr [r11],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@46 (line 93)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  6.99%    0x0000019f106c67c3: test    rdi,rdi
           0x0000019f106c67c6: jne     19f106c6790h      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@17 (line 91)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::reduce@16 (line 67)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_reduce_jmhTest::reduce_thrpt_jmhStub@17 (line 142)
  1.39%    0x0000019f106c67c8: jmp     19f106c671ch
           0x0000019f106c67cd: mov     edx,0ffffffe4h
           0x0000019f106c67d2: mov     r10,qword ptr [rsp+98h]
           0x0000019f106c67da: mov     qword ptr [rsp+90h],r10
           0x0000019f106c67e2: mov     r10,qword ptr [rsp+30h]
           0x0000019f106c67e7: mov     qword ptr [rsp+0a0h],r13
           0x0000019f106c67ef: vmovsd  qword ptr [rsp+20h],xmm0
           0x0000019f106c67f5: mov     qword ptr [rsp+28h],r10
....................................................................................................
 98.59%  <total for region 1>

This shows the cost of a data dependency in a loop. The operation we want to perform is associative, so we could even vectorise this. In C++ that might happen automatically, or could be ensured with intrinsics, but C2 has various heuristics: it won’t try to vectorise a simple reduction, and 64 would probably be on the short side for most cases it would try to vectorise.

Acknowledging Runs

You might be tempted to transfer even more control to the callback, by accumulating runs and then calling the callback once per run. It simplifies the code to exclude incomplete start and end words from the run.


private interface RunConsumer {
    void acceptRun(int start, int end);
  }

  public void forEach(long[] bitmap, IntConsumer intConsumer, RunConsumer runConsumer) {
    int runStart = -1;
    for (int i = 0; i < bitmap.length; ++i) {
      long word = bitmap[i];
      if (word == -1L) {
        if (runStart == -1) {
          runStart = i;
        }
      } else {
        if (runStart != -1) {
          runConsumer.acceptRun(runStart * Long.SIZE, i * Long.SIZE);
          runStart = -1;
        }
        while (word != 0) {
          intConsumer.accept(Long.SIZE * i + Long.numberOfTrailingZeros(word));
          word ^= Long.lowestOneBit(word);
        }
      }
    }
    if (runStart != -1) {
      runConsumer.acceptRun(runStart * Long.SIZE, bitmap.length * Long.SIZE);
    }
  }

For a simple reduction, the extra complexity isn’t justified: you’re better off with the WordIterator.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: scenario
reduce thrpt 1 10 160.502749 2.960568 ops/ms SPARSE_16_FULL_WORDS
reduce thrpt 1 10 7.294747 0.186678 ops/ms FULL
reduce thrpt 1 10 258.064511 8.902233 ops/ms ONE_BIT_PER_WORD
reduce thrpt 1 10 159.613877 3.424432 ops/ms SPARSE_1_16_WORD_RUN
reduceWithRunConsumer thrpt 1 10 251.683131 6.799639 ops/ms SPARSE_16_FULL_WORDS
reduceWithRunConsumer thrpt 1 10 37.809154 0.723198 ops/ms FULL
reduceWithRunConsumer thrpt 1 10 218.133560 13.756779 ops/ms ONE_BIT_PER_WORD
reduceWithRunConsumer thrpt 1 10 140.896826 8.495777 ops/ms SPARSE_1_16_WORD_RUN
reduceWithWordConsumer thrpt 1 10 257.961783 5.892072 ops/ms SPARSE_16_FULL_WORDS
reduceWithWordConsumer thrpt 1 10 43.909471 0.601319 ops/ms FULL
reduceWithWordConsumer thrpt 1 10 213.731758 20.398077 ops/ms ONE_BIT_PER_WORD
reduceWithWordConsumer thrpt 1 10 258.280428 11.316647 ops/ms SPARSE_1_16_WORD_RUN

It’s simplistic to measure this and conclude that this is a bad approach though. There are several other dimensions to this problem:

  1. Vectorised callbacks
  2. Inlining failures preventing optimisations
  3. The number of runs and their lengths (i.e. your data and how you structure it)

Vectorisable Callbacks

There are real benefits to batching up callbacks if the workload in the callback can be vectorised. The code doesn’t need to get much more complicated to start benefitting from larger iteration batches. Mapping each bit to a scaled and squared value from the data array and storing it into an output array illustrates this.


  @Benchmark
  public void map(Blackhole bh) {
    forEach(bitmap, i -> output[i] = data[i] * data[i] * factor);
    bh.consume(output);
  }

  @Benchmark
  public void mapWithWordConsumer(Blackhole bh) {
    forEach(bitmap, i -> output[0] = data[i] * factor, (WordConsumer) (index, word) -> {
      if (word != -1L) {
        throw new IllegalStateException();
      }
      for (int i = index * Long.SIZE; i < (index + 1) * Long.SIZE; ++i) {
        output[i] = data[i] * data[i] * factor;
      }
    });
    bh.consume(output);
  }

  @Benchmark
  public void mapWithRunConsumer(Blackhole bh) {
    forEach(bitmap, i -> output[0] = data[i] * factor, (RunConsumer) (start, end) -> {
      for (int i = start; i < end; ++i) {
        output[i] = data[i] * data[i] * factor;
      }
    });
    bh.consume(output);
  }

The RunConsumer does much better in the full case, never much worse than the WordConsumer and always better than the basic strategy – even when there is only one run in the entire bitset, or when there are a few full words in an otherwise sparse bitset.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: scenario
map thrpt 1 10 127.876662 3.411741 ops/ms SPARSE_16_FULL_WORDS
map thrpt 1 10 10.598974 0.022404 ops/ms FULL
map thrpt 1 10 126.434666 18.608547 ops/ms ONE_BIT_PER_WORD
map thrpt 1 10 115.977840 20.449258 ops/ms SPARSE_1_16_WORD_RUN
mapWithRunConsumer thrpt 1 10 199.186167 8.138446 ops/ms SPARSE_16_FULL_WORDS
mapWithRunConsumer thrpt 1 10 64.230868 2.871434 ops/ms FULL
mapWithRunConsumer thrpt 1 10 219.963063 4.257561 ops/ms ONE_BIT_PER_WORD
mapWithRunConsumer thrpt 1 10 203.403804 6.907366 ops/ms SPARSE_1_16_WORD_RUN
mapWithWordConsumer thrpt 1 10 229.822235 5.276084 ops/ms SPARSE_16_FULL_WORDS
mapWithWordConsumer thrpt 1 10 48.381990 3.845642 ops/ms FULL
mapWithWordConsumer thrpt 1 10 218.907803 5.331011 ops/ms ONE_BIT_PER_WORD
mapWithWordConsumer thrpt 1 10 240.795280 10.204818 ops/ms SPARSE_1_16_WORD_RUN

This is simply because the callback was vectorised, and the style of the RunConsumer API allows this to be exploited. This can be seen with perfasm. Both the WordConsumer and RunConsumer are actually vectorised, but the thing to notice is that there are two hot regions in the WordConsumer benchmark: the iteration and the callback, this boundary is often crossed. On the other hand, the RunConsumer implementation spends most of its time in the callback.

WordConsumer


....[Hottest Region 1]..............................................................................
c2, com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub, version 172 (227 bytes) 

           0x000001c2aa13c788: ud2                       ;*athrow {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@21
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@47 (line 226)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001c2aa13c78a: mov     r8d,ecx
  0.23%    0x000001c2aa13c78d: mov     ecx,r8d           ;*aload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@62 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.55%    0x000001c2aa13c790: vmovdqu ymm1,ymmword ptr [r9+r10*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@69 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.15%    0x000001c2aa13c797: vpmulld ymm1,ymm1,ymm1
  3.72%    0x000001c2aa13c79c: vpmulld ymm1,ymm1,ymm2
 16.02%    0x000001c2aa13c7a1: vmovdqu ymmword ptr [rdx+r10*4+10h],ymm1
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@78 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.69%    0x000001c2aa13c7a8: movsxd  r8,r10d
  1.55%    0x000001c2aa13c7ab: vmovdqu ymm1,ymmword ptr [r9+r8*4+30h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@69 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.46%    0x000001c2aa13c7b2: vpmulld ymm1,ymm1,ymm1
  1.71%    0x000001c2aa13c7b7: vpmulld ymm1,ymm1,ymm2
  3.20%    0x000001c2aa13c7bc: vmovdqu ymmword ptr [rdx+r8*4+30h],ymm1
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@78 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.07%    0x000001c2aa13c7c3: add     r10d,10h          ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@79 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.70%    0x000001c2aa13c7c7: cmp     r10d,r11d
           0x000001c2aa13c7ca: jl      1c2aa13c790h      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1453625478::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1453625478::get$Lambda@5
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@10
                                                         ; - java.lang.invoke.LambdaForm$MH/1166726978::linkToTargetMethod@5
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@6 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.02%    0x000001c2aa13c7cc: mov     r8,qword ptr [r15+70h]  ; ImmutableOopMap{r9=Oop rdx=Oop rax=Oop r13=NarrowOop xmm0=Oop [144]=Oop [152]=Oop [48]=Oop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@82 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.50%    0x000001c2aa13c7d0: test    dword ptr [r8],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@82 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  0.04%    0x000001c2aa13c7d3: cmp     r10d,r11d
           0x000001c2aa13c7d6: jl      1c2aa13c78ah
  0.05%    0x000001c2aa13c7d8: mov     r11d,dword ptr [rsp+5ch]
  0.02%    0x000001c2aa13c7dd: add     r11d,39h
  1.57%    0x000001c2aa13c7e1: mov     r8d,ecx
  0.02%    0x000001c2aa13c7e4: cmp     r8d,r11d
  0.06%    0x000001c2aa13c7e7: mov     ecx,80000000h
  0.02%    0x000001c2aa13c7ec: cmovl   r11d,ecx
  1.50%    0x000001c2aa13c7f0: cmp     r10d,r11d
           0x000001c2aa13c7f3: jnl     1c2aa13c819h
  0.02%    0x000001c2aa13c7f5: nop                       ;*aload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@62 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.06%    0x000001c2aa13c7f8: vmovdqu ymm1,ymmword ptr [r9+r10*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@69 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.21%    0x000001c2aa13c7ff: vpmulld ymm1,ymm1,ymm1
  2.16%    0x000001c2aa13c804: vpmulld ymm1,ymm1,ymm2
  1.80%    0x000001c2aa13c809: vmovdqu ymmword ptr [rdx+r10*4+10h],ymm1
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@78 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.56%    0x000001c2aa13c810: add     r10d,8h           ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@79 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c814: cmp     r10d,r11d
           0x000001c2aa13c817: jl      1c2aa13c7f8h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@59 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c819: cmp     r10d,r8d
           0x000001c2aa13c81c: jnl     1c2aa13c83ah
  0.01%    0x000001c2aa13c81e: nop                       ;*aload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@62 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.58%    0x000001c2aa13c820: mov     ecx,dword ptr [r9+r10*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@69 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.03%    0x000001c2aa13c825: imul    ecx,ecx
  1.58%    0x000001c2aa13c828: imul    ecx,dword ptr [rsp+58h]
  0.03%    0x000001c2aa13c82d: mov     dword ptr [rdx+r10*4+10h],ecx
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@78 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.56%    0x000001c2aa13c832: inc     r10d              ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@79 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c835: cmp     r10d,r8d
           0x000001c2aa13c838: jl      1c2aa13c820h      ;*putfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1453625478::&lt;init&gt;@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$43/1453625478::get$Lambda@5
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@10
                                                         ; - java.lang.invoke.LambdaForm$MH/1166726978::linkToTargetMethod@5
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@6 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.55%    0x000001c2aa13c83a: mov     r10,qword ptr [r15+70h]
  0.01%    0x000001c2aa13c83e: inc     esi               ; ImmutableOopMap{rax=Oop r13=NarrowOop xmm0=Oop [144]=Oop [152]=Oop [48]=Oop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@78 (line 168)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.02%    0x000001c2aa13c840: test    dword ptr [r10],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@78 (line 168)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ;   {poll}
  0.02%    0x000001c2aa13c843: cmp     esi,dword ptr [rsp+0a0h]
           0x000001c2aa13c84a: jnl     1c2aa13cb2ah      ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@10 (line 169)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.54%    0x000001c2aa13c850: mov     r8,rax
  0.01%    0x000001c2aa13c853: mov     r14,qword ptr [rsp+90h]
  0.01%    0x000001c2aa13c85b: mov     rbx,qword ptr [rsp+50h]
  0.01%    0x000001c2aa13c860: mov     r10d,r13d
  1.54%    0x000001c2aa13c863: vmovq   rax,xmm0
  0.01%    0x000001c2aa13c868: mov     r9d,dword ptr [rsp+0a0h]
  0.02%    0x000001c2aa13c870: jmp     1c2aa13c904h
           0x000001c2aa13c875: mov     r13,qword ptr [rsp+98h]
           0x000001c2aa13c87d: mov     r8,qword ptr [rsp+68h]  ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@7 (line 168)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001c2aa13c882: mov     ebp,dword ptr [r8+18h]  ;*getfield output {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@22 (line 141)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001c2aa13c886: mov     r10,qword ptr [rsp+30h]
           0x000001c2aa13c88b: test    r10,r10
....................................................................................................
 53.26%  <total for region 1>

....[Hottest Region 2]..............................................................................
c2, com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub, version 172 (391 bytes) 

                                                         ; implicit exception: dispatches to 0x000001c2aa13cdd6
           0x000001c2aa13c8c8: test    r11d,r11d
           0x000001c2aa13c8cb: jne     1c2aa13cc5eh      ;*ifeq {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@30 (line 144)
           0x000001c2aa13c8d1: mov     ebx,1h            ;*invokespecial &lt;init&gt; {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::get$Lambda@5
                                                         ; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@10
                                                         ; - java.lang.invoke.LambdaForm$MH/1166726978::linkToTargetMethod@5
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@12 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c8d6: mov     r8,qword ptr [rsp+0a0h]
  0.01%    0x000001c2aa13c8de: mov     r10d,dword ptr [r8+10h]  ;*getfield bitmap {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@2 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001c2aa13c8e2: mov     r9d,dword ptr [r12+r10*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@6 (line 168)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001c2aa13cdc6
  0.04%    0x000001c2aa13c8e7: test    r9d,r9d
           0x000001c2aa13c8ea: jbe     1c2aa13cb2fh      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@7 (line 168)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001c2aa13c8f0: mov     ecx,r9d
           0x000001c2aa13c8f3: dec     ecx
           0x000001c2aa13c8f5: cmp     ecx,r9d
           0x000001c2aa13c8f8: jnb     1c2aa13cd1ah
           0x000001c2aa13c8fe: lea     rax,[r12+r10*8]   ;*getfield bitmap {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@2 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001c2aa13c902: xor     esi,esi           ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@10 (line 169)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c904: mov     r11,qword ptr [rax+rsi*8+10h]
                                                         ;*laload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@13 (line 169)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.57%    0x000001c2aa13c909: cmp     r11,0ffffffffffffffffh
           0x000001c2aa13c90d: jne     1c2aa13cccah      ;*ifne {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@22 (line 170)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c913: mov     edx,dword ptr [r8+14h]  ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@3 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c917: mov     edi,dword ptr [r12+rdx*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@35 (line 225)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001c2aa13cda6
  0.01%    0x000001c2aa13c91c: mov     dword ptr [rsp+0a0h],r9d
  1.56%    0x000001c2aa13c924: mov     r13d,dword ptr [r8+0ch]  ;*getfield factor {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@11 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c928: mov     r11d,esi
  0.02%    0x000001c2aa13c92b: shl     r11d,6h           ;*imul {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@19 (line 223)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c92f: lea     r9,[r12+rdx*8]    ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@3 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  1.55%    0x000001c2aa13c933: mov     ecx,r11d
  0.01%    0x000001c2aa13c936: add     ecx,40h           ;*imul {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@27 (line 224)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c939: mov     ebp,dword ptr [r8+18h]  ;*getfield output {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@7 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c93d: test    r11d,r11d
  1.55%    0x000001c2aa13c940: jl      1c2aa13ca62h      ;*iflt {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@1
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@36 (line 225)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c946: cmp     r11d,ecx
           0x000001c2aa13c949: jnle    1c2aa13ca62h      ;*if_icmpgt {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@6
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@36 (line 225)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c94f: cmp     ecx,edi
           0x000001c2aa13c951: jnle    1c2aa13ca62h      ;*if_icmple {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@11
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@36 (line 225)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.00%    0x000001c2aa13c957: mov     dword ptr [rsp+60h],ecx
  1.51%    0x000001c2aa13c95b: mov     dword ptr [rsp+5ch],r11d
  0.02%    0x000001c2aa13c960: mov     dword ptr [rsp+58h],r13d
  0.01%    0x000001c2aa13c965: mov     ecx,edi
  0.01%    0x000001c2aa13c967: vmovd   xmm1,edx
  1.48%    0x000001c2aa13c96b: vmovq   xmm0,rax
  0.02%    0x000001c2aa13c970: mov     r13d,r10d
  0.02%    0x000001c2aa13c973: mov     qword ptr [rsp+50h],rbx
  0.01%    0x000001c2aa13c978: mov     qword ptr [rsp+90h],r14
  1.52%    0x000001c2aa13c980: mov     rax,r8
  0.01%    0x000001c2aa13c983: mov     edi,dword ptr [r12+rbp*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@46 (line 226)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
                                                         ; implicit exception: dispatches to 0x000001c2aa13cdb6
  0.01%    0x000001c2aa13c988: lea     rdx,[r12+rbp*8]   ;*getfield output {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@7 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.02%    0x000001c2aa13c98c: mov     r10d,dword ptr [rsp+60h]
  1.50%    0x000001c2aa13c991: cmp     r10d,edi
           0x000001c2aa13c994: jnle    1c2aa13cac2h      ;*if_icmple {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@11
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@47 (line 226)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13c99a: cmp     r11d,r10d
           0x000001c2aa13c99d: jnl     1c2aa13c83ah      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@59 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.02%    0x000001c2aa13c9a3: cmp     r11d,ecx
           0x000001c2aa13c9a6: jnb     1c2aa13cc9ah
  0.01%    0x000001c2aa13c9ac: movsxd  r10,dword ptr [rsp+60h]
  1.58%    0x000001c2aa13c9b1: movsxd  r11,ecx
  0.01%    0x000001c2aa13c9b4: dec     r10
  0.02%    0x000001c2aa13c9b7: cmp     r10,r11
           0x000001c2aa13c9ba: jnb     1c2aa13cc9ah
  0.01%    0x000001c2aa13c9c0: mov     r11d,dword ptr [rsp+5ch]
  1.50%    0x000001c2aa13c9c5: cmp     r11d,edi
           0x000001c2aa13c9c8: jnb     1c2aa13cc9ah
  0.02%    0x000001c2aa13c9ce: movsxd  r11,edi
  0.02%    0x000001c2aa13c9d1: cmp     r10,r11
           0x000001c2aa13c9d4: jnb     1c2aa13cc9ah
  0.02%    0x000001c2aa13c9da: mov     r11d,edx
  1.62%    0x000001c2aa13c9dd: shr     r11d,2h
  0.02%    0x000001c2aa13c9e1: and     r11d,7h
  0.01%    0x000001c2aa13c9e5: add     r11d,dword ptr [rsp+5ch]
  0.02%    0x000001c2aa13c9ea: mov     r8d,3h
  1.55%    0x000001c2aa13c9f0: sub     r8d,r11d
  0.01%    0x000001c2aa13c9f3: and     r8d,7h
  0.01%    0x000001c2aa13c9f7: add     r8d,dword ptr [rsp+5ch]
  0.01%    0x000001c2aa13c9fc: inc     r8d
  1.58%    0x000001c2aa13c9ff: cmp     r8d,dword ptr [rsp+60h]
  0.02%    0x000001c2aa13ca04: mov     ecx,dword ptr [rsp+60h]
  0.02%    0x000001c2aa13ca08: cmovnle r8d,ecx
  0.01%    0x000001c2aa13ca0c: mov     r10d,dword ptr [rsp+5ch]
  1.54%    0x000001c2aa13ca11: mov     ebx,ecx           ;*aload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@62 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.11%    0x000001c2aa13ca13: mov     r11d,dword ptr [r9+r10*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@69 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  4.67%    0x000001c2aa13ca18: imul    r11d,r11d
  0.14%    0x000001c2aa13ca1c: imul    r11d,dword ptr [rsp+58h]
  4.67%    0x000001c2aa13ca22: mov     dword ptr [rdx+r10*4+10h],r11d
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@78 (line 228)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.16%    0x000001c2aa13ca27: inc     r10d              ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@79 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  4.63%    0x000001c2aa13ca2a: cmp     r10d,r8d
           0x000001c2aa13ca2d: jl      1c2aa13ca13h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@59 (line 227)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
  0.01%    0x000001c2aa13ca2f: mov     r11d,dword ptr [rsp+5ch]
  0.01%    0x000001c2aa13ca34: add     r11d,31h
  0.02%    0x000001c2aa13ca38: mov     r8d,ebx
  1.48%    0x000001c2aa13ca3b: cmp     r8d,r11d
  0.01%    0x000001c2aa13ca3e: mov     ebx,80000000h
  0.02%    0x000001c2aa13ca43: cmovl   r11d,ebx
  0.04%    0x000001c2aa13ca47: cmp     r10d,r11d
           0x000001c2aa13ca4a: jnl     1c2aa13c819h
  2.48%    0x000001c2aa13ca50: vpshufd xmm2,xmmword ptr [rsp+58h],0h
  2.82%    0x000001c2aa13ca57: vinserti128 ymm2,ymm2,xmm2,1h
  2.88%    0x000001c2aa13ca5d: jmp     1c2aa13c78dh      ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@14
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareForWord@36 (line 225)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithWordConsumer$7@14 (line 140)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1872198645::acceptWord@6
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::forEach@30 (line 171)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::mapWithWordConsumer@17 (line 138)
                                                         ; - com.openkappa.simd.iterate.generated.BitSetIterator_mapWithWordConsumer_jmhTest::mapWithWordConsumer_thrpt_jmhStub@17 (line 142)
           0x000001c2aa13ca62: mov     qword ptr [rsp+90h],r14
           0x000001c2aa13ca6a: mov     qword ptr [rsp+28h],r8
....................................................................................................
 46.39%  <total for region 2>

RunConsumer


....[Hottest Region 1]..............................................................................
c2, com.openkappa.simd.iterate.BitSetIterator$$Lambda$44.1209658195::acceptRun, version 166 (816 bytes) 

           # parm1:    r9        = int
           #           [sp+0x50]  (sp of caller)
           0x0000016658954740: mov     r10d,dword ptr [rdx+8h]
           0x0000016658954744: shl     r10,3h
           0x0000016658954748: cmp     rax,r10
           0x000001665895474b: jne     166588c6d80h      ;   {runtime_call ic_miss_stub}
           0x0000016658954751: nop
           0x0000016658954754: nop     dword ptr [rax+rax+0h]
           0x000001665895475c: nop
         [Verified Entry Point]
  0.00%    0x0000016658954760: mov     dword ptr [rsp+0ffffffffffff9000h],eax
  0.00%    0x0000016658954767: push    rbp
           0x0000016658954768: sub     rsp,40h           ;*synchronization entry
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@-1
           0x000001665895476c: mov     ecx,dword ptr [rdx+0ch]  ;*getfield arg$1 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@1
  0.00%    0x000001665895476f: mov     ebx,dword ptr [r12+rcx*8+14h]
                                                         ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@3 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
                                                         ; implicit exception: dispatches to 0x0000016658954b76
           0x0000016658954774: mov     eax,dword ptr [r12+rbx*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@3 (line 211)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
                                                         ; implicit exception: dispatches to 0x0000016658954b8e
  0.06%    0x0000016658954779: mov     r11d,dword ptr [r12+rcx*8+0ch]
                                                         ;*getfield factor {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@11 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x000001665895477e: mov     edi,dword ptr [r12+rcx*8+18h]
                                                         ;*getfield output {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@7 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954783: test    r8d,r8d
           0x0000016658954786: jl      16658954ab6h      ;*iflt {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@1
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@4 (line 211)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x000001665895478c: cmp     r8d,r9d
           0x000001665895478f: jnle    16658954ae6h      ;*if_icmpgt {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@6
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@4 (line 211)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954795: cmp     r9d,eax
           0x0000016658954798: jnle    16658954b1ah      ;*if_icmple {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@11
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@4 (line 211)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x000001665895479e: mov     ebp,dword ptr [r12+rdi*8+0ch]
                                                         ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@11 (line 212)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
                                                         ; implicit exception: dispatches to 0x0000016658954b9e
  0.05%    0x00000166589547a3: cmp     r9d,ebp
           0x00000166589547a6: jnle    16658954b4ah      ;*if_icmple {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - jdk.internal.util.Preconditions::checkFromToIndex@11
                                                         ; - java.util.Objects::checkFromToIndex@4
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@12 (line 212)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x00000166589547ac: cmp     r8d,r9d
           0x00000166589547af: jnl     16658954a81h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@22 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x00000166589547b5: cmp     r8d,eax
           0x00000166589547b8: jnb     16658954a91h
  0.00%    0x00000166589547be: movsxd  r10,r9d
           0x00000166589547c1: movsxd  rcx,eax
           0x00000166589547c4: dec     r10
           0x00000166589547c7: cmp     r10,rcx
           0x00000166589547ca: jnb     16658954a91h
           0x00000166589547d0: cmp     r8d,ebp
           0x00000166589547d3: jnb     16658954a91h
           0x00000166589547d9: movsxd  rcx,ebp
  0.01%    0x00000166589547dc: cmp     r10,rcx
           0x00000166589547df: jnb     16658954a91h
  0.00%    0x00000166589547e5: shl     rdi,3h            ;*getfield output {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@7 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x00000166589547e9: lea     rdx,[r12+rbx*8]   ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@3 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x00000166589547ed: mov     r10d,edi
  0.00%    0x00000166589547f0: shr     r10d,2h
  0.00%    0x00000166589547f4: and     r10d,7h
           0x00000166589547f8: add     r10d,r8d
           0x00000166589547fb: mov     ecx,3h
  0.00%    0x0000016658954800: sub     ecx,r10d
  0.00%    0x0000016658954803: and     ecx,7h
           0x0000016658954806: add     ecx,r8d
           0x0000016658954809: inc     ecx
  0.00%    0x000001665895480b: cmp     ecx,r9d
  0.00%    0x000001665895480e: cmovnle ecx,r9d           ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@25 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954812: mov     r10d,dword ptr [rdx+r8*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954817: imul    r10d,r10d
  0.01%    0x000001665895481b: imul    r10d,r11d
  0.00%    0x000001665895481f: mov     dword ptr [rdi+r8*4+10h],r10d
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954824: inc     r8d               ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@41 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954827: cmp     r8d,ecx
           0x000001665895482a: jl      16658954812h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@22 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x000001665895482c: mov     r10d,r9d
           0x000001665895482f: add     r10d,0ffffff81h
  0.00%    0x0000016658954833: mov     ebx,80000000h
  0.00%    0x0000016658954838: cmp     r9d,r10d
           0x000001665895483b: cmovl   r10d,ebx
           0x000001665895483f: cmp     r8d,r10d
           0x0000016658954842: jnl     16658954a61h
  0.00%    0x0000016658954848: vmovd   xmm1,r11d
           0x000001665895484d: vpshufd xmm1,xmm1,0h
           0x0000016658954852: vinserti128 ymm1,ymm1,xmm1,1h  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@44 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954858: nop     dword ptr [rax+rax+0h]  ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@25 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.92%    0x0000016658954860: vmovdqu ymm0,ymmword ptr [rdx+r8*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.31%    0x0000016658954867: vpmulld ymm0,ymm0,ymm0
  1.74%    0x000001665895486c: vpmulld ymm0,ymm0,ymm1
  4.55%    0x0000016658954871: vmovdqu ymmword ptr [rdi+r8*4+10h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.69%    0x0000016658954878: movsxd  rcx,r8d
  0.01%    0x000001665895487b: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+30h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.41%    0x0000016658954881: vpmulld ymm0,ymm0,ymm0
  0.78%    0x0000016658954886: vpmulld ymm0,ymm0,ymm1
  0.83%    0x000001665895488b: vmovdqu ymmword ptr [rdi+rcx*4+30h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.25%    0x0000016658954891: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+50h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.29%    0x0000016658954897: vpmulld ymm0,ymm0,ymm0
  1.51%    0x000001665895489c: vpmulld ymm0,ymm0,ymm1
  3.65%    0x00000166589548a1: vmovdqu ymmword ptr [rdi+rcx*4+50h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.54%    0x00000166589548a7: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+70h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.31%    0x00000166589548ad: vpmulld ymm0,ymm0,ymm0
  0.47%    0x00000166589548b2: vpmulld ymm0,ymm0,ymm1
  1.11%    0x00000166589548b7: vmovdqu ymmword ptr [rdi+rcx*4+70h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.28%    0x00000166589548bd: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+90h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.17%    0x00000166589548c6: vpmulld ymm0,ymm0,ymm0
  1.89%    0x00000166589548cb: vpmulld ymm0,ymm0,ymm1
  3.56%    0x00000166589548d0: vmovdqu ymmword ptr [rdi+rcx*4+90h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.73%    0x00000166589548d9: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+0b0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.21%    0x00000166589548e2: vpmulld ymm0,ymm0,ymm0
  0.34%    0x00000166589548e7: vpmulld ymm0,ymm0,ymm1
  1.29%    0x00000166589548ec: vmovdqu ymmword ptr [rdi+rcx*4+0b0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.33%    0x00000166589548f5: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+0d0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.97%    0x00000166589548fe: vpmulld ymm0,ymm0,ymm0
  1.90%    0x0000016658954903: vpmulld ymm0,ymm0,ymm1
  3.59%    0x0000016658954908: vmovdqu ymmword ptr [rdi+rcx*4+0d0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.82%    0x0000016658954911: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+0f0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.18%    0x000001665895491a: vpmulld ymm0,ymm0,ymm0
  0.29%    0x000001665895491f: vpmulld ymm0,ymm0,ymm1
  1.25%    0x0000016658954924: vmovdqu ymmword ptr [rdi+rcx*4+0f0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.33%    0x000001665895492d: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+110h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.10%    0x0000016658954936: vpmulld ymm0,ymm0,ymm0
  2.11%    0x000001665895493b: vpmulld ymm0,ymm0,ymm1
  3.67%    0x0000016658954940: vmovdqu ymmword ptr [rdi+rcx*4+110h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.93%    0x0000016658954949: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+130h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.13%    0x0000016658954952: vpmulld ymm0,ymm0,ymm0
  0.25%    0x0000016658954957: vpmulld ymm0,ymm0,ymm1
  1.35%    0x000001665895495c: vmovdqu ymmword ptr [rdi+rcx*4+130h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.32%    0x0000016658954965: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+150h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.93%    0x000001665895496e: vpmulld ymm0,ymm0,ymm0
  2.16%    0x0000016658954973: vpmulld ymm0,ymm0,ymm1
  3.73%    0x0000016658954978: vmovdqu ymmword ptr [rdi+rcx*4+150h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.95%    0x0000016658954981: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+170h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.14%    0x000001665895498a: vpmulld ymm0,ymm0,ymm0
  0.21%    0x000001665895498f: vpmulld ymm0,ymm0,ymm1
  1.39%    0x0000016658954994: vmovdqu ymmword ptr [rdi+rcx*4+170h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.29%    0x000001665895499d: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+190h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.42%    0x00000166589549a6: vpmulld ymm0,ymm0,ymm0
  2.61%    0x00000166589549ab: vpmulld ymm0,ymm0,ymm1
  4.42%    0x00000166589549b0: vmovdqu ymmword ptr [rdi+rcx*4+190h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.01%    0x00000166589549b9: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+1b0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.10%    0x00000166589549c2: vpmulld ymm0,ymm0,ymm0
  0.17%    0x00000166589549c7: vpmulld ymm0,ymm0,ymm1
  1.46%    0x00000166589549cc: vmovdqu ymmword ptr [rdi+rcx*4+1b0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.27%    0x00000166589549d5: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+1d0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
 13.60%    0x00000166589549de: vpmulld ymm0,ymm0,ymm0
  3.51%    0x00000166589549e3: vpmulld ymm0,ymm0,ymm1
  4.69%    0x00000166589549e8: vmovdqu ymmword ptr [rdi+rcx*4+1d0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  1.00%    0x00000166589549f1: vmovdqu ymm0,ymmword ptr [rdx+rcx*4+1f0h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.11%    0x00000166589549fa: vpmulld ymm0,ymm0,ymm0
  0.15%    0x00000166589549ff: vpmulld ymm0,ymm0,ymm1
  1.46%    0x0000016658954a04: vmovdqu ymmword ptr [rdi+rcx*4+1f0h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.26%    0x0000016658954a0d: add     r8d,80h           ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@41 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.01%    0x0000016658954a14: cmp     r8d,r10d
           0x0000016658954a17: jl      16658954860h      ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@44 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954a1d: mov     r14,qword ptr [r15+70h]  ; ImmutableOopMap{rdi=Oop rdx=Oop }
                                                         ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@44 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.06%    0x0000016658954a21: test    dword ptr [r14],eax  ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@44 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
                                                         ;   {poll}
  0.17%    0x0000016658954a24: cmp     r8d,r10d
           0x0000016658954a27: jl      16658954860h
           0x0000016658954a2d: mov     r10d,r9d
           0x0000016658954a30: add     r10d,0fffffff9h
           0x0000016658954a34: cmp     r9d,r10d
  0.00%    0x0000016658954a37: cmovl   r10d,ebx
           0x0000016658954a3b: cmp     r8d,r10d
           0x0000016658954a3e: jnl     16658954a61h      ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@25 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954a40: vmovdqu ymm0,ymmword ptr [rdx+r8*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.14%    0x0000016658954a47: vpmulld ymm0,ymm0,ymm0
  0.05%    0x0000016658954a4c: vpmulld ymm0,ymm0,ymm1
  0.03%    0x0000016658954a51: vmovdqu ymmword ptr [rdi+r8*4+10h],ymm0
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.04%    0x0000016658954a58: add     r8d,8h            ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@41 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954a5c: cmp     r8d,r10d
           0x0000016658954a5f: jl      16658954a40h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@22 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954a61: cmp     r8d,r9d
           0x0000016658954a64: jnl     16658954a81h
           0x0000016658954a66: nop                       ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@25 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954a68: mov     ecx,dword ptr [rdx+r8*4+10h]
                                                         ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@31 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954a6d: imul    ecx,ecx
           0x0000016658954a70: imul    ecx,r11d
           0x0000016658954a74: mov     dword ptr [rdi+r8*4+10h],ecx
                                                         ;*iastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@40 (line 214)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954a79: inc     r8d               ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@41 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
           0x0000016658954a7c: cmp     r8d,r9d
  0.00%    0x0000016658954a7f: jl      16658954a68h      ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::scaleSquareInRun@22 (line 213)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator::lambda$mapWithRunConsumer$9@14 (line 148)
                                                         ; - com.openkappa.simd.iterate.BitSetIterator$$Lambda$44/1209658195::acceptRun@6
  0.00%    0x0000016658954a81: vzeroupper
  0.00%    0x0000016658954a84: add     rsp,40h
           0x0000016658954a88: pop     rbp
  0.00%    0x0000016658954a89: mov     r10,qword ptr [r15+70h]
           0x0000016658954a8d: test    dword ptr [r10],eax  ;   {poll_return}
  0.00%    0x0000016658954a90: ret
           0x0000016658954a91: mov     edx,0ffffff86h
           0x0000016658954a96: mov     ebp,r9d
           0x0000016658954a99: mov     dword ptr [rsp],r11d
           0x0000016658954a9d: mov     dword ptr [rsp+4h],r8d
           0x0000016658954aa2: mov     dword ptr [rsp+8h],ebx
           0x0000016658954aa6: mov     dword ptr [rsp+0ch],edi
           0x0000016658954aaa: nop
           0x0000016658954aac: vzeroupper
           0x0000016658954aaf: call    166588c8a00h      ; ImmutableOopMap{[8]=NarrowOop [12]=NarrowOop }
                                                         ;*aload_3 {reexecute=0 rethrow=0 return_oop=0}
....................................................................................................
 96.10%  <total for region 1>

Inlining

So far, everything has been inlined. Java optimistically assumes you only have one implementation and aggressively inlines at first, deoptimising to add a branch when it sees a second implementation, deoptimising again and replacing with a virtual call if it sees a third implementation. This doesn’t matter much usually, but the cost of this not only dwarfs any savings in an iteration strategy; it also prevents various optimisations which can be applied if the code is inlined. Once again, passing a batch of work into the callback completely ameliorates this, because even if the call is virtual, the callback itself might be hot and aggressively optimised. I haven’t benchmarked this because I think the point is self-evident to anyone who would read this far.

Number of runs

It’s clear to see from the benchmark results that the best choice of iteration strategy is sensitive to what you want to do with the data, but also how it is arranged. It is well documented in database literature that real data sets tend to contain runs. If you are building a bitmap index on some attribute of your data, and you sort your data by that attribute, you will have as many bitmaps as you have attribute values, and each attribute value bitmap will contain a single run. This is almost true for any index on attributes correlated with the attribute chosen for the sort order, and is completely untrue for uncorellated attributes. There are a range of iteration strategies to choose from, and the best iteration strategy for one index may not be the best for another.

My benchmarks are available at github.

Posted on

Faster Floating Point Reductions

At the moment, I am working on a hobby project called SplitMap, which aims to evaluate aggregations over complex boolean expressions as fast as possible using the same high level constructs of the streams API. It’s already capable of performing logic that takes vanilla parallel streams 20ms in under 300μs, but I think sub 100μs is possible for these calculations. I have reached the stage where the bottleneck is floating point reductions: Java won’t vectorise these because the result would be numerically unstable. This is a bit limiting, because it often doesn’t matter very much: nobody represents money with floating point numbers, and if you’re solving stiff differential equations it won’t be numerical stability that stops you from using a more suitable language. The reality is, somebody somewhere probably really cares about this, and that’s probably why there’s no fastfp semantics in the language. This ancient proposal lay stagnant before being withdrawn, and several optimisations just can’t be implemented by the JIT compiler without violating language guarantees. An intrinsic for FMA only arrived in Java 9, 15 years after JSR 84 was withdrawn, and vectorised FMA is only available in JDK10, 18 years after JSR 84 was proposed. Computing an average is hardly numerical computing, but Java just isn’t a friendly language for this sort of thing.

Back to SplitMap. At this point, I’ve already maxed out the parallelism I can get from the fork join pool, so I want the code below to vectorise to squeeze out more performance:

  private double reduce(double[] data) {
    double reduced = 0D;
    for (int i = 0; i < data.length; ++i) {
      reduced += data[i];
    }
    return reduced;
  }

Looking at this with perfasm, it’s clear that unrolled scalar code is generated:


  0.00%    0x000001db0c6f5730: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+10h]
  6.17%    0x000001db0c6f5736: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+18h]
  6.15%    0x000001db0c6f573c: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+20h]
  6.23%    0x000001db0c6f5742: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+28h]
  6.16%    0x000001db0c6f5748: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+30h]
  6.37%    0x000001db0c6f574e: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+38h]
  6.22%    0x000001db0c6f5754: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+40h]
  6.21%    0x000001db0c6f575a: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+48h]
  6.11%    0x000001db0c6f5760: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+50h]
  6.18%    0x000001db0c6f5766: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+58h]
  6.18%    0x000001db0c6f576c: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+60h]
  6.30%    0x000001db0c6f5772: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+68h]
  6.23%    0x000001db0c6f5778: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+70h]
  6.33%    0x000001db0c6f577e: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+78h]
  6.25%    0x000001db0c6f5784: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+80h]
  6.31%    0x000001db0c6f578d: vaddsd  xmm0,xmm0,mmword ptr [rdx+rdi*8+88h]

Let’s do something really dumb – allocate an array! Then I’ll reduce vertically onto it, and do a small horizontal reduction at the end.


  @Benchmark
  public double reduceBuffered() {
    double[] buffer = new double[1024];
    for (int i = 0; i < data.length; ++i) {
      buffer[i & 1023] += data[i];
    }
    return reduce(buffer);
  }

I benchmarked this against reduce. Using size 1024 as a sanity check, it’s clear the work is just being done twice, which is reassuring. Once the array gets a bit bigger, the gains of (what I think should be) the faster vertical reduction prior to the horizontal reduction pays for the array allocation.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
reduceBuffered thrpt 1 10 333.990079 3.542656 ops/ms 1024
reduceBuffered thrpt 1 10 18.639314 0.488300 ops/ms 65536
reduceBuffered thrpt 1 10 9.313916 0.343261 ops/ms 131072
reduceSimple thrpt 1 10 656.408971 1.771530 ops/ms 1024
reduceSimple thrpt 1 10 9.840417 0.032713 ops/ms 65536
reduceSimple thrpt 1 10 4.881353 0.076707 ops/ms 131072

The code in reduceBuffered produces a slightly different result because the elements are summated in a different order, though you’re hardly likely to notice. While, by any pragmatic definition, the function performs the same operation, it is semantically different. C2 actually doesn’t vectorise this code, and I have no idea why it’s so much faster! I won’t dwell on this because this is a dead end. In any case, here’s the perfasm output:


  0.20%    0x000001eaf2e49f50: vmovsd  xmm0,qword ptr [rdx+r9*8+10h]
  0.27%    0x000001eaf2e49f57: mov     ebx,r9d
  1.95%    0x000001eaf2e49f5a: add     ebx,0fh
  0.36%    0x000001eaf2e49f5d: and     ebx,3ffh
  0.21%    0x000001eaf2e49f63: mov     ecx,r9d
  0.30%    0x000001eaf2e49f66: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  2.96%    0x000001eaf2e49f6c: vaddsd  xmm0,xmm0,mmword ptr [r8+rcx*8+10h]
  0.61%    0x000001eaf2e49f73: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.50%    0x000001eaf2e49f7a: mov     ecx,r9d
  1.78%    0x000001eaf2e49f7d: inc     ecx
  0.37%    0x000001eaf2e49f7f: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.17%    0x000001eaf2e49f85: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  0.49%    0x000001eaf2e49f8c: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+18h]
  2.51%    0x000001eaf2e49f93: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.21%    0x000001eaf2e49f9a: mov     ecx,r9d
  0.34%    0x000001eaf2e49f9d: add     ecx,2h
  0.90%    0x000001eaf2e49fa0: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.32%    0x000001eaf2e49fa6: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.28%    0x000001eaf2e49fad: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+20h]
  1.43%    0x000001eaf2e49fb4: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.38%    0x000001eaf2e49fbb: vmovsd  xmm0,qword ptr [rdx+r9*8+28h]
  0.47%    0x000001eaf2e49fc2: mov     ecx,r9d
  0.28%    0x000001eaf2e49fc5: add     ecx,3h
  0.77%    0x000001eaf2e49fc8: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.34%    0x000001eaf2e49fce: vaddsd  xmm0,xmm0,mmword ptr [r8+rcx*8+10h]
  0.93%    0x000001eaf2e49fd5: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.36%    0x000001eaf2e49fdc: mov     ecx,r9d
  0.75%    0x000001eaf2e49fdf: add     ecx,4h
  0.43%    0x000001eaf2e49fe2: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.32%    0x000001eaf2e49fe8: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.32%    0x000001eaf2e49fef: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+30h]
  1.32%    0x000001eaf2e49ff6: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.24%    0x000001eaf2e49ffd: mov     ecx,r9d
  0.40%    0x000001eaf2e4a000: add     ecx,5h
  0.79%    0x000001eaf2e4a003: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.46%    0x000001eaf2e4a009: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.22%    0x000001eaf2e4a010: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+38h]
  4.18%    0x000001eaf2e4a017: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.61%    0x000001eaf2e4a01e: mov     ecx,r9d
  0.24%    0x000001eaf2e4a021: add     ecx,6h
  0.34%    0x000001eaf2e4a024: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.69%    0x000001eaf2e4a02a: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.55%    0x000001eaf2e4a031: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+40h]
  0.95%    0x000001eaf2e4a038: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.97%    0x000001eaf2e4a03f: mov     ecx,r9d
  0.37%    0x000001eaf2e4a042: add     ecx,7h
  0.20%    0x000001eaf2e4a045: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.32%    0x000001eaf2e4a04b: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.95%    0x000001eaf2e4a052: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+48h]
  0.92%    0x000001eaf2e4a059: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.95%    0x000001eaf2e4a060: mov     ecx,r9d
  0.42%    0x000001eaf2e4a063: add     ecx,8h
  0.35%    0x000001eaf2e4a066: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.20%    0x000001eaf2e4a06c: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.97%    0x000001eaf2e4a073: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+50h]
  1.22%    0x000001eaf2e4a07a: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.67%    0x000001eaf2e4a081: mov     ecx,r9d
  0.48%    0x000001eaf2e4a084: add     ecx,9h
  0.39%    0x000001eaf2e4a087: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.32%    0x000001eaf2e4a08d: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.66%    0x000001eaf2e4a094: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+58h]
  1.14%    0x000001eaf2e4a09b: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.45%    0x000001eaf2e4a0a2: mov     ecx,r9d
  0.65%    0x000001eaf2e4a0a5: add     ecx,0ah
  0.49%    0x000001eaf2e4a0a8: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.39%    0x000001eaf2e4a0ae: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.45%    0x000001eaf2e4a0b5: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+60h]
  1.32%    0x000001eaf2e4a0bc: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.39%    0x000001eaf2e4a0c3: mov     ecx,r9d
  0.39%    0x000001eaf2e4a0c6: add     ecx,0bh
  0.65%    0x000001eaf2e4a0c9: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.53%    0x000001eaf2e4a0cf: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.37%    0x000001eaf2e4a0d6: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+68h]
  1.22%    0x000001eaf2e4a0dd: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.46%    0x000001eaf2e4a0e4: mov     ecx,r9d
  0.42%    0x000001eaf2e4a0e7: add     ecx,0ch
  0.40%    0x000001eaf2e4a0ea: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.60%    0x000001eaf2e4a0f0: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.47%    0x000001eaf2e4a0f7: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+70h]
  1.04%    0x000001eaf2e4a0fe: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  1.74%    0x000001eaf2e4a105: mov     ecx,r9d
  0.37%    0x000001eaf2e4a108: add     ecx,0dh
  0.43%    0x000001eaf2e4a10b: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.36%    0x000001eaf2e4a111: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  1.68%    0x000001eaf2e4a118: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+78h]
  2.82%    0x000001eaf2e4a11f: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  2.04%    0x000001eaf2e4a126: mov     ecx,r9d
  0.19%    0x000001eaf2e4a129: add     ecx,0eh
  0.29%    0x000001eaf2e4a12c: and     ecx,3ffh          ;*iand {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@22 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)
  0.36%    0x000001eaf2e4a132: vmovsd  xmm0,qword ptr [r8+rcx*8+10h]
  2.02%    0x000001eaf2e4a139: vaddsd  xmm0,xmm0,mmword ptr [rdx+r9*8+80h]
  0.76%    0x000001eaf2e4a143: vmovsd  qword ptr [r8+rcx*8+10h],xmm0
  1.99%    0x000001eaf2e4a14a: vmovsd  xmm0,qword ptr [rdx+r9*8+88h]
  0.39%    0x000001eaf2e4a154: vaddsd  xmm0,xmm0,mmword ptr [r8+rbx*8+10h]
  0.56%    0x000001eaf2e4a15b: vmovsd  qword ptr [r8+rbx*8+10h],xmm0
                                                         ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                         ; - com.openkappa.simd.reduction.ReduceArray::reduceBuffered@32 (line 36)
                                                         ; - com.openkappa.simd.reduction.generated.ReduceArray_reduceBuffered_jmhTest::reduceBuffered_thrpt_jmhStub@17 (line 119)

Using the same idea, but employing a trick I’ve used before for matrix multiplication, the code gets a lot faster!

  @Benchmark
  public double reduceVectorised() {
    double[] buffer = new double[1024];
    double[] temp = new double[1024];
    for (int i = 0; i < data.length >>> 10; ++i) {
      System.arraycopy(data, i * 1024, temp, 0,  temp.length);
      for (int j = 0; j < 1024; ++j) {
        buffer[j] += temp[j];
      }
    }
    return reduce(buffer);
  }

This generates a vectorised main loop:


  0.05%    0x000001e3fc0d71e0: vmovdqu ymm0,ymmword ptr [r9+r10*8+10h]
  0.16%    0x000001e3fc0d71e7: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+10h]
  1.26%    0x000001e3fc0d71ee: vmovdqu ymmword ptr [r13+r10*8+10h],ymm0
  0.39%    0x000001e3fc0d71f5: vmovdqu ymm0,ymmword ptr [r9+r10*8+30h]
  0.18%    0x000001e3fc0d71fc: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+30h]
  0.93%    0x000001e3fc0d7203: vmovdqu ymmword ptr [r13+r10*8+30h],ymm0
  0.68%    0x000001e3fc0d720a: vmovdqu ymm0,ymmword ptr [r9+r10*8+50h]
  0.17%    0x000001e3fc0d7211: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+50h]
  0.86%    0x000001e3fc0d7218: vmovdqu ymmword ptr [r13+r10*8+50h],ymm0
  0.73%    0x000001e3fc0d721f: vmovdqu ymm0,ymmword ptr [r9+r10*8+70h]
  0.19%    0x000001e3fc0d7226: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+70h]
  0.86%    0x000001e3fc0d722d: vmovdqu ymmword ptr [r13+r10*8+70h],ymm0
  0.78%    0x000001e3fc0d7234: vmovdqu ymm0,ymmword ptr [r9+r10*8+90h]
  0.17%    0x000001e3fc0d723e: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+90h]
  0.75%    0x000001e3fc0d7248: vmovdqu ymmword ptr [r13+r10*8+90h],ymm0
  0.84%    0x000001e3fc0d7252: vmovdqu ymm0,ymmword ptr [r9+r10*8+0b0h]
  0.15%    0x000001e3fc0d725c: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+0b0h]
  0.64%    0x000001e3fc0d7266: vmovdqu ymmword ptr [r13+r10*8+0b0h],ymm0
  0.92%    0x000001e3fc0d7270: vmovdqu ymm0,ymmword ptr [r9+r10*8+0d0h]
  0.15%    0x000001e3fc0d727a: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+0d0h]
  0.71%    0x000001e3fc0d7284: vmovdqu ymmword ptr [r13+r10*8+0d0h],ymm0
  0.91%    0x000001e3fc0d728e: vmovdqu ymm0,ymmword ptr [r9+r10*8+0f0h]
  0.15%    0x000001e3fc0d7298: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+0f0h]
  0.74%    0x000001e3fc0d72a2: vmovdqu ymmword ptr [r13+r10*8+0f0h],ymm0
  0.96%    0x000001e3fc0d72ac: vmovdqu ymm0,ymmword ptr [r9+r10*8+110h]
  0.12%    0x000001e3fc0d72b6: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+110h]
  0.70%    0x000001e3fc0d72c0: vmovdqu ymmword ptr [r13+r10*8+110h],ymm0
  0.99%    0x000001e3fc0d72ca: vmovdqu ymm0,ymmword ptr [r9+r10*8+130h]
  0.13%    0x000001e3fc0d72d4: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+130h]
  0.71%    0x000001e3fc0d72de: vmovdqu ymmword ptr [r13+r10*8+130h],ymm0
  0.94%    0x000001e3fc0d72e8: vmovdqu ymm0,ymmword ptr [r9+r10*8+150h]
  0.12%    0x000001e3fc0d72f2: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+150h]
  0.70%    0x000001e3fc0d72fc: vmovdqu ymmword ptr [r13+r10*8+150h],ymm0
  1.01%    0x000001e3fc0d7306: vmovdqu ymm0,ymmword ptr [r9+r10*8+170h]
  0.14%    0x000001e3fc0d7310: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+170h]
  0.75%    0x000001e3fc0d731a: vmovdqu ymmword ptr [r13+r10*8+170h],ymm0
  1.00%    0x000001e3fc0d7324: vmovdqu ymm0,ymmword ptr [r9+r10*8+190h]
  0.13%    0x000001e3fc0d732e: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+190h]
  0.67%    0x000001e3fc0d7338: vmovdqu ymmword ptr [r13+r10*8+190h],ymm0
  0.97%    0x000001e3fc0d7342: vmovdqu ymm0,ymmword ptr [r9+r10*8+1b0h]
  0.14%    0x000001e3fc0d734c: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+1b0h]
  0.72%    0x000001e3fc0d7356: vmovdqu ymmword ptr [r13+r10*8+1b0h],ymm0
  0.99%    0x000001e3fc0d7360: vmovdqu ymm0,ymmword ptr [r9+r10*8+1d0h]
  0.12%    0x000001e3fc0d736a: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+1d0h]
  0.69%    0x000001e3fc0d7374: vmovdqu ymmword ptr [r13+r10*8+1d0h],ymm0
  0.96%    0x000001e3fc0d737e: vmovdqu ymm0,ymmword ptr [r9+r10*8+1f0h]
  0.14%    0x000001e3fc0d7388: vaddpd  ymm0,ymm0,ymmword ptr [r13+r10*8+1f0h]
  0.70%    0x000001e3fc0d7392: vmovdqu ymmword ptr [r13+r10*8+1f0h],ymm0

This implementation is more than 3x faster than the original code, which includes the fuss of the buffers and copies. Wouldn’t it be nice to be able to opt in to fast floating point semantics somehow? Here are the final results (as usual, this is a throughput benchmark and higher is better):

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
reduceBuffered thrpt 1 10 329.935462 6.295871 ops/ms 1024
reduceBuffered thrpt 1 10 18.380467 0.455724 ops/ms 65536
reduceBuffered thrpt 1 10 9.257122 0.402876 ops/ms 131072
reduceSimple thrpt 1 10 654.809021 5.812795 ops/ms 1024
reduceSimple thrpt 1 10 9.694730 0.325011 ops/ms 65536
reduceSimple thrpt 1 10 4.772520 0.265691 ops/ms 131072
reduceVectorised thrpt 1 10 287.712794 27.492846 ops/ms 1024
reduceVectorised thrpt 1 10 34.454235 1.293985 ops/ms 65536
reduceVectorised thrpt 1 10 17.867701 0.813367 ops/ms 131072

The benchmark is on github.

Posted on

Sum of Squares

Streams and lambdas, especially the limited support offered for primitive types, are a fantastic addition to the Java language. They’re not supposed to be fast, but how do these features compare to a good old for loop? For a simple calculation amenable to instruction level parallelism, I compare modern and traditional implementations and observe the differences in instructions generated.

Sum of Squares

The sum of squares is the building block of a linear regression analysis so is ubiquitous in statistical computing. It is associative and therefore data-parallel. I compare four implementations: a sequential stream wrapping an array, a parallel stream wrapping an array, a generative sequential stream and a traditional for loop. The benchmark code is on github.


  @Param({"1024", "8192"})
  int size;

  private double[] data;

  @Setup(Level.Iteration)
  public void init() {
    this.data = createDoubleArray(size);
  }

  @Benchmark
  public double SS_SequentialStream() {
    return DoubleStream.of(data)
            .map(x -> x * x)
            .reduce((x, y) -> x + y)
            .orElse(0D);
  }

  @Benchmark
  public double SS_ParallelStream() {
    return DoubleStream.of(data)
            .parallel()
            .map(x -> x * x)
            .reduce((x, y) -> x + y)
            .orElse(0);
  }

  @Benchmark
  public double SS_ForLoop() {
    double result = 0D;
    for (int i = 0; i < data.length; ++i) {
      result += data[i] * data[i];
    }
    return result;
  }

  @Benchmark
  public double SS_GenerativeSequentialStream() {
    return IntStream.iterate(0, i -> i < size, i -> i + 1)
            .mapToDouble(i -> data[i])
            .map(x -> x * x)
            .reduce((x, y) -> x + y)
            .orElse(0);
  }

I must admit I prefer the readability of the stream versions, but let’s see if there is a comedown after the syntactic sugar rush.

Running a Benchmark

I compare the four implementations on an array of one million doubles. I am using JDK 9.0.1, VM 9.0.1+11 on a fairly powerful laptop with 8 processors:

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 94
model name      : Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
stepping        : 3
cpu MHz         : 2592.000
cache size      : 256 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pni dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt aes xsave osxsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

Before running the benchmark we might expect the for loop and stream to have similar performance, and the parallel version to be about eight times faster (though remember that the arrays aren’t too big). The generative version is very similar to the for loop so a slow down might not be expected.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
SS_ForLoop thrpt 1 10 258351.774491 39797.567968 ops/s 1024
SS_ForLoop thrpt 1 10 29463.408428 4814.826388 ops/s 8192
SS_GenerativeSequentialStream thrpt 1 10 219699.607567 9095.569546 ops/s 1024
SS_GenerativeSequentialStream thrpt 1 10 28351.900454 828.513989 ops/s 8192
SS_ParallelStream thrpt 1 10 22827.821827 2826.577213 ops/s 1024
SS_ParallelStream thrpt 1 10 23230.623610 273.415352 ops/s 8192
SS_SequentialStream thrpt 1 10 225431.985145 9051.538442 ops/s 1024
SS_SequentialStream thrpt 1 10 29123.734157 1333.721437 ops/s 8192

The for loop and stream are similar. The parallel version is a long way behind (yes that’s right: more threads less power), but exhibits constant scaling (incidentally, a measurement like this is a good way to guess the minimum unit of work in a parallelised implementation). If the data is large it could become profitable to use it. The generative stream is surprisingly good, almost as good as the version that wraps the array, though there is a fail-safe way to slow it down: add a limit clause to the method chain (try it…).

Profiling with perfasm, it is clear that the for loop body is being vectorised, but only the loads and multiplications are done in parallel – the complicated string of SSE instructions is the reduction, which must be done in order.

<-- unrolled load -->
  0.01%    0x00000243d8969170: vmovdqu ymm1,ymmword ptr [r11+r8*8+0f0h]
  0.07%    0x00000243d896917a: vmovdqu ymm2,ymmword ptr [r11+r8*8+0d0h]
  0.75%    0x00000243d8969184: vmovdqu ymm3,ymmword ptr [r11+r8*8+0b0h]
  0.01%    0x00000243d896918e: vmovdqu ymm4,ymmword ptr [r11+r8*8+90h]
  0.02%    0x00000243d8969198: vmovdqu ymm5,ymmword ptr [r11+r8*8+70h]
  0.03%    0x00000243d896919f: vmovdqu ymm6,ymmword ptr [r11+r8*8+50h]
  0.77%    0x00000243d89691a6: vmovdqu ymm10,ymmword ptr [r11+r8*8+30h]
  0.02%    0x00000243d89691ad: vmovdqu ymm7,ymmword ptr [r11+r8*8+10h]
<-- multiplication starts -->
  0.01%    0x00000243d89691b4: vmulpd  ymm1,ymm1,ymm1
  0.02%    0x00000243d89691b8: vmovdqu ymmword ptr [rsp+28h],ymm1
  0.76%    0x00000243d89691be: vmulpd  ymm15,ymm7,ymm7
  0.00%    0x00000243d89691c2: vmulpd  ymm12,ymm2,ymm2
  0.01%    0x00000243d89691c6: vmulpd  ymm7,ymm3,ymm3
  0.02%    0x00000243d89691ca: vmulpd  ymm8,ymm4,ymm4
  0.72%    0x00000243d89691ce: vmulpd  ymm9,ymm5,ymm5
  0.00%    0x00000243d89691d2: vmulpd  ymm11,ymm6,ymm6
  0.01%    0x00000243d89691d6: vmulpd  ymm13,ymm10,ymm10
<-- multiplication ends here, scalar reduction starts -->
  0.03%    0x00000243d89691db: vaddsd  xmm0,xmm0,xmm15
  0.72%    0x00000243d89691e0: vpshufd xmm5,xmm15,0eh
  0.01%    0x00000243d89691e6: vaddsd  xmm0,xmm0,xmm5
  2.14%    0x00000243d89691ea: vextractf128 xmm6,ymm15,1h
  0.03%    0x00000243d89691f0: vaddsd  xmm0,xmm0,xmm6
  3.21%    0x00000243d89691f4: vpshufd xmm5,xmm6,0eh
  0.02%    0x00000243d89691f9: vaddsd  xmm0,xmm0,xmm5
  2.81%    0x00000243d89691fd: vaddsd  xmm0,xmm0,xmm13
  2.82%    0x00000243d8969202: vpshufd xmm5,xmm13,0eh
  0.03%    0x00000243d8969208: vaddsd  xmm0,xmm0,xmm5
  2.87%    0x00000243d896920c: vextractf128 xmm6,ymm13,1h
  0.01%    0x00000243d8969212: vaddsd  xmm0,xmm0,xmm6
  3.03%    0x00000243d8969216: vpshufd xmm5,xmm6,0eh
  0.03%    0x00000243d896921b: vaddsd  xmm0,xmm0,xmm5
  2.94%    0x00000243d896921f: vaddsd  xmm0,xmm0,xmm11
  2.70%    0x00000243d8969224: vpshufd xmm5,xmm11,0eh
  0.03%    0x00000243d896922a: vaddsd  xmm0,xmm0,xmm5
  2.98%    0x00000243d896922e: vextractf128 xmm6,ymm11,1h
  0.01%    0x00000243d8969234: vaddsd  xmm0,xmm0,xmm6
  3.11%    0x00000243d8969238: vpshufd xmm5,xmm6,0eh
  0.03%    0x00000243d896923d: vaddsd  xmm0,xmm0,xmm5
  2.95%    0x00000243d8969241: vaddsd  xmm0,xmm0,xmm9
  2.61%    0x00000243d8969246: vpshufd xmm5,xmm9,0eh
  0.02%    0x00000243d896924c: vaddsd  xmm0,xmm0,xmm5
  2.89%    0x00000243d8969250: vextractf128 xmm6,ymm9,1h
  0.04%    0x00000243d8969256: vaddsd  xmm0,xmm0,xmm6
  3.13%    0x00000243d896925a: vpshufd xmm5,xmm6,0eh
  0.01%    0x00000243d896925f: vaddsd  xmm0,xmm0,xmm5
  2.96%    0x00000243d8969263: vaddsd  xmm0,xmm0,xmm8
  2.83%    0x00000243d8969268: vpshufd xmm4,xmm8,0eh
  0.01%    0x00000243d896926e: vaddsd  xmm0,xmm0,xmm4
  3.00%    0x00000243d8969272: vextractf128 xmm10,ymm8,1h
  0.02%    0x00000243d8969278: vaddsd  xmm0,xmm0,xmm10
  3.13%    0x00000243d896927d: vpshufd xmm4,xmm10,0eh
  0.01%    0x00000243d8969283: vaddsd  xmm0,xmm0,xmm4
  3.01%    0x00000243d8969287: vaddsd  xmm0,xmm0,xmm7
  2.95%    0x00000243d896928b: vpshufd xmm1,xmm7,0eh
  0.02%    0x00000243d8969290: vaddsd  xmm0,xmm0,xmm1
  3.06%    0x00000243d8969294: vextractf128 xmm2,ymm7,1h
  0.01%    0x00000243d896929a: vaddsd  xmm0,xmm0,xmm2
  3.07%    0x00000243d896929e: vpshufd xmm1,xmm2,0eh
  0.02%    0x00000243d89692a3: vaddsd  xmm0,xmm0,xmm1
  3.07%    0x00000243d89692a7: vaddsd  xmm0,xmm0,xmm12
  2.92%    0x00000243d89692ac: vpshufd xmm3,xmm12,0eh
  0.02%    0x00000243d89692b2: vaddsd  xmm0,xmm0,xmm3
  3.11%    0x00000243d89692b6: vextractf128 xmm1,ymm12,1h
  0.01%    0x00000243d89692bc: vaddsd  xmm0,xmm0,xmm1
  3.02%    0x00000243d89692c0: vpshufd xmm3,xmm1,0eh
  0.02%    0x00000243d89692c5: vaddsd  xmm0,xmm0,xmm3
  2.97%    0x00000243d89692c9: vmovdqu ymm1,ymmword ptr [rsp+28h]
  0.02%    0x00000243d89692cf: vaddsd  xmm0,xmm0,xmm1
  3.05%    0x00000243d89692d3: vpshufd xmm2,xmm1,0eh
  0.03%    0x00000243d89692d8: vaddsd  xmm0,xmm0,xmm2
  2.97%    0x00000243d89692dc: vextractf128 xmm14,ymm1,1h
  0.01%    0x00000243d89692e2: vaddsd  xmm0,xmm0,xmm14
  2.99%    0x00000243d89692e7: vpshufd xmm2,xmm14,0eh
  0.02%    0x00000243d89692ed: vaddsd  xmm0,xmm0,xmm2 

The sequential stream code is not as good – it is scalar – but the difference in performance is not as stark as it might be because of the inefficient scalar reduction in the for loop: this is JLS floating point semantics twisting C2’s arm behind its back.

  0.00%    0x0000021a1df54c24: vmovsd  xmm0,qword ptr [rbx+r9*8+48h]
  0.00%    0x0000021a1df54c2b: vmovsd  xmm2,qword ptr [rbx+r9*8+18h]
  0.02%    0x0000021a1df54c32: vmovsd  xmm3,qword ptr [rbx+r9*8+40h]
  2.93%    0x0000021a1df54c39: vmovsd  xmm4,qword ptr [rbx+r9*8+38h]
  0.00%    0x0000021a1df54c40: vmovsd  xmm5,qword ptr [rbx+r9*8+30h]
  0.01%    0x0000021a1df54c47: vmovsd  xmm6,qword ptr [rbx+r9*8+28h]
  0.02%    0x0000021a1df54c4e: vmovsd  xmm7,qword ptr [rbx+r9*8+20h]
  2.99%    0x0000021a1df54c55: vmulsd  xmm8,xmm0,xmm0
  0.00%    0x0000021a1df54c59: vmulsd  xmm0,xmm7,xmm7
           0x0000021a1df54c5d: vmulsd  xmm6,xmm6,xmm6
  0.01%    0x0000021a1df54c61: vmulsd  xmm5,xmm5,xmm5
  2.91%    0x0000021a1df54c65: vmulsd  xmm4,xmm4,xmm4
  0.00%    0x0000021a1df54c69: vmulsd  xmm3,xmm3,xmm3
  0.00%    0x0000021a1df54c6d: vmulsd  xmm2,xmm2,xmm2
  0.02%    0x0000021a1df54c71: vaddsd  xmm1,xmm2,xmm1
  6.10%    0x0000021a1df54c75: vaddsd  xmm0,xmm0,xmm1
  5.97%    0x0000021a1df54c79: vaddsd  xmm0,xmm6,xmm0
 16.22%    0x0000021a1df54c7d: vaddsd  xmm0,xmm5,xmm0
  7.86%    0x0000021a1df54c81: vaddsd  xmm0,xmm4,xmm0
 11.16%    0x0000021a1df54c85: vaddsd  xmm1,xmm3,xmm0
 11.90%    0x0000021a1df54c89: vaddsd  xmm0,xmm8,xmm1

The same code can be seen in SS_ParallelStream. SS_GenerativeSequentialStream is much more interesting because it hasn’t been unrolled – see the interleaved control statements. It is also not vectorised.

           0x0000013c1a639c17: vmovsd  xmm0,qword ptr [rbp+r9*8+10h]
  0.01%    0x0000013c1a639c1e: vmulsd  xmm2,xmm0,xmm0    
  0.01%    0x0000013c1a639c22: test    r8d,r8d
           0x0000013c1a639c25: jne     13c1a639e09h   
           0x0000013c1a639c2b: mov     r10d,dword ptr [r12+rax*8+8h]
           0x0000013c1a639c30: cmp     r10d,0f8022d85h 
           0x0000013c1a639c37: jne     13c1a639e3bh     
  0.01%    0x0000013c1a639c3d: vaddsd  xmm2,xmm1,xmm2
  0.01%    0x0000013c1a639c41: vmovsd  qword ptr [rdi+10h],xmm2
  0.00%    0x0000013c1a639c46: movsxd  r10,r9d
           0x0000013c1a639c49: vmovsd  xmm0,qword ptr [rbp+r10*8+18h]
  0.01%    0x0000013c1a639c50: vmulsd  xmm0,xmm0,xmm0
  0.01%    0x0000013c1a639c54: mov     r10d,dword ptr [r12+rax*8+8h]
  0.00%    0x0000013c1a639c59: cmp     r10d,0f8022d85h
           0x0000013c1a639c60: jne     13c1a639e30h
           0x0000013c1a639c66: vaddsd  xmm0,xmm0,xmm2
  0.02%    0x0000013c1a639c6a: vmovsd  qword ptr [rdi+10h],xmm0
  0.02%    0x0000013c1a639c6f: mov     r10d,r9d
           0x0000013c1a639c72: add     r10d,2h 
           0x0000013c1a639c76: cmp     r10d,r11d
           0x0000013c1a639c79: jnl     13c1a639d96h 
  0.01%    0x0000013c1a639c7f: add     r9d,4h 
  0.02%    0x0000013c1a639c83: vmovsd  xmm1,qword ptr [rbp+r10*8+10h]
  0.00%    0x0000013c1a639c8a: movzx   r8d,byte ptr [rdi+0ch]
  0.00%    0x0000013c1a639c8f: vmulsd  xmm1,xmm1,xmm1
  0.01%    0x0000013c1a639c93: test    r8d,r8d
           0x0000013c1a639c96: jne     13c1a639dfbh
  0.01%    0x0000013c1a639c9c: vaddsd  xmm1,xmm0,xmm1
  0.01%    0x0000013c1a639ca0: vmovsd  qword ptr [rdi+10h],xmm1
  0.02%    0x0000013c1a639ca5: movsxd  r8,r10d
  0.00%    0x0000013c1a639ca8: vmovsd  xmm0,qword ptr [rbp+r8*8+18h]
           0x0000013c1a639caf: vmulsd  xmm0,xmm0,xmm0
           0x0000013c1a639cb3: vaddsd  xmm0,xmm0,xmm1
  0.06%    0x0000013c1a639cb7: vmovsd  qword ptr [rdi+10h],xmm0

So it looks like streams don’t vectorise like good old for loops, and you won’t gain from Stream.parallelStream unless you have humungous arrays (which you might be avoiding for other reasons). This was actually a very nice case for the Stream because optimal code can’t be generated for floating point reductions. What happens with sum of squares for ints? Generating data in an unsurprising way:


  @Benchmark
  public int SS_SequentialStream_Int() {
    return IntStream.of(intData)
            .map(x -> x * x)
            .reduce((x, y) -> x + y)
            .orElse(0);
  }

  @Benchmark
  public int SS_ParallelStream_Int() {
    return IntStream.of(intData)
            .parallel()
            .map(x -> x * x)
            .reduce((x, y) -> x + y)
            .orElse(0);
  }

  @Benchmark
  public int SS_ForLoop_Int() {
    int result = 0;
    for (int i = 0; i < intData.length; ++i) {
      result += intData[i] * intData[i];
    }
    return result;
  }

  @Benchmark
  public int SS_GenerativeSequentialStream_Int() {
    return IntStream.iterate(0, i -> i < size, i -> i + 1)
            .map(i -> intData[i])
            .map(x -> x * x)
            .reduce((x, y) -> x + y)
            .orElse(0);
  }

The landscape has completely changed, thanks to the exploitation of associative arithmetic and the VPHADDD instruction which simplifies the reduction in the for loop.

<-- load -->
  0.00%    0x000001f5cdd8cd30: vmovdqu ymm0,ymmword ptr [rdi+r10*4+0f0h]
  1.93%    0x000001f5cdd8cd3a: vmovdqu ymm1,ymmword ptr [rdi+r10*4+0d0h]
  0.10%    0x000001f5cdd8cd44: vmovdqu ymm2,ymmword ptr [rdi+r10*4+0b0h]
  0.07%    0x000001f5cdd8cd4e: vmovdqu ymm3,ymmword ptr [rdi+r10*4+90h]
  0.05%    0x000001f5cdd8cd58: vmovdqu ymm4,ymmword ptr [rdi+r10*4+70h]
  1.75%    0x000001f5cdd8cd5f: vmovdqu ymm5,ymmword ptr [rdi+r10*4+50h]
  0.08%    0x000001f5cdd8cd66: vmovdqu ymm6,ymmword ptr [rdi+r10*4+30h]
  0.07%    0x000001f5cdd8cd6d: vmovdqu ymm7,ymmword ptr [rdi+r10*4+10h]
<-- multiply -->
  0.01%    0x000001f5cdd8cd74: vpmulld ymm0,ymm0,ymm0
  1.81%    0x000001f5cdd8cd79: vmovdqu ymmword ptr [rsp+28h],ymm0
  0.02%    0x000001f5cdd8cd7f: vpmulld ymm15,ymm7,ymm7
  1.79%    0x000001f5cdd8cd84: vpmulld ymm11,ymm1,ymm1
  0.06%    0x000001f5cdd8cd89: vpmulld ymm8,ymm2,ymm2
  1.82%    0x000001f5cdd8cd8e: vpmulld ymm9,ymm3,ymm3
  0.06%    0x000001f5cdd8cd93: vpmulld ymm10,ymm4,ymm4
  1.79%    0x000001f5cdd8cd98: vpmulld ymm12,ymm5,ymm5
  0.08%    0x000001f5cdd8cd9d: vpmulld ymm6,ymm6,ymm6
<-- vectorised reduce -->
  1.83%    0x000001f5cdd8cda2: vphaddd ymm4,ymm15,ymm15
  0.04%    0x000001f5cdd8cda7: vphaddd ymm4,ymm4,ymm7
  1.85%    0x000001f5cdd8cdac: vextracti128 xmm7,ymm4,1h
  0.07%    0x000001f5cdd8cdb2: vpaddd  xmm4,xmm4,xmm7
  1.78%    0x000001f5cdd8cdb6: vmovd   xmm7,r8d
  0.01%    0x000001f5cdd8cdbb: vpaddd  xmm7,xmm7,xmm4
  0.11%    0x000001f5cdd8cdbf: vmovd   r11d,xmm7
  0.05%    0x000001f5cdd8cdc4: vphaddd ymm4,ymm6,ymm6
  1.84%    0x000001f5cdd8cdc9: vphaddd ymm4,ymm4,ymm7
  5.43%    0x000001f5cdd8cdce: vextracti128 xmm7,ymm4,1h
  0.13%    0x000001f5cdd8cdd4: vpaddd  xmm4,xmm4,xmm7
  4.34%    0x000001f5cdd8cdd8: vmovd   xmm7,r11d
  0.36%    0x000001f5cdd8cddd: vpaddd  xmm7,xmm7,xmm4
  1.40%    0x000001f5cdd8cde1: vmovd   r8d,xmm7
  0.01%    0x000001f5cdd8cde6: vphaddd ymm6,ymm12,ymm12
  2.89%    0x000001f5cdd8cdeb: vphaddd ymm6,ymm6,ymm4
  3.25%    0x000001f5cdd8cdf0: vextracti128 xmm4,ymm6,1h
  0.87%    0x000001f5cdd8cdf6: vpaddd  xmm6,xmm6,xmm4
  6.36%    0x000001f5cdd8cdfa: vmovd   xmm4,r8d
  0.01%    0x000001f5cdd8cdff: vpaddd  xmm4,xmm4,xmm6
  1.69%    0x000001f5cdd8ce03: vmovd   r8d,xmm4
  0.03%    0x000001f5cdd8ce08: vphaddd ymm4,ymm10,ymm10
  1.83%    0x000001f5cdd8ce0d: vphaddd ymm4,ymm4,ymm7
  0.10%    0x000001f5cdd8ce12: vextracti128 xmm7,ymm4,1h
  3.29%    0x000001f5cdd8ce18: vpaddd  xmm4,xmm4,xmm7
  0.72%    0x000001f5cdd8ce1c: vmovd   xmm7,r8d
  0.23%    0x000001f5cdd8ce21: vpaddd  xmm7,xmm7,xmm4
  4.42%    0x000001f5cdd8ce25: vmovd   r11d,xmm7
  0.12%    0x000001f5cdd8ce2a: vphaddd ymm5,ymm9,ymm9
  1.69%    0x000001f5cdd8ce2f: vphaddd ymm5,ymm5,ymm1
  0.12%    0x000001f5cdd8ce34: vextracti128 xmm1,ymm5,1h
  3.28%    0x000001f5cdd8ce3a: vpaddd  xmm5,xmm5,xmm1
  0.22%    0x000001f5cdd8ce3e: vmovd   xmm1,r11d
  0.14%    0x000001f5cdd8ce43: vpaddd  xmm1,xmm1,xmm5
  3.81%    0x000001f5cdd8ce47: vmovd   r11d,xmm1
  0.22%    0x000001f5cdd8ce4c: vphaddd ymm0,ymm8,ymm8
  1.58%    0x000001f5cdd8ce51: vphaddd ymm0,ymm0,ymm3
  0.22%    0x000001f5cdd8ce56: vextracti128 xmm3,ymm0,1h
  2.82%    0x000001f5cdd8ce5c: vpaddd  xmm0,xmm0,xmm3
  0.36%    0x000001f5cdd8ce60: vmovd   xmm3,r11d
  0.20%    0x000001f5cdd8ce65: vpaddd  xmm3,xmm3,xmm0
  4.55%    0x000001f5cdd8ce69: vmovd   r8d,xmm3
  0.10%    0x000001f5cdd8ce6e: vphaddd ymm2,ymm11,ymm11
  1.71%    0x000001f5cdd8ce73: vphaddd ymm2,ymm2,ymm1
  0.09%    0x000001f5cdd8ce78: vextracti128 xmm1,ymm2,1h
  2.91%    0x000001f5cdd8ce7e: vpaddd  xmm2,xmm2,xmm1
  1.57%    0x000001f5cdd8ce82: vmovd   xmm1,r8d
  0.05%    0x000001f5cdd8ce87: vpaddd  xmm1,xmm1,xmm2
  4.84%    0x000001f5cdd8ce8b: vmovd   r11d,xmm1
  0.06%    0x000001f5cdd8ce90: vmovdqu ymm0,ymmword ptr [rsp+28h]
  0.03%    0x000001f5cdd8ce96: vphaddd ymm13,ymm0,ymm0
  1.83%    0x000001f5cdd8ce9b: vphaddd ymm13,ymm13,ymm14
  2.16%    0x000001f5cdd8cea0: vextracti128 xmm14,ymm13,1h
  0.14%    0x000001f5cdd8cea6: vpaddd  xmm13,xmm13,xmm14
  0.09%    0x000001f5cdd8ceab: vmovd   xmm14,r11d
  0.51%    0x000001f5cdd8ceb0: vpaddd  xmm14,xmm14,xmm13

If you’re the guy replacing all the for loops with streams because it’s 2018, you may be committing performance vandalism! That nice declarative API (as opposed to language feature) is at arms length and it really isn’t well optimised yet.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
SS_ForLoop_Int thrpt 1 10 1021725.018981 74264.883362 ops/s 1024
SS_ForLoop_Int thrpt 1 10 129250.855026 5764.608094 ops/s 8192
SS_GenerativeSequentialStream_Int thrpt 1 10 55069.227826 1111.903102 ops/s 1024
SS_GenerativeSequentialStream_Int thrpt 1 10 6769.176830 684.970867 ops/s 8192
SS_ParallelStream_Int thrpt 1 10 20970.387258 719.846643 ops/s 1024
SS_ParallelStream_Int thrpt 1 10 19621.397202 1514.374286 ops/s 8192
SS_SequentialStream_Int thrpt 1 10 586847.001223 22390.512706 ops/s 1024
SS_SequentialStream_Int thrpt 1 10 87620.959677 3437.083075 ops/s 8192

Parallel streams might not be the best thing to reach for.

Posted on

The Much Aligned Garbage Collector

A power of two is often a good choice for the size of an array. Sometimes you might see this being exploited to replace an integer division with a bitwise intersection. You can see why with a toy benchmark of a bloom filter, which deliberately folds in a representative cost of a hash function and array access to highlight the significance of the differential cost of the division mechanism to a method that does real work:

@State(Scope.Thread)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
public class BloomFilter {

  private long[] bitset;

  @Param({"1000", "1024"})
  int size;

  @Setup(Level.Trial)
  public void init() {
    bitset = DataUtil.createLongArray(size);
  }

  @Benchmark
  public boolean containsAnd() {
    int hash = hash();
    int pos = hash & (size - 1);
    return (bitset[pos >>> 6] & (1L << pos)) != 0;
  }

  @Benchmark
  public boolean containsAbsMod() {
    int hash = hash();
    int pos = Math.abs(hash % size);
    return (bitset[pos >>> 6] & (1L << pos)) != 0;
  }

  private int hash() {
    return ThreadLocalRandom.current().nextInt(); // a stand in for a hash function;
  }
}

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
containsAbsMod thrpt 1 10 104.063744 4.068283 ops/us 1000
containsAbsMod thrpt 1 10 103.849577 4.991040 ops/us 1024
containsAnd thrpt 1 10 161.917397 3.807912 ops/us 1024

Disregarding the case which produces an incorrect result, you can do two thirds as many lookups again in the same period of time if you just use a 1024 element bloom filter. Note that the compiler clearly won’t magically transform cases like AbsMod 1024; you need to do this yourself. You can readily see this property exploited in any open source bit set, hash set, or bloom filter you care to look at. This is boring, at least, we often get this right by accident. What is quite interesting is a multiplicative decrease in throughput of DAXPY as a result of this same choice of lengths:

@OutputTimeUnit(TimeUnit.MICROSECONDS)
@State(Scope.Thread)
public class DAXPYAlignment {

  @Param({"250", "256", "1000", "1024"})
  int size;

  double s;
  double[] a;
  double[] b;

  @Setup(Level.Trial)
  public void init() {
    s = ThreadLocalRandom.current().nextDouble();
    a = createDoubleArray(size);
    b = createDoubleArray(size);
  }

  @Benchmark
  public void daxpy(Blackhole bh) {
    for (int i = 0; i < a.length; ++i) {
      a[i] += s * b[i];
    }
    bh.consume(a);
  }
}

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
daxpy thrpt 1 10 23.499857 0.891309 ops/us 250
daxpy thrpt 1 10 22.425412 0.989512 ops/us 256
daxpy thrpt 1 10 2.420674 0.098991 ops/us 1000
daxpy thrpt 1 10 6.263005 0.175048 ops/us 1024

1000 and 1024 are somehow very different, yet 250 and 256 are almost equivalent. This will either shock you, or you will guess that my page size is 4KB. My page size is indeed 4KB (which is a power of 2 precisely because of the prohibitive cost of integer division). The 250 element array only needs 2000B of contiguous space, so the next array allocated will reside in the same page, assuming the start of the array is at the start of the page. On the other hand, the 1000 element array takes up 8000B, that’s 1.95 pages. If the second array starts immediately after this array, the array will start in the 5% remaining in the current page, take up another page, and then 90% of yet another. Is this a big deal? Perhaps, when a page is accessed for the first time, it needs to be looked up in the page table, and is then cached in the TLB. Access to the cache is fast, whereas accessing the page table is slow. That extra page access costs something, but can it really cost this much, and is this all that’s going on? Since these measurements are made on an Intel Skylake processor, they will also be affected by 4K aliasing. Let’s allocate an array in between the two we want to loop over, to vary the offsets:

  @Param({"0", "6", "12", "18", "24"})
  int offset;

  double s;
  double[] a;
  double[] b;
  double[] padding;

  @Setup(Level.Trial)
  public void init() {
    s = ThreadLocalRandom.current().nextDouble();
    a = createDoubleArray(size);
    padding = new double[offset];
    b = createDoubleArray(size);
  }

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: offset Param: size
daxpy thrpt 1 10 2.224875 0.247778 ops/us 0 1000
daxpy thrpt 1 10 6.159791 0.441525 ops/us 0 1024
daxpy thrpt 1 10 2.350425 0.136992 ops/us 6 1000
daxpy thrpt 1 10 6.047009 0.360723 ops/us 6 1024
daxpy thrpt 1 10 3.332370 0.253739 ops/us 12 1000
daxpy thrpt 1 10 6.506141 0.155733 ops/us 12 1024
daxpy thrpt 1 10 6.621031 0.345151 ops/us 18 1000
daxpy thrpt 1 10 6.827635 0.970527 ops/us 18 1024
daxpy thrpt 1 10 7.456584 0.214229 ops/us 24 1000
daxpy thrpt 1 10 7.451441 0.104871 ops/us 24 1024

The pattern is curious (pay attention to the offset parameter) – the ratio of the throughputs for each size ranging from 3x throughput degradation through to parity:

The loop in question is vectorised, which can be disabled by setting -XX:-UseSuperWord. Doing so is revealing, because the trend is still present but it is dampened to the extent it could be waved away as noise:

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: offset Param: size
daxpy thrpt 1 10 1.416452 0.079905 ops/us 0 1000
daxpy thrpt 1 10 1.806841 0.200231 ops/us 0 1024
daxpy thrpt 1 10 1.408526 0.085147 ops/us 6 1000
daxpy thrpt 1 10 1.921026 0.049655 ops/us 6 1024
daxpy thrpt 1 10 1.459186 0.076427 ops/us 12 1000
daxpy thrpt 1 10 1.809220 0.199885 ops/us 12 1024
daxpy thrpt 1 10 1.824435 0.169680 ops/us 18 1000
daxpy thrpt 1 10 1.842230 0.204414 ops/us 18 1024
daxpy thrpt 1 10 1.934717 0.229822 ops/us 24 1000
daxpy thrpt 1 10 1.964316 0.039893 ops/us 24 1024

The point is, you may not have cared about alignment much before because it’s unlikely you would have noticed unless you were really looking for it. Decent autovectorisation seems to raise the stakes enormously.

Analysis with Perfasm

It’s impossible to know for sure what the cause of this behaviour is without profiling. Since I observed this effect on my Windows development laptop, I use xperf via WinPerfAsmProfiler, which is part of JMH.

I did some instruction profiling. The same code is going to get generated in each case, with a preloop, main loop and post loop, but by looking at the sampled instruction frequency we can see what’s taking the most time in the vectorised main loop. From now on, superword parallelism is never disabled. The full output of this run can be seen at github. Here is the main loop for size=1024, offset=0, which is unrolled, spending most time loading and storing data (vmovdqu) but spending a decent amount of time in the multiplication:

  0.18%    0x0000020dddc5af90: vmovdqu ymm0,ymmword ptr [r10+r8*8+10h]
  9.27%    0x0000020dddc5af97: vmulpd  ymm0,ymm0,ymm2
  0.22%    0x0000020dddc5af9b: vaddpd  ymm0,ymm0,ymmword ptr [r11+r8*8+10h]
  7.48%    0x0000020dddc5afa2: vmovdqu ymmword ptr [r11+r8*8+10h],ymm0
 10.16%    0x0000020dddc5afa9: vmovdqu ymm0,ymmword ptr [r10+r8*8+30h]
  0.09%    0x0000020dddc5afb0: vmulpd  ymm0,ymm0,ymm2
  3.62%    0x0000020dddc5afb4: vaddpd  ymm0,ymm0,ymmword ptr [r11+r8*8+30h]
 10.60%    0x0000020dddc5afbb: vmovdqu ymmword ptr [r11+r8*8+30h],ymm0
  0.26%    0x0000020dddc5afc2: vmovdqu ymm0,ymmword ptr [r10+r8*8+50h]
  3.76%    0x0000020dddc5afc9: vmulpd  ymm0,ymm0,ymm2
  0.20%    0x0000020dddc5afcd: vaddpd  ymm0,ymm0,ymmword ptr [r11+r8*8+50h]
 13.23%    0x0000020dddc5afd4: vmovdqu ymmword ptr [r11+r8*8+50h],ymm0
  9.46%    0x0000020dddc5afdb: vmovdqu ymm0,ymmword ptr [r10+r8*8+70h]
  0.11%    0x0000020dddc5afe2: vmulpd  ymm0,ymm0,ymm2
  4.63%    0x0000020dddc5afe6: vaddpd  ymm0,ymm0,ymmword ptr [r11+r8*8+70h]
  9.78%    0x0000020dddc5afed: vmovdqu ymmword ptr [r11+r8*8+70h],ymm0

In the worst performer (size=1000, offset=0) a lot more time is spent on the stores, a much smaller fraction of observed instructions are involved with multiplication or addition. This indicates either a measurement bias (perhaps there’s some mechanism that makes a store/load easier to observe) or an increase in load/store cost.

  0.24%    0x000002d1a946f510: vmovdqu ymm0,ymmword ptr [r10+r8*8+10h]
  3.61%    0x000002d1a946f517: vmulpd  ymm0,ymm0,ymm2
  4.63%    0x000002d1a946f51b: vaddpd  ymm0,ymm0,ymmword ptr [r11+r8*8+10h]
  9.73%    0x000002d1a946f522: vmovdqu ymmword ptr [r11+r8*8+10h],ymm0
  4.34%    0x000002d1a946f529: vmovdqu ymm0,ymmword ptr [r10+r8*8+30h]
  2.13%    0x000002d1a946f530: vmulpd  ymm0,ymm0,ymm2
  7.77%    0x000002d1a946f534: vaddpd  ymm0,ymm0,ymmword ptr [r11+r8*8+30h]
 13.46%    0x000002d1a946f53b: vmovdqu ymmword ptr [r11+r8*8+30h],ymm0
  3.37%    0x000002d1a946f542: vmovdqu ymm0,ymmword ptr [r10+r8*8+50h]
  0.47%    0x000002d1a946f549: vmulpd  ymm0,ymm0,ymm2
  1.47%    0x000002d1a946f54d: vaddpd  ymm0,ymm0,ymmword ptr [r11+r8*8+50h]
 13.00%    0x000002d1a946f554: vmovdqu ymmword ptr [r11+r8*8+50h],ymm0
  4.24%    0x000002d1a946f55b: vmovdqu ymm0,ymmword ptr [r10+r8*8+70h]
  2.40%    0x000002d1a946f562: vmulpd  ymm0,ymm0,ymm2
  8.92%    0x000002d1a946f566: vaddpd  ymm0,ymm0,ymmword ptr [r11+r8*8+70h]
 14.10%    0x000002d1a946f56d: vmovdqu ymmword ptr [r11+r8*8+70h],ymm0

This trend can be seen to generally improve as 1024 is approached from below, and do bear in mind that this is a noisy measure. Interpret the numbers below as probabilities: were you to stop the execution of daxpy at random, at offset zero, you would have a 94% chance of finding yourself within the main vectorised loop. You would have a 50% chance of observing a store, and only 31% chance of observing a multiply or add. As we get further from 1024, the stores dominate the main loop, and the main loop comes to dominate the method. Again, this is approximate. When the arrays aren’t well aligned, we spend less time loading, less time multiplying and adding, and much more time storing.

classification offset = 0 offset = 6 offset = 12 offset = 18 offset = 24
add 22.79 21.46 15.41 7.77 8.03
load 12.19 11.95 15.55 21.9 21.19
multiply 8.61 7.7 9.54 13.15 8.33
store 50.29 51.3 49.16 42.34 44.56
main loop 93.88 92.41 89.66 85.16 82.11

I stop short of counting TLB misses because this is on Windows and it would be a complete pain in the arse to capture. The truth is, I still don’t have enough information to say what the cause is, but I know the stores are getting more expensive, and that I would need to be quite unlucky to have put these two arrays next to each other. I may update this post with a Linux measurement where it’s much easier to profile hardware events with perf.

After discussing the post on Twitter (see here and here), Vsevolod and Aleksey Shipilёv correctly attributed this performance bug to 4K aliasing. The effect observed here is also a contributing factor to fluctuations in throughput observed in JDK-8150730.

Garbage Collection

Is it necessary to make sure all arrays are of a size equal to a power of two and aligned with pages? In this microbenchmark, it’s easy to arrange that, for typical developers this probably isn’t feasible (which isn’t to say there aren’t people out there who do this). Fortunately, this isn’t necessary for most use cases. True to the title, this post has something to do with garbage collection. The arrays were allocated in order, and no garbage would be produced during the benchmarks, so the second array will be split across pages. Let’s put some code into the initialisation of the benchmark bound to trigger garbage collection:

  String acc = "";

  @Setup(Level.Trial)
  public void init() {
    s = ThreadLocalRandom.current().nextDouble();
    a = createDoubleArray(size);
    b = createDoubleArray(size);
    // don't do this in production
    for (int i = 0; i < 10000; ++i) {
      acc += UUID.randomUUID().toString();
    }
  }

A miracle occurs: the code speeds up!

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
daxpy thrpt 1 10 6.854161 0.261247 ops/us 1000
daxpy thrpt 1 10 6.328602 0.163391 ops/us 1024

Why? G1 has rearranged the heap and that second array probably isn’t straddling three pages any more, and is unlikely to remain in such an unlucky position relative to the first array. Future garbage collectors are expected to be even better at doing this than G1. This makes the cost of garbage collection difficult to quantify, because if it takes resources with one hand it gives them back with another.

The benchmark code is available at github.

Posted on