Vectorised Logical Operations in Java 9

This is a short post for my own reference, since I feel I have already done the topic of does Java 9 use AVX for this? to death. Cutting to the chase, Java 9 autovectorises loops to compute logical ANDs, XORs, ORs and ANDNOTs between arrays, making use of the instructions VPXOR, VPOR and VPAND. You can replicate this by running the code at github.

XOR


    @Benchmark
    public long[] xor(LongData state) {
        long[] result = new long[state.data1.length];
        long[] data1 = state.data1;
        long[] data2 = state.data2;
        for (int i = 0; i < data1.length && i < data2.length; ++i) {
            result[i] = data1[i] ^ data2[i];
        }
        return result;
    }

  0x000002884a58a5a0: vmovdqu ymm0,ymmword ptr [r10+r13*8+10h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...44
                                                ;...ea
                                                ;...10

  0x000002884a58a5a7: vpxor   ymm0,ymm0,ymmword ptr [rbx+r13*8+10h]
                                                ;...c4
                                                ;...a1
                                                ;...7d
                                                ;...ef
                                                ;...44
                                                ;...eb
                                                ;...10

  0x000002884a58a5ae: vmovdqu ymmword ptr [rax+r13*8+10h],ymm0
                                                ;...c4
                                                ;...a1
                                                ;...7e
                                                ;...7f
                                                ;...44
                                                ;...e8
                                                ;...10

OR


    @Benchmark
    public long[] or(LongData state) {
        long[] result = new long[state.data1.length];
        long[] data1 = state.data1;
        long[] data2 = state.data2;
        for (int i = 0; i < data1.length && i < data2.length; ++i) {
            result[i] = data1[i] | data2[i];
        }
        return result;
    }

  0x000001dbc4a6b738: vmovdqu ymm0,ymmword ptr [r10+rsi*8+30h]
                                                ;...c4
                                                ;...c1
                                                ;...7e
                                                ;...6f
                                                ;...44
                                                ;...f2
                                                ;...30

  0x000001dbc4a6b73f: vpor    ymm0,ymm0,ymmword ptr [rbx+rsi*8+30h]
                                                ;...c5
                                                ;...fd
                                                ;...eb
                                                ;...44
                                                ;...f3
                                                ;...30

  0x000001dbc4a6b745: vmovdqu ymmword ptr [rax+rsi*8+30h],ymm0
                                                ;...c5
                                                ;...fe
                                                ;...7f
                                                ;...44
                                                ;...f0
                                                ;...30

AND


    @Benchmark
    public long[] and(LongData state) {
        long[] result = new long[state.data1.length];
        long[] data1 = state.data1;
        long[] data2 = state.data2;
        for (int i = 0; i < data1.length && i < data2.length; ++i) {
            result[i] = data1[i] & data2[i];
        }
        return result;
    }

  0x000001d615c49da0: vmovdqu ymm0,ymmword ptr [r10+r13*8+10h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...44
                                                ;...ea
                                                ;...10

  0x000001d615c49da7: vpand   ymm0,ymm0,ymmword ptr [rbx+r13*8+10h]
                                                ;...c4
                                                ;...a1
                                                ;...7d
                                                ;...db
                                                ;...44
                                                ;...eb
                                                ;...10

  0x000001d615c49dae: vmovdqu ymmword ptr [rax+r13*8+10h],ymm0
                                                ;...c4
                                                ;...a1
                                                ;...7e
                                                ;...7f
                                                ;...44
                                                ;...e8
                                                ;...10

ANDNOT


    @Benchmark
    public long[] andNot(LongData state) {
        long[] result = new long[state.data1.length];
        long[] data1 = state.data1;
        long[] data2 = state.data2;
        for (int i = 0; i < data1.length && i < data2.length; ++i) {
            result[i] = data1[i] & ~data2[i];
        }
        return result;
    }

  0x0000018187ae8a3f: vpunpcklqdq xmm0,xmm0,xmm0  ;...c5
                                                ;...f9
                                                ;...6c
                                                ;...c0

  0x0000018187ae8a43: vinserti128 ymm0,ymm0,xmm0,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...38
                                                ;...c0
                                                ;...01

  0x000001dec4cbb14c: vmovdqu ymm1,ymmword ptr [rbx+r13*8+10h]
                                                ;...c4
                                                ;...a1
                                                ;...7e
                                                ;...6f
                                                ;...4c
                                                ;...eb
                                                ;...10

  0x000001dec4cbb153: vpxor   ymm1,ymm1,ymm0    ;...c5
                                                ;...f5
                                                ;...ef
                                                ;...c8

  0x000001dec4cbb157: vpand   ymm1,ymm1,ymmword ptr [r10+r13*8+10h]
                                                ;...c4
                                                ;...81
                                                ;...75
                                                ;...db
                                                ;...4c
                                                ;...ea
                                                ;...10

  0x000001dec4cbb15e: vmovdqu ymmword ptr [rax+r13*8+10h],ymm1
                                                ;...c4
                                                ;...a1
                                                ;...7e
                                                ;...7f
                                                ;...4c
                                                ;...e8
                                                ;...10
                                                ;*lastore {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.logical.Logicals::andNot@54 (line 51)

How much Algebra does C2 Know? Part 2: Distributivity

In part one of this series of posts, I looked at how important associativity and independence are for fast loops. C2 seems to utilise these properties to generate unrolled and pipelined machine code for loops, achieving higher throughput even in cases where the kernel of the loop is 3x slower according to vendor advertised instruction throughputs. C2 has a weird and wonderful relationship with distributivity, and hints from the programmer can both and help hinder the generation of good quality machine code.

Viability and Correctness

Distributivity is the simple notion of factoring out brackets. Is this, in general, a viable loop rewrite strategy? This can be utilised to transform the method Scale into FactoredScale, both of which perform floating point arithmetic:


    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    @Benchmark
    public double Scale(DoubleData state) {
        double value = 0D;
        double[] data = state.data1;
        for (int i = 0; i < data.length; ++i) {
            value += 3.14159 * data[i];
        }
        return value;
    }

    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    @Benchmark
    public double FactoredScale(DoubleData state) {
        double value = 0D;
        double[] data = state.data1;
        for (int i = 0; i < data.length; ++i) {
            value += data[i];
        }
        return 3.14159 * value;
    }

Running the project at github with the argument --include .*scale.*, there may be a performance gain to be had from this rewrite, but it isn’t clear cut:

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
FactoredScale thrpt 1 10 7.011606 0.274742 ops/ms 100000
FactoredScale thrpt 1 10 0.621515 0.026853 ops/ms 1000000
Scale thrpt 1 10 6.962434 0.240180 ops/ms 100000
Scale thrpt 1 10 0.671042 0.011686 ops/ms 1000000

With the real numbers it would be completely valid, but floating point arithmetic is not associative. Joseph Darcy explains why in this deep dive on floating point semantics. Broken associativity of addition entails broken distributivity of any operation over it, so the two loops are not equivalent, and they give different outputs (e.g. 15662.513298516365 vs 15662.51329851632 for one sample input). The rewrite isn’t correct even for floating point data, so it isn’t an optimisation that could be applied in good faith, except in a very small number of cases. You have to rewrite the loop yourself and figure out if the small but inevitable differences are acceptable.

Counterintuitive Performance

Integer multiplication is distributive over addition, and we can check if C2 does this rewrite by running the same code with 32 bit integer values, for now fixing a scale factor of 10 (which seems like an innocuous value, no?)


    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    @Benchmark
    public int Scale_Int(IntData state) {
        int value = 0;
        int[] data = state.data1;
        for (int i = 0; i < data.length; ++i) {
            value += 10 * data[i];
        }
        return value;
    }

    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    @Benchmark
    public int FactoredScale_Int(IntData state) {
        int value = 0;
        int[] data = state.data1;
        for (int i = 0; i < data.length; ++i) {
            value += data[i];
        }
        return 10 * value;
    }

The results are fascinating:

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
FactoredScale_Int thrpt 1 10 28.339699 0.608075 ops/ms 100000
FactoredScale_Int thrpt 1 10 2.392579 0.506413 ops/ms 1000000
Scale_Int thrpt 1 10 33.335721 0.295334 ops/ms 100000
Scale_Int thrpt 1 10 2.838242 0.448213 ops/ms 1000000

The code is doing thousands more multiplications in less time when the multiplication is not factored out of the loop. So what the devil is going on? Inspecting the assembly for the faster loop is revealing

com/openkappa/simd/scale/Scale.Scale_Int(Lcom/openkappa/simd/state/IntData;)I  [0x000001c89e499320, 0x000001c89e4996f8]  984 bytes
Argument 0 is unknown.RIP: 0x1c89e499320 Code size: 0x000003d8
[Entry Point]
[Verified Entry Point]
[Constants]
  # {method} {0x000001c8b3701b10} 'Scale_Int' '(Lcom/openkappa/simd/state/IntData;)I' in 'com/openkappa/simd/scale/Scale'
  0x000001c89e499320: int3                      ;...cc

  0x000001c89e499321: nop     word ptr [rax+rax+0h]  ;...66
                                                ;...66
                                                ;...66
                                                ;...0f
                                                ;...1f
                                                ;...84
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001c89e49932c: nop                       ;...66
                                                ;...66
                                                ;...66
                                                ;...90

  0x000001c89e499330: mov     dword ptr [rsp+0ffffffffffff9000h],eax
                                                ;...89
                                                ;...84
                                                ;...24
                                                ;...00
                                                ;...90
                                                ;...ff
                                                ;...ff

  0x000001c89e499337: push    rbp               ;...55

  0x000001c89e499338: sub     rsp,40h           ;...48
                                                ;...83
                                                ;...ec
                                                ;...40

  0x000001c89e49933c: mov     rbp,qword ptr [rdx+8h]  ;...48
                                                ;...8b
                                                ;...6a
                                                ;...08

  0x000001c89e499340: mov     ebx,dword ptr [rdx+10h]  ;...8b
                                                ;...5a
                                                ;...10

  0x000001c89e499343: mov     r13d,dword ptr [rdx]  ;...44
                                                ;...8b
                                                ;...2a

  0x000001c89e499346: mov     rcx,rdx           ;...48
                                                ;...8b
                                                ;...ca

  0x000001c89e499349: vzeroupper                ;...c5
                                                ;...f8
                                                ;...77

  0x000001c89e49934c: mov     r10,51da8d20h     ;...49
                                                ;...ba
                                                ;...20
                                                ;...8d
                                                ;...da
                                                ;...51
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001c89e499356: call indirect r10         ;...41
                                                ;...ff
                                                ;...d2

  0x000001c89e499359: mov     r11d,dword ptr [rbp+8h]  ;...44
                                                ;...8b
                                                ;...5d
                                                ;...08
                                                ; implicit exception: dispatches to 0x000001c89e4996c1
  0x000001c89e49935d: cmp     r11d,0f800016dh   ;...41
                                                ;...81
                                                ;...fb
                                                ;...6d
                                                ;...01
                                                ;...00
                                                ;...f8
                                                ;   {metadata({type array int})}
  0x000001c89e499364: jne     1c89e4996a9h      ;...0f
                                                ;...85
                                                ;...3f
                                                ;...03
                                                ;...00
                                                ;...00
                                                ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@10 (line 41)

  0x000001c89e49936a: mov     edi,dword ptr [rbp+0ch]  ;...8b
                                                ;...7d
                                                ;...0c
                                                ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@13 (line 41)

  0x000001c89e49936d: cmp     r13d,edi          ;...44
                                                ;...3b
                                                ;...ef

  0x000001c89e499370: jnl     1c89e49967ah      ;...0f
                                                ;...8d
                                                ;...04
                                                ;...03
                                                ;...00
                                                ;...00
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@14 (line 41)

  0x000001c89e499376: mov     r10d,ebp          ;...44
                                                ;...8b
                                                ;...d5

  0x000001c89e499379: mov     r8d,r13d          ;...45
                                                ;...8b
                                                ;...c5

  0x000001c89e49937c: inc     r8d               ;...41
                                                ;...ff
                                                ;...c0

  0x000001c89e49937f: shr     r10d,2h           ;...41
                                                ;...c1
                                                ;...ea
                                                ;...02

  0x000001c89e499383: and     r10d,7h           ;...41
                                                ;...83
                                                ;...e2
                                                ;...07

  0x000001c89e499387: xor     r9d,r9d           ;...45
                                                ;...33
                                                ;...c9

  0x000001c89e49938a: cmp     r8d,r9d           ;...45
                                                ;...3b
                                                ;...c1

  0x000001c89e49938d: cmovl   r8d,r9d           ;...45
                                                ;...0f
                                                ;...4c
                                                ;...c1

  0x000001c89e499391: cmp     r8d,edi           ;...44
                                                ;...3b
                                                ;...c7

  0x000001c89e499394: cmovnle r8d,edi           ;...44
                                                ;...0f
                                                ;...4f
                                                ;...c7

  0x000001c89e499398: add     r10d,r8d          ;...45
                                                ;...03
                                                ;...d0

  0x000001c89e49939b: mov     r11d,4h           ;...41
                                                ;...bb
                                                ;...04
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001c89e4993a1: sub     r11d,r10d         ;...45
                                                ;...2b
                                                ;...da

  0x000001c89e4993a4: and     r11d,7h           ;...41
                                                ;...83
                                                ;...e3
                                                ;...07

  0x000001c89e4993a8: add     r11d,r8d          ;...45
                                                ;...03
                                                ;...d8

  0x000001c89e4993ab: cmp     r11d,edi          ;...44
                                                ;...3b
                                                ;...df

  0x000001c89e4993ae: cmovnle r11d,edi          ;...44
                                                ;...0f
                                                ;...4f
                                                ;...df

  0x000001c89e4993b2: nop                       ;...66
                                                ;...90

  0x000001c89e4993b4: cmp     r13d,edi          ;...44
                                                ;...3b
                                                ;...ef

  0x000001c89e4993b7: jnb     1c89e49968bh      ;...0f
                                                ;...83
                                                ;...ce
                                                ;...02
                                                ;...00
                                                ;...00

  0x000001c89e4993bd: mov     r10d,dword ptr [rbp+r13*4+10h]
                                                ;...46
                                                ;...8b
                                                ;...54
                                                ;...ad
                                                ;...10
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@23 (line 42)

  0x000001c89e4993c2: mov     r9d,r10d          ;...45
                                                ;...8b
                                                ;...ca

  0x000001c89e4993c5: shl     r9d,3h            ;...41
                                                ;...c1
                                                ;...e1
                                                ;...03

  0x000001c89e4993c9: shl     r10d,1h           ;...41
                                                ;...d1
                                                ;...e2

  0x000001c89e4993cc: add     r9d,r10d          ;...45
                                                ;...03
                                                ;...ca

  0x000001c89e4993cf: add     ebx,r9d           ;...41
                                                ;...03
                                                ;...d9
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@25 (line 42)

  0x000001c89e4993d2: inc     r13d              ;...41
                                                ;...ff
                                                ;...c5
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@27 (line 41)

  0x000001c89e4993d5: cmp     r13d,r11d         ;...45
                                                ;...3b
                                                ;...eb

  0x000001c89e4993d8: jl      1c89e4993b4h      ;...7c
                                                ;...da
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@14 (line 41)

  0x000001c89e4993da: mov     r8d,edi           ;...44
                                                ;...8b
                                                ;...c7

  0x000001c89e4993dd: add     r8d,0ffffffc1h    ;...41
                                                ;...83
                                                ;...c0
                                                ;...c1

  0x000001c89e4993e1: mov     ecx,80000000h     ;...b9
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...80

  0x000001c89e4993e6: cmp     edi,r8d           ;...41
                                                ;...3b
                                                ;...f8

  0x000001c89e4993e9: cmovl   r8d,ecx           ;...44
                                                ;...0f
                                                ;...4c
                                                ;...c1

  0x000001c89e4993ed: cmp     r13d,r8d          ;...45
                                                ;...3b
                                                ;...e8

  0x000001c89e4993f0: jnl     1c89e499651h      ;...0f
                                                ;...8d
                                                ;...5b
                                                ;...02
                                                ;...00
                                                ;...00

  0x000001c89e4993f6: nop     word ptr [rax+rax+0h]  ;...66
                                                ;...66
                                                ;...0f
                                                ;...1f
                                                ;...84
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001c89e499400: vmovdqu ymm8,ymmword ptr [rbp+r13*4+10h]
                                                ;...c4
                                                ;...21
                                                ;...7e
                                                ;...6f
                                                ;...44
                                                ;...ad
                                                ;...10

  0x000001c89e499407: movsxd  r10,r13d          ;...4d
                                                ;...63
                                                ;...d5

  0x000001c89e49940a: vmovdqu ymm9,ymmword ptr [rbp+r10*4+30h]
                                                ;...c4
                                                ;...21
                                                ;...7e
                                                ;...6f
                                                ;...4c
                                                ;...95
                                                ;...30

  0x000001c89e499411: vmovdqu ymm13,ymmword ptr [rbp+r10*4+0f0h]
                                                ;...c4
                                                ;...21
                                                ;...7e
                                                ;...6f
                                                ;...ac
                                                ;...95
                                                ;...f0
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001c89e49941b: vmovdqu ymm12,ymmword ptr [rbp+r10*4+50h]
                                                ;...c4
                                                ;...21
                                                ;...7e
                                                ;...6f
                                                ;...64
                                                ;...95
                                                ;...50

  0x000001c89e499422: vmovdqu ymm4,ymmword ptr [rbp+r10*4+70h]
                                                ;...c4
                                                ;...a1
                                                ;...7e
                                                ;...6f
                                                ;...64
                                                ;...95
                                                ;...70

  0x000001c89e499429: vmovdqu ymm3,ymmword ptr [rbp+r10*4+90h]
                                                ;...c4
                                                ;...a1
                                                ;...7e
                                                ;...6f
                                                ;...9c
                                                ;...95
                                                ;...90
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001c89e499433: vmovdqu ymm2,ymmword ptr [rbp+r10*4+0b0h]
                                                ;...c4
                                                ;...a1
                                                ;...7e
                                                ;...6f
                                                ;...94
                                                ;...95
                                                ;...b0
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001c89e49943d: vmovdqu ymm0,ymmword ptr [rbp+r10*4+0d0h]
                                                ;...c4
                                                ;...a1
                                                ;...7e
                                                ;...6f
                                                ;...84
                                                ;...95
                                                ;...d0
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@23 (line 42)

  0x000001c89e499447: vpslld  ymm11,ymm8,1h     ;...c4
                                                ;...c1
                                                ;...25
                                                ;...72
                                                ;...f0
                                                ;...01

  0x000001c89e49944d: vpslld  ymm1,ymm0,1h      ;...c5
                                                ;...f5
                                                ;...72
                                                ;...f0
                                                ;...01

  0x000001c89e499452: vpslld  ymm0,ymm0,3h      ;...c5
                                                ;...fd
                                                ;...72
                                                ;...f0
                                                ;...03

  0x000001c89e499457: vpaddd  ymm5,ymm0,ymm1    ;...c5
                                                ;...fd
                                                ;...fe
                                                ;...e9

  0x000001c89e49945b: vpslld  ymm0,ymm2,3h      ;...c5
                                                ;...fd
                                                ;...72
                                                ;...f2
                                                ;...03

  0x000001c89e499460: vpslld  ymm7,ymm3,3h      ;...c5
                                                ;...c5
                                                ;...72
                                                ;...f3
                                                ;...03

  0x000001c89e499465: vpslld  ymm10,ymm4,3h     ;...c5
                                                ;...ad
                                                ;...72
                                                ;...f4
                                                ;...03

  0x000001c89e49946a: vpslld  ymm15,ymm12,3h    ;...c4
                                                ;...c1
                                                ;...05
                                                ;...72
                                                ;...f4
                                                ;...03

  0x000001c89e499470: vpslld  ymm14,ymm13,3h    ;...c4
                                                ;...c1
                                                ;...0d
                                                ;...72
                                                ;...f5
                                                ;...03

  0x000001c89e499476: vpslld  ymm1,ymm9,3h      ;...c4
                                                ;...c1
                                                ;...75
                                                ;...72
                                                ;...f1
                                                ;...03

  0x000001c89e49947c: vpslld  ymm2,ymm2,1h      ;...c5
                                                ;...ed
                                                ;...72
                                                ;...f2
                                                ;...01

  0x000001c89e499481: vpaddd  ymm6,ymm0,ymm2    ;...c5
                                                ;...fd
                                                ;...fe
                                                ;...f2

  0x000001c89e499485: vpslld  ymm0,ymm3,1h      ;...c5
                                                ;...fd
                                                ;...72
                                                ;...f3
                                                ;...01

  0x000001c89e49948a: vpaddd  ymm7,ymm7,ymm0    ;...c5
                                                ;...c5
                                                ;...fe
                                                ;...f8

  0x000001c89e49948e: vpslld  ymm0,ymm4,1h      ;...c5
                                                ;...fd
                                                ;...72
                                                ;...f4
                                                ;...01

  0x000001c89e499493: vpaddd  ymm10,ymm10,ymm0  ;...c5
                                                ;...2d
                                                ;...fe
                                                ;...d0

  0x000001c89e499497: vpslld  ymm0,ymm12,1h     ;...c4
                                                ;...c1
                                                ;...7d
                                                ;...72
                                                ;...f4
                                                ;...01

  0x000001c89e49949d: vpaddd  ymm12,ymm15,ymm0  ;...c5
                                                ;...05
                                                ;...fe
                                                ;...e0

  0x000001c89e4994a1: vpslld  ymm0,ymm13,1h     ;...c4
                                                ;...c1
                                                ;...7d
                                                ;...72
                                                ;...f5
                                                ;...01

  0x000001c89e4994a7: vpaddd  ymm4,ymm14,ymm0   ;...c5
                                                ;...8d
                                                ;...fe
                                                ;...e0

  0x000001c89e4994ab: vpslld  ymm0,ymm9,1h      ;...c4
                                                ;...c1
                                                ;...7d
                                                ;...72
                                                ;...f1
                                                ;...01

  0x000001c89e4994b1: vpaddd  ymm2,ymm1,ymm0    ;...c5
                                                ;...f5
                                                ;...fe
                                                ;...d0

  0x000001c89e4994b5: vpslld  ymm0,ymm8,3h      ;...c4
                                                ;...c1
                                                ;...7d
                                                ;...72
                                                ;...f0
                                                ;...03

  0x000001c89e4994bb: vpaddd  ymm8,ymm0,ymm11   ;...c4
                                                ;...41
                                                ;...7d
                                                ;...fe
                                                ;...c3

  0x000001c89e4994c0: vphaddd ymm0,ymm8,ymm8    ;...c4
                                                ;...c2
                                                ;...3d
                                                ;...02
                                                ;...c0

  0x000001c89e4994c5: vphaddd ymm0,ymm0,ymm3    ;...c4
                                                ;...e2
                                                ;...7d
                                                ;...02
                                                ;...c3

  0x000001c89e4994ca: vextracti128 xmm3,ymm0,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c3
                                                ;...01

  0x000001c89e4994d0: vpaddd  xmm0,xmm0,xmm3    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c3

  0x000001c89e4994d4: vmovd   xmm3,ebx          ;...c5
                                                ;...f9
                                                ;...6e
                                                ;...db

  0x000001c89e4994d8: vpaddd  xmm3,xmm3,xmm0    ;...c5
                                                ;...e1
                                                ;...fe
                                                ;...d8

  0x000001c89e4994dc: vmovd   r10d,xmm3         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...da

  0x000001c89e4994e1: vphaddd ymm0,ymm2,ymm2    ;...c4
                                                ;...e2
                                                ;...6d
                                                ;...02
                                                ;...c2

  0x000001c89e4994e6: vphaddd ymm0,ymm0,ymm3    ;...c4
                                                ;...e2
                                                ;...7d
                                                ;...02
                                                ;...c3

  0x000001c89e4994eb: vextracti128 xmm3,ymm0,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c3
                                                ;...01

  0x000001c89e4994f1: vpaddd  xmm0,xmm0,xmm3    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c3

  0x000001c89e4994f5: vmovd   xmm3,r10d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...da

  0x000001c89e4994fa: vpaddd  xmm3,xmm3,xmm0    ;...c5
                                                ;...e1
                                                ;...fe
                                                ;...d8

  0x000001c89e4994fe: vmovd   r11d,xmm3         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...db

  0x000001c89e499503: vphaddd ymm2,ymm12,ymm12  ;...c4
                                                ;...c2
                                                ;...1d
                                                ;...02
                                                ;...d4

  0x000001c89e499508: vphaddd ymm2,ymm2,ymm0    ;...c4
                                                ;...e2
                                                ;...6d
                                                ;...02
                                                ;...d0

  0x000001c89e49950d: vextracti128 xmm0,ymm2,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...d0
                                                ;...01

  0x000001c89e499513: vpaddd  xmm2,xmm2,xmm0    ;...c5
                                                ;...e9
                                                ;...fe
                                                ;...d0

  0x000001c89e499517: vmovd   xmm0,r11d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...c3

  0x000001c89e49951c: vpaddd  xmm0,xmm0,xmm2    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c2

  0x000001c89e499520: vmovd   r10d,xmm0         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...c2

  0x000001c89e499525: vphaddd ymm0,ymm10,ymm10  ;...c4
                                                ;...c2
                                                ;...2d
                                                ;...02
                                                ;...c2

  0x000001c89e49952a: vphaddd ymm0,ymm0,ymm3    ;...c4
                                                ;...e2
                                                ;...7d
                                                ;...02
                                                ;...c3

  0x000001c89e49952f: vextracti128 xmm3,ymm0,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c3
                                                ;...01

  0x000001c89e499535: vpaddd  xmm0,xmm0,xmm3    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c3

  0x000001c89e499539: vmovd   xmm3,r10d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...da

  0x000001c89e49953e: vpaddd  xmm3,xmm3,xmm0    ;...c5
                                                ;...e1
                                                ;...fe
                                                ;...d8

  0x000001c89e499542: vmovd   r11d,xmm3         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...db

  0x000001c89e499547: vphaddd ymm2,ymm7,ymm7    ;...c4
                                                ;...e2
                                                ;...45
                                                ;...02
                                                ;...d7

  0x000001c89e49954c: vphaddd ymm2,ymm2,ymm0    ;...c4
                                                ;...e2
                                                ;...6d
                                                ;...02
                                                ;...d0

  0x000001c89e499551: vextracti128 xmm0,ymm2,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...d0
                                                ;...01

  0x000001c89e499557: vpaddd  xmm2,xmm2,xmm0    ;...c5
                                                ;...e9
                                                ;...fe
                                                ;...d0

  0x000001c89e49955b: vmovd   xmm0,r11d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...c3

  0x000001c89e499560: vpaddd  xmm0,xmm0,xmm2    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c2

  0x000001c89e499564: vmovd   r10d,xmm0         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...c2

  0x000001c89e499569: vphaddd ymm0,ymm6,ymm6    ;...c4
                                                ;...e2
                                                ;...4d
                                                ;...02
                                                ;...c6

  0x000001c89e49956e: vphaddd ymm0,ymm0,ymm3    ;...c4
                                                ;...e2
                                                ;...7d
                                                ;...02
                                                ;...c3

  0x000001c89e499573: vextracti128 xmm3,ymm0,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c3
                                                ;...01

  0x000001c89e499579: vpaddd  xmm0,xmm0,xmm3    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c3

  0x000001c89e49957d: vmovd   xmm3,r10d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...da

  0x000001c89e499582: vpaddd  xmm3,xmm3,xmm0    ;...c5
                                                ;...e1
                                                ;...fe
                                                ;...d8

  0x000001c89e499586: vmovd   r11d,xmm3         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...db

  0x000001c89e49958b: vphaddd ymm2,ymm5,ymm5    ;...c4
                                                ;...e2
                                                ;...55
                                                ;...02
                                                ;...d5

  0x000001c89e499590: vphaddd ymm2,ymm2,ymm0    ;...c4
                                                ;...e2
                                                ;...6d
                                                ;...02
                                                ;...d0

  0x000001c89e499595: vextracti128 xmm0,ymm2,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...d0
                                                ;...01

  0x000001c89e49959b: vpaddd  xmm2,xmm2,xmm0    ;...c5
                                                ;...e9
                                                ;...fe
                                                ;...d0

  0x000001c89e49959f: vmovd   xmm0,r11d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...c3

  0x000001c89e4995a4: vpaddd  xmm0,xmm0,xmm2    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c2

  0x000001c89e4995a8: vmovd   r10d,xmm0         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...c2

  0x000001c89e4995ad: vphaddd ymm2,ymm4,ymm4    ;...c4
                                                ;...e2
                                                ;...5d
                                                ;...02
                                                ;...d4

  0x000001c89e4995b2: vphaddd ymm2,ymm2,ymm1    ;...c4
                                                ;...e2
                                                ;...6d
                                                ;...02
                                                ;...d1

  0x000001c89e4995b7: vextracti128 xmm1,ymm2,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...d1
                                                ;...01

  0x000001c89e4995bd: vpaddd  xmm2,xmm2,xmm1    ;...c5
                                                ;...e9
                                                ;...fe
                                                ;...d1

  0x000001c89e4995c1: vmovd   xmm1,r10d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...ca

  0x000001c89e4995c6: vpaddd  xmm1,xmm1,xmm2    ;...c5
                                                ;...f1
                                                ;...fe
                                                ;...ca

  0x000001c89e4995ca: vmovd   ebx,xmm1          ;...c5
                                                ;...f9
                                                ;...7e
                                                ;...cb
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@25 (line 42)

  0x000001c89e4995ce: add     r13d,40h          ;...41
                                                ;...83
                                                ;...c5
                                                ;...40
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@27 (line 41)

  0x000001c89e4995d2: cmp     r13d,r8d          ;...45
                                                ;...3b
                                                ;...e8

  0x000001c89e4995d5: jl      1c89e499400h      ;...0f
                                                ;...8c
                                                ;...25
                                                ;...fe
                                                ;...ff
                                                ;...ff
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@14 (line 41)

  0x000001c89e4995db: mov     r10d,edi          ;...44
                                                ;...8b
                                                ;...d7

  0x000001c89e4995de: add     r10d,0fffffff9h   ;...41
                                                ;...83
                                                ;...c2
                                                ;...f9

  0x000001c89e4995e2: cmp     edi,r10d          ;...41
                                                ;...3b
                                                ;...fa

  0x000001c89e4995e5: cmovl   r10d,ecx          ;...44
                                                ;...0f
                                                ;...4c
                                                ;...d1

  0x000001c89e4995e9: cmp     r13d,r10d         ;...45
                                                ;...3b
                                                ;...ea

  0x000001c89e4995ec: jnl     1c89e49962eh      ;...7d
                                                ;...40

  0x000001c89e4995ee: nop                       ;...66
                                                ;...90

  0x000001c89e4995f0: vmovdqu ymm0,ymmword ptr [rbp+r13*4+10h]
                                                ;...c4
                                                ;...a1
                                                ;...7e
                                                ;...6f
                                                ;...44
                                                ;...ad
                                                ;...10
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@23 (line 42)

  0x000001c89e4995f7: vpslld  ymm1,ymm0,3h      ;...c5
                                                ;...f5
                                                ;...72
                                                ;...f0
                                                ;...03

  0x000001c89e4995fc: vpslld  ymm0,ymm0,1h      ;...c5
                                                ;...fd
                                                ;...72
                                                ;...f0
                                                ;...01

  0x000001c89e499601: vpaddd  ymm0,ymm1,ymm0    ;...c5
                                                ;...f5
                                                ;...fe
                                                ;...c0

  0x000001c89e499605: vphaddd ymm3,ymm0,ymm0    ;...c4
                                                ;...e2
                                                ;...7d
                                                ;...02
                                                ;...d8

  0x000001c89e49960a: vphaddd ymm3,ymm3,ymm1    ;...c4
                                                ;...e2
                                                ;...65
                                                ;...02
                                                ;...d9

  0x000001c89e49960f: vextracti128 xmm1,ymm3,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...d9
                                                ;...01

  0x000001c89e499615: vpaddd  xmm3,xmm3,xmm1    ;...c5
                                                ;...e1
                                                ;...fe
                                                ;...d9

  0x000001c89e499619: vmovd   xmm1,ebx          ;...c5
                                                ;...f9
                                                ;...6e
                                                ;...cb

  0x000001c89e49961d: vpaddd  xmm1,xmm1,xmm3    ;...c5
                                                ;...f1
                                                ;...fe
                                                ;...cb

  0x000001c89e499621: vmovd   ebx,xmm1          ;...c5
                                                ;...f9
                                                ;...7e
                                                ;...cb
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@25 (line 42)

  0x000001c89e499625: add     r13d,8h           ;...41
                                                ;...83
                                                ;...c5
                                                ;...08
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@27 (line 41)

  0x000001c89e499629: cmp     r13d,r10d         ;...45
                                                ;...3b
                                                ;...ea

  0x000001c89e49962c: jl      1c89e4995f0h      ;...7c
                                                ;...c2
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@14 (line 41)

  0x000001c89e49962e: cmp     r13d,edi          ;...44
                                                ;...3b
                                                ;...ef

  0x000001c89e499631: jnl     1c89e499651h      ;...7d
                                                ;...1e

  0x000001c89e499633: nop                       ;...90

  0x000001c89e499634: mov     r11d,dword ptr [rbp+r13*4+10h]
                                                ;...46
                                                ;...8b
                                                ;...5c
                                                ;...ad
                                                ;...10
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@23 (line 42)

  0x000001c89e499639: mov     r10d,r11d         ;...45
                                                ;...8b
                                                ;...d3

  0x000001c89e49963c: shl     r10d,3h           ;...41
                                                ;...c1
                                                ;...e2
                                                ;...03

  0x000001c89e499640: shl     r11d,1h           ;...41
                                                ;...d1
                                                ;...e3

  0x000001c89e499643: add     r10d,r11d         ;...45
                                                ;...03
                                                ;...d3

  0x000001c89e499646: add     ebx,r10d          ;...41
                                                ;...03
                                                ;...da
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@25 (line 42)

  0x000001c89e499649: inc     r13d              ;...41
                                                ;...ff
                                                ;...c5
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@27 (line 41)

  0x000001c89e49964c: cmp     r13d,edi          ;...44
                                                ;...3b
                                                ;...ef

  0x000001c89e49964f: jl      1c89e499634h      ;...7c
                                                ;...e3
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@14 (line 41)

  0x000001c89e499651: cmp     r13d,edi          ;...44
                                                ;...3b
                                                ;...ef

  0x000001c89e499654: jnl     1c89e49967ah      ;...7d
                                                ;...24

  0x000001c89e499656: nop                       ;...66
                                                ;...90

  0x000001c89e499658: cmp     r13d,edi          ;...44
                                                ;...3b
                                                ;...ef

  0x000001c89e49965b: jnb     1c89e499691h      ;...73
                                                ;...34

  0x000001c89e49965d: mov     r11d,dword ptr [rbp+r13*4+10h]
                                                ;...46
                                                ;...8b
                                                ;...5c
                                                ;...ad
                                                ;...10
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@23 (line 42)

  0x000001c89e499662: mov     r10d,r11d         ;...45
                                                ;...8b
                                                ;...d3

  0x000001c89e499665: shl     r10d,3h           ;...41
                                                ;...c1
                                                ;...e2
                                                ;...03

  0x000001c89e499669: shl     r11d,1h           ;...41
                                                ;...d1
                                                ;...e3

  0x000001c89e49966c: add     r10d,r11d         ;...45
                                                ;...03
                                                ;...d3

  0x000001c89e49966f: add     ebx,r10d          ;...41
                                                ;...03
                                                ;...da
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@25 (line 42)

  0x000001c89e499672: inc     r13d              ;...41
                                                ;...ff
                                                ;...c5
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@27 (line 41)

  0x000001c89e499675: cmp     r13d,edi          ;...44
                                                ;...3b
                                                ;...ef

  0x000001c89e499678: jl      1c89e499658h      ;...7c
                                                ;...de
                                                ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int@10 (line 41)

  0x000001c89e49967a: mov     eax,ebx           ;...8b
                                                ;...c3

  0x000001c89e49967c: vzeroupper                ;...c5
                                                ;...f8
                                                ;...77

  0x000001c89e49967f: add     rsp,40h           ;...48
                                                ;...83
                                                ;...c4
                                                ;...40

  0x000001c89e499683: pop     rbp               ;...5d

  0x000001c89e499684: test    dword ptr [1c88a9e0000h],eax
                                                ;...85
                                                ;...05
                                                ;...76
                                                ;...69
                                                ;...54
                                                ;...ec
                                                ;   {poll_return}
  0x000001c89e49968a: ret 

The loop is aggressively unrolled, pipelined, and vectorised. Moreover, the multiplication by ten results not in a multiplication but two left shifts (see VPSLLD) and an addition. Note that x << 1 + x << 3 = x * 10 and C2 seems to know it; this rewrite can be applied because it can be proven statically that the factor is always 10. The “optimised” loop doesn’t vectorise at all (and I have no idea why not – isn’t this a bug? Yes it is.)

com/openkappa/simd/scale/Scale.FactoredScale_Int(Lcom/openkappa/simd/state/IntData;)I  [0x000002bbebeda320, 0x000002bbebeda4b8]  408 bytes
Argument 0 is unknown.RIP: 0x2bbebeda320 Code size: 0x00000198
[Entry Point]
[Verified Entry Point]
[Constants]
  # {method} {0x000002bb81241148} 'FactoredScale_Int' '(Lcom/openkappa/simd/state/IntData;)I' in 'com/openkappa/simd/scale/Scale'
  0x000002bbebeda320: int3                      ;...cc

  0x000002bbebeda321: nop     word ptr [rax+rax+0h]  ;...66
                                                ;...66
                                                ;...66
                                                ;...0f
                                                ;...1f
                                                ;...84
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002bbebeda32c: nop                       ;...66
                                                ;...66
                                                ;...66
                                                ;...90

  0x000002bbebeda330: mov     dword ptr [rsp+0ffffffffffff9000h],eax
                                                ;...89
                                                ;...84
                                                ;...24
                                                ;...00
                                                ;...90
                                                ;...ff
                                                ;...ff

  0x000002bbebeda337: push    rbp               ;...55

  0x000002bbebeda338: sub     rsp,40h           ;...48
                                                ;...83
                                                ;...ec
                                                ;...40

  0x000002bbebeda33c: mov     rbp,qword ptr [rdx+8h]  ;...48
                                                ;...8b
                                                ;...6a
                                                ;...08

  0x000002bbebeda340: mov     ebx,dword ptr [rdx+10h]  ;...8b
                                                ;...5a
                                                ;...10

  0x000002bbebeda343: mov     r13d,dword ptr [rdx]  ;...44
                                                ;...8b
                                                ;...2a

  0x000002bbebeda346: mov     rcx,rdx           ;...48
                                                ;...8b
                                                ;...ca

  0x000002bbebeda349: mov     r10,51da8d20h     ;...49
                                                ;...ba
                                                ;...20
                                                ;...8d
                                                ;...da
                                                ;...51
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002bbebeda353: call indirect r10         ;...41
                                                ;...ff
                                                ;...d2

  0x000002bbebeda356: mov     r10d,dword ptr [rbp+8h]  ;...44
                                                ;...8b
                                                ;...55
                                                ;...08
                                                ; implicit exception: dispatches to 0x000002bbebeda46d
  0x000002bbebeda35a: cmp     r10d,0f800016dh   ;...41
                                                ;...81
                                                ;...fa
                                                ;...6d
                                                ;...01
                                                ;...00
                                                ;...f8
                                                ;   {metadata({type array int})}
  0x000002bbebeda361: jne     2bbebeda459h      ;...0f
                                                ;...85
                                                ;...f2
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@10 (line 52)

  0x000002bbebeda367: mov     r10d,dword ptr [rbp+0ch]
                                                ;...44
                                                ;...8b
                                                ;...55
                                                ;...0c
                                                ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@13 (line 52)

  0x000002bbebeda36b: cmp     r13d,r10d         ;...45
                                                ;...3b
                                                ;...ea

  0x000002bbebeda36e: jnl     2bbebeda422h      ;...0f
                                                ;...8d
                                                ;...ae
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@14 (line 52)

  0x000002bbebeda374: mov     r11d,r13d         ;...45
                                                ;...8b
                                                ;...dd

  0x000002bbebeda377: inc     r11d              ;...41
                                                ;...ff
                                                ;...c3

  0x000002bbebeda37a: xor     r8d,r8d           ;...45
                                                ;...33
                                                ;...c0

  0x000002bbebeda37d: cmp     r11d,r8d          ;...45
                                                ;...3b
                                                ;...d8

  0x000002bbebeda380: cmovl   r11d,r8d          ;...45
                                                ;...0f
                                                ;...4c
                                                ;...d8

  0x000002bbebeda384: cmp     r11d,r10d         ;...45
                                                ;...3b
                                                ;...da

  0x000002bbebeda387: cmovnle r11d,r10d         ;...45
                                                ;...0f
                                                ;...4f
                                                ;...da

  0x000002bbebeda38b: nop                       ;...90
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@21 (line 53)

  0x000002bbebeda38c: cmp     r13d,r10d         ;...45
                                                ;...3b
                                                ;...ea

  0x000002bbebeda38f: jnb     2bbebeda43ch      ;...0f
                                                ;...83
                                                ;...a7
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002bbebeda395: add     ebx,dword ptr [rbp+r13*4+10h]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...ad
                                                ;...10
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@22 (line 53)

  0x000002bbebeda39a: inc     r13d              ;...41
                                                ;...ff
                                                ;...c5
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@24 (line 52)

  0x000002bbebeda39d: cmp     r13d,r11d         ;...45
                                                ;...3b
                                                ;...eb

  0x000002bbebeda3a0: jl      2bbebeda38ch      ;...7c
                                                ;...ea
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@14 (line 52)

  0x000002bbebeda3a2: mov     r11d,r10d         ;...45
                                                ;...8b
                                                ;...da

  0x000002bbebeda3a5: add     r11d,0fffffff9h   ;...41
                                                ;...83
                                                ;...c3
                                                ;...f9

  0x000002bbebeda3a9: mov     r8d,80000000h     ;...41
                                                ;...b8
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...80

  0x000002bbebeda3af: cmp     r10d,r11d         ;...45
                                                ;...3b
                                                ;...d3

  0x000002bbebeda3b2: cmovl   r11d,r8d          ;...45
                                                ;...0f
                                                ;...4c
                                                ;...d8

  0x000002bbebeda3b6: cmp     r13d,r11d         ;...45
                                                ;...3b
                                                ;...eb

  0x000002bbebeda3b9: jnl     2bbebeda409h      ;...7d
                                                ;...4e

  0x000002bbebeda3bb: nop     dword ptr [rax+rax+0h]  ;...0f
                                                ;...1f
                                                ;...44
                                                ;...00
                                                ;...00
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@21 (line 53)

  0x000002bbebeda3c0: add     ebx,dword ptr [rbp+r13*4+10h]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...ad
                                                ;...10

  0x000002bbebeda3c5: movsxd  r8,r13d           ;...4d
                                                ;...63
                                                ;...c5

  0x000002bbebeda3c8: add     ebx,dword ptr [rbp+r8*4+14h]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...85
                                                ;...14

  0x000002bbebeda3cd: add     ebx,dword ptr [rbp+r8*4+18h]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...85
                                                ;...18

  0x000002bbebeda3d2: add     ebx,dword ptr [rbp+r8*4+1ch]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...85
                                                ;...1c

  0x000002bbebeda3d7: add     ebx,dword ptr [rbp+r8*4+20h]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...85
                                                ;...20

  0x000002bbebeda3dc: add     ebx,dword ptr [rbp+r8*4+24h]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...85
                                                ;...24

  0x000002bbebeda3e1: add     ebx,dword ptr [rbp+r8*4+28h]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...85
                                                ;...28

  0x000002bbebeda3e6: add     ebx,dword ptr [rbp+r8*4+2ch]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...85
                                                ;...2c
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@22 (line 53)

  0x000002bbebeda3eb: add     r13d,8h           ;...41
                                                ;...83
                                                ;...c5
                                                ;...08
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@24 (line 52)

  0x000002bbebeda3ef: cmp     r13d,r11d         ;...45
                                                ;...3b
                                                ;...eb

  0x000002bbebeda3f2: jl      2bbebeda3c0h      ;...7c
                                                ;...cc
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@14 (line 52)

  0x000002bbebeda3f4: cmp     r13d,r10d         ;...45
                                                ;...3b
                                                ;...ea

  0x000002bbebeda3f7: jnl     2bbebeda409h      ;...7d
                                                ;...10

  0x000002bbebeda3f9: nop                       ;...66
                                                ;...66
                                                ;...90
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@21 (line 53)

  0x000002bbebeda3fc: add     ebx,dword ptr [rbp+r13*4+10h]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...ad
                                                ;...10
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@22 (line 53)

  0x000002bbebeda401: inc     r13d              ;...41
                                                ;...ff
                                                ;...c5
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@24 (line 52)

  0x000002bbebeda404: cmp     r13d,r10d         ;...45
                                                ;...3b
                                                ;...ea

  0x000002bbebeda407: jl      2bbebeda3fch      ;...7c
                                                ;...f3
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@14 (line 52)

  0x000002bbebeda409: cmp     r13d,r10d         ;...45
                                                ;...3b
                                                ;...ea

  0x000002bbebeda40c: jnl     2bbebeda422h      ;...7d
                                                ;...14

  0x000002bbebeda40e: nop                       ;...66
                                                ;...90
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@21 (line 53)

  0x000002bbebeda410: cmp     r13d,r10d         ;...45
                                                ;...3b
                                                ;...ea

  0x000002bbebeda413: jnb     2bbebeda442h      ;...73
                                                ;...2d

  0x000002bbebeda415: add     ebx,dword ptr [rbp+r13*4+10h]
                                                ;...42
                                                ;...03
                                                ;...5c
                                                ;...ad
                                                ;...10
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@22 (line 53)

  0x000002bbebeda41a: inc     r13d              ;...41
                                                ;...ff
                                                ;...c5
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@24 (line 52)

  0x000002bbebeda41d: cmp     r13d,r10d         ;...45
                                                ;...3b
                                                ;...ea

  0x000002bbebeda420: jl      2bbebeda410h      ;...7c
                                                ;...ee
                                                ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@10 (line 52)

  0x000002bbebeda422: mov     r11d,ebx          ;...44
                                                ;...8b
                                                ;...db

  0x000002bbebeda425: shl     r11d,3h           ;...41
                                                ;...c1
                                                ;...e3
                                                ;...03

  0x000002bbebeda429: shl     ebx,1h            ;...d1
                                                ;...e3

  0x000002bbebeda42b: add     ebx,r11d          ;...41
                                                ;...03
                                                ;...db
                                                ;*imul {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::FactoredScale_Int@33 (line 55)

  0x000002bbebeda42e: mov     eax,ebx           ;...8b
                                                ;...c3

  0x000002bbebeda430: add     rsp,40h           ;...48
                                                ;...83
                                                ;...c4
                                                ;...40

  0x000002bbebeda434: pop     rbp               ;...5d

  0x000002bbebeda435: test    dword ptr [2bbd8590000h],eax
                                                ;...85
                                                ;...05
                                                ;...c5
                                                ;...5b
                                                ;...6b
                                                ;...ec
                                                ;   {poll_return}
  0x000002bbebeda43b: ret                       ;...c3

This is a special case: data is usually dynamic and variable, so the loop cannot always be proven to be equivalent to a linear combination of bit shifts. The routine is compiled for all possible parameters, not just statically contrived cases like the one above, so you may never see this assembly in the wild. However, even with random factors, the slow looking loop is aggressively optimised in a way the hand “optimised” code is not:


    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    @Benchmark
    public int Scale_Int_Dynamic(ScaleState state) {
        int value = 0;
        int[] data = state.data;
        int factor = state.randomFactor();
        for (int i = 0; i < data.length; ++i) {
            value += factor * data[i];
        }
        return value;
    }

    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    @Benchmark
    public int FactoredScale_Int_Dynamic(ScaleState state) {
        int value = 0;
        int[] data = state.data;
        int factor = state.randomFactor();
        for (int i = 0; i < data.length; ++i) {
            value += data[i];
        }
        return factor * value;
    }

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
FactoredScale_Int_Dynamic thrpt 1 10 26.100439 0.340069 ops/ms 100000
FactoredScale_Int_Dynamic thrpt 1 10 1.918011 0.297925 ops/ms 1000000
Scale_Int_Dynamic thrpt 1 10 30.219809 2.977389 ops/ms 100000
Scale_Int_Dynamic thrpt 1 10 2.314159 0.378442 ops/ms 1000000

Far from seeking to exploit distributivity to reduce the number of multiplication instructions, it seems to almost embrace the extraneous operations as metadata to drive optimisations. The assembly for Scale_Int_Dynamic confirms this (it shows vectorised multiplication, not shifts, within the loop):

com/openkappa/simd/scale/Scale.Scale_Int_Dynamic(Lcom/openkappa/simd/scale/ScaleState;)I  [0x000001f5ca2fa120, 0x000001f5ca2fa498]  888 bytes
Argument 0 is unknown.RIP: 0x1f5ca2fa120 Code size: 0x00000378
[Entry Point]
[Verified Entry Point]
[Constants]
  # {method} {0x000001f5df561a20} 'Scale_Int_Dynamic' '(Lcom/openkappa/simd/scale/ScaleState;)I' in 'com/openkappa/simd/scale/Scale'
  0x000001f5ca2fa120: int3                      ;...cc

  0x000001f5ca2fa121: nop     word ptr [rax+rax+0h]  ;...66
                                                ;...66
                                                ;...66
                                                ;...0f
                                                ;...1f
                                                ;...84
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001f5ca2fa12c: nop                       ;...66
                                                ;...66
                                                ;...66
                                                ;...90

  0x000001f5ca2fa130: mov     dword ptr [rsp+0ffffffffffff9000h],eax
                                                ;...89
                                                ;...84
                                                ;...24
                                                ;...00
                                                ;...90
                                                ;...ff
                                                ;...ff

  0x000001f5ca2fa137: push    rbp               ;...55

  0x000001f5ca2fa138: sub     rsp,50h           ;...48
                                                ;...83
                                                ;...ec
                                                ;...50

  0x000001f5ca2fa13c: mov     r13,qword ptr [rdx+10h]  ;...4c
                                                ;...8b
                                                ;...6a
                                                ;...10

  0x000001f5ca2fa140: mov     ebx,dword ptr [rdx+18h]  ;...8b
                                                ;...5a
                                                ;...18

  0x000001f5ca2fa143: mov     ebp,dword ptr [rdx+8h]  ;...8b
                                                ;...6a
                                                ;...08

  0x000001f5ca2fa146: mov     r14d,dword ptr [rdx]  ;...44
                                                ;...8b
                                                ;...32

  0x000001f5ca2fa149: mov     rcx,rdx           ;...48
                                                ;...8b
                                                ;...ca

  0x000001f5ca2fa14c: vzeroupper                ;...c5
                                                ;...f8
                                                ;...77

  0x000001f5ca2fa14f: mov     r10,51da8d20h     ;...49
                                                ;...ba
                                                ;...20
                                                ;...8d
                                                ;...da
                                                ;...51
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001f5ca2fa159: call indirect r10         ;...41
                                                ;...ff
                                                ;...d2

  0x000001f5ca2fa15c: mov     r10d,dword ptr [r13+8h]  ;...45
                                                ;...8b
                                                ;...55
                                                ;...08
                                                ; implicit exception: dispatches to 0x000001f5ca2fa461
  0x000001f5ca2fa160: cmp     r10d,0f800016dh   ;...41
                                                ;...81
                                                ;...fa
                                                ;...6d
                                                ;...01
                                                ;...00
                                                ;...f8
                                                ;   {metadata({type array int})}
  0x000001f5ca2fa167: jne     1f5ca2fa445h      ;...0f
                                                ;...85
                                                ;...d8
                                                ;...02
                                                ;...00
                                                ;...00
                                                ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@16 (line 64)

  0x000001f5ca2fa16d: mov     edi,dword ptr [r13+0ch]  ;...41
                                                ;...8b
                                                ;...7d
                                                ;...0c
                                                ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@19 (line 64)

  0x000001f5ca2fa171: cmp     r14d,edi          ;...44
                                                ;...3b
                                                ;...f7

  0x000001f5ca2fa174: jnl     1f5ca2fa411h      ;...0f
                                                ;...8d
                                                ;...97
                                                ;...02
                                                ;...00
                                                ;...00
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@20 (line 64)

  0x000001f5ca2fa17a: mov     r11d,r13d         ;...45
                                                ;...8b
                                                ;...dd

  0x000001f5ca2fa17d: mov     r10d,r14d         ;...45
                                                ;...8b
                                                ;...d6

  0x000001f5ca2fa180: inc     r10d              ;...41
                                                ;...ff
                                                ;...c2

  0x000001f5ca2fa183: shr     r11d,2h           ;...41
                                                ;...c1
                                                ;...eb
                                                ;...02

  0x000001f5ca2fa187: and     r11d,7h           ;...41
                                                ;...83
                                                ;...e3
                                                ;...07

  0x000001f5ca2fa18b: xor     r8d,r8d           ;...45
                                                ;...33
                                                ;...c0

  0x000001f5ca2fa18e: cmp     r10d,r8d          ;...45
                                                ;...3b
                                                ;...d0

  0x000001f5ca2fa191: cmovl   r10d,r8d          ;...45
                                                ;...0f
                                                ;...4c
                                                ;...d0

  0x000001f5ca2fa195: cmp     r10d,edi          ;...44
                                                ;...3b
                                                ;...d7

  0x000001f5ca2fa198: cmovnle r10d,edi          ;...44
                                                ;...0f
                                                ;...4f
                                                ;...d7

  0x000001f5ca2fa19c: add     r11d,r10d         ;...45
                                                ;...03
                                                ;...da

  0x000001f5ca2fa19f: mov     r9d,4h            ;...41
                                                ;...b9
                                                ;...04
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001f5ca2fa1a5: sub     r9d,r11d          ;...45
                                                ;...2b
                                                ;...cb

  0x000001f5ca2fa1a8: and     r9d,7h            ;...41
                                                ;...83
                                                ;...e1
                                                ;...07

  0x000001f5ca2fa1ac: add     r9d,r10d          ;...45
                                                ;...03
                                                ;...ca

  0x000001f5ca2fa1af: cmp     r9d,edi           ;...44
                                                ;...3b
                                                ;...cf

  0x000001f5ca2fa1b2: cmovnle r9d,edi           ;...44
                                                ;...0f
                                                ;...4f
                                                ;...cf
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@29 (line 65)

  0x000001f5ca2fa1b6: cmp     r14d,edi          ;...44
                                                ;...3b
                                                ;...f7

  0x000001f5ca2fa1b9: jnb     1f5ca2fa422h      ;...0f
                                                ;...83
                                                ;...63
                                                ;...02
                                                ;...00
                                                ;...00

  0x000001f5ca2fa1bf: mov     r11d,ebp          ;...44
                                                ;...8b
                                                ;...dd

  0x000001f5ca2fa1c2: imul    r11d,dword ptr [r13+r14*4+10h]
                                                ;...47
                                                ;...0f
                                                ;...af
                                                ;...5c
                                                ;...b5
                                                ;...10

  0x000001f5ca2fa1c8: add     ebx,r11d          ;...41
                                                ;...03
                                                ;...db
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@31 (line 65)

  0x000001f5ca2fa1cb: inc     r14d              ;...41
                                                ;...ff
                                                ;...c6
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@33 (line 64)

  0x000001f5ca2fa1ce: cmp     r14d,r9d          ;...45
                                                ;...3b
                                                ;...f1

  0x000001f5ca2fa1d1: jl      1f5ca2fa1b6h      ;...7c
                                                ;...e3
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@20 (line 64)

  0x000001f5ca2fa1d3: mov     r9d,edi           ;...44
                                                ;...8b
                                                ;...cf

  0x000001f5ca2fa1d6: add     r9d,0ffffffc1h    ;...41
                                                ;...83
                                                ;...c1
                                                ;...c1

  0x000001f5ca2fa1da: mov     r8d,80000000h     ;...41
                                                ;...b8
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...80

  0x000001f5ca2fa1e0: cmp     edi,r9d           ;...41
                                                ;...3b
                                                ;...f9

  0x000001f5ca2fa1e3: cmovl   r9d,r8d           ;...45
                                                ;...0f
                                                ;...4c
                                                ;...c8

  0x000001f5ca2fa1e7: cmp     r14d,r9d          ;...45
                                                ;...3b
                                                ;...f1

  0x000001f5ca2fa1ea: jnl     1f5ca2fa3f0h      ;...0f
                                                ;...8d
                                                ;...00
                                                ;...02
                                                ;...00
                                                ;...00

  0x000001f5ca2fa1f0: vmovd   xmm2,ebp          ;...c5
                                                ;...f9
                                                ;...6e
                                                ;...d5

  0x000001f5ca2fa1f4: vpshufd xmm2,xmm2,0h      ;...c5
                                                ;...f9
                                                ;...70
                                                ;...d2
                                                ;...00

  0x000001f5ca2fa1f9: vinserti128 ymm2,ymm2,xmm2,1h  ;...c4
                                                ;...e3
                                                ;...6d
                                                ;...38
                                                ;...d2
                                                ;...01

  0x000001f5ca2fa1ff: nop                       ;...90
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@29 (line 65)

  0x000001f5ca2fa200: vmovdqu ymm0,ymmword ptr [r13+r14*4+10h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...44
                                                ;...b5
                                                ;...10

  0x000001f5ca2fa207: vpmulld ymm11,ymm0,ymm2   ;...c4
                                                ;...62
                                                ;...7d
                                                ;...40
                                                ;...da

  0x000001f5ca2fa20c: movsxd  r10,r14d          ;...4d
                                                ;...63
                                                ;...d6

  0x000001f5ca2fa20f: vmovdqu ymm0,ymmword ptr [r13+r10*4+30h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...44
                                                ;...95
                                                ;...30

  0x000001f5ca2fa216: vmovdqu ymm1,ymmword ptr [r13+r10*4+0f0h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...8c
                                                ;...95
                                                ;...f0
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001f5ca2fa220: vmovdqu ymm3,ymmword ptr [r13+r10*4+50h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...5c
                                                ;...95
                                                ;...50

  0x000001f5ca2fa227: vmovdqu ymm7,ymmword ptr [r13+r10*4+70h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...7c
                                                ;...95
                                                ;...70

  0x000001f5ca2fa22e: vmovdqu ymm6,ymmword ptr [r13+r10*4+90h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...b4
                                                ;...95
                                                ;...90
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001f5ca2fa238: vmovdqu ymm5,ymmword ptr [r13+r10*4+0b0h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...ac
                                                ;...95
                                                ;...b0
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001f5ca2fa242: vmovdqu ymm4,ymmword ptr [r13+r10*4+0d0h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...a4
                                                ;...95
                                                ;...d0
                                                ;...00
                                                ;...00
                                                ;...00

  0x000001f5ca2fa24c: vpmulld ymm9,ymm0,ymm2    ;...c4
                                                ;...62
                                                ;...7d
                                                ;...40
                                                ;...ca

  0x000001f5ca2fa251: vpmulld ymm4,ymm4,ymm2    ;...c4
                                                ;...e2
                                                ;...5d
                                                ;...40
                                                ;...e2

  0x000001f5ca2fa256: vpmulld ymm5,ymm5,ymm2    ;...c4
                                                ;...e2
                                                ;...55
                                                ;...40
                                                ;...ea

  0x000001f5ca2fa25b: vpmulld ymm6,ymm6,ymm2    ;...c4
                                                ;...e2
                                                ;...4d
                                                ;...40
                                                ;...f2

  0x000001f5ca2fa260: vpmulld ymm8,ymm7,ymm2    ;...c4
                                                ;...62
                                                ;...45
                                                ;...40
                                                ;...c2

  0x000001f5ca2fa265: vpmulld ymm10,ymm3,ymm2   ;...c4
                                                ;...62
                                                ;...65
                                                ;...40
                                                ;...d2

  0x000001f5ca2fa26a: vpmulld ymm3,ymm1,ymm2    ;...c4
                                                ;...e2
                                                ;...75
                                                ;...40
                                                ;...da

  0x000001f5ca2fa26f: vphaddd ymm1,ymm11,ymm11  ;...c4
                                                ;...c2
                                                ;...25
                                                ;...02
                                                ;...cb

  0x000001f5ca2fa274: vphaddd ymm1,ymm1,ymm0    ;...c4
                                                ;...e2
                                                ;...75
                                                ;...02
                                                ;...c8

  0x000001f5ca2fa279: vextracti128 xmm0,ymm1,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c8
                                                ;...01

  0x000001f5ca2fa27f: vpaddd  xmm1,xmm1,xmm0    ;...c5
                                                ;...f1
                                                ;...fe
                                                ;...c8

  0x000001f5ca2fa283: vmovd   xmm0,ebx          ;...c5
                                                ;...f9
                                                ;...6e
                                                ;...c3

  0x000001f5ca2fa287: vpaddd  xmm0,xmm0,xmm1    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c1

  0x000001f5ca2fa28b: vmovd   r10d,xmm0         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...c2

  0x000001f5ca2fa290: vphaddd ymm1,ymm9,ymm9    ;...c4
                                                ;...c2
                                                ;...35
                                                ;...02
                                                ;...c9

  0x000001f5ca2fa295: vphaddd ymm1,ymm1,ymm0    ;...c4
                                                ;...e2
                                                ;...75
                                                ;...02
                                                ;...c8

  0x000001f5ca2fa29a: vextracti128 xmm0,ymm1,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c8
                                                ;...01

  0x000001f5ca2fa2a0: vpaddd  xmm1,xmm1,xmm0    ;...c5
                                                ;...f1
                                                ;...fe
                                                ;...c8

  0x000001f5ca2fa2a4: vmovd   xmm0,r10d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...c2

  0x000001f5ca2fa2a9: vpaddd  xmm0,xmm0,xmm1    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c1

  0x000001f5ca2fa2ad: vmovd   r11d,xmm0         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...c3

  0x000001f5ca2fa2b2: vphaddd ymm0,ymm10,ymm10  ;...c4
                                                ;...c2
                                                ;...2d
                                                ;...02
                                                ;...c2

  0x000001f5ca2fa2b7: vphaddd ymm0,ymm0,ymm1    ;...c4
                                                ;...e2
                                                ;...7d
                                                ;...02
                                                ;...c1

  0x000001f5ca2fa2bc: vextracti128 xmm1,ymm0,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c1
                                                ;...01

  0x000001f5ca2fa2c2: vpaddd  xmm0,xmm0,xmm1    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c1

  0x000001f5ca2fa2c6: vmovd   xmm1,r11d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...cb

  0x000001f5ca2fa2cb: vpaddd  xmm1,xmm1,xmm0    ;...c5
                                                ;...f1
                                                ;...fe
                                                ;...c8

  0x000001f5ca2fa2cf: vmovd   r10d,xmm1         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...ca

  0x000001f5ca2fa2d4: vphaddd ymm1,ymm8,ymm8    ;...c4
                                                ;...c2
                                                ;...3d
                                                ;...02
                                                ;...c8

  0x000001f5ca2fa2d9: vphaddd ymm1,ymm1,ymm0    ;...c4
                                                ;...e2
                                                ;...75
                                                ;...02
                                                ;...c8

  0x000001f5ca2fa2de: vextracti128 xmm0,ymm1,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c8
                                                ;...01

  0x000001f5ca2fa2e4: vpaddd  xmm1,xmm1,xmm0    ;...c5
                                                ;...f1
                                                ;...fe
                                                ;...c8

  0x000001f5ca2fa2e8: vmovd   xmm0,r10d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...c2

  0x000001f5ca2fa2ed: vpaddd  xmm0,xmm0,xmm1    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c1

  0x000001f5ca2fa2f1: vmovd   r11d,xmm0         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...c3

  0x000001f5ca2fa2f6: vphaddd ymm0,ymm6,ymm6    ;...c4
                                                ;...e2
                                                ;...4d
                                                ;...02
                                                ;...c6

  0x000001f5ca2fa2fb: vphaddd ymm0,ymm0,ymm1    ;...c4
                                                ;...e2
                                                ;...7d
                                                ;...02
                                                ;...c1

  0x000001f5ca2fa300: vextracti128 xmm1,ymm0,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c1
                                                ;...01

  0x000001f5ca2fa306: vpaddd  xmm0,xmm0,xmm1    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c1

  0x000001f5ca2fa30a: vmovd   xmm1,r11d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...cb

  0x000001f5ca2fa30f: vpaddd  xmm1,xmm1,xmm0    ;...c5
                                                ;...f1
                                                ;...fe
                                                ;...c8

  0x000001f5ca2fa313: vmovd   r10d,xmm1         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...ca

  0x000001f5ca2fa318: vphaddd ymm1,ymm5,ymm5    ;...c4
                                                ;...e2
                                                ;...55
                                                ;...02
                                                ;...cd

  0x000001f5ca2fa31d: vphaddd ymm1,ymm1,ymm0    ;...c4
                                                ;...e2
                                                ;...75
                                                ;...02
                                                ;...c8

  0x000001f5ca2fa322: vextracti128 xmm0,ymm1,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c8
                                                ;...01

  0x000001f5ca2fa328: vpaddd  xmm1,xmm1,xmm0    ;...c5
                                                ;...f1
                                                ;...fe
                                                ;...c8

  0x000001f5ca2fa32c: vmovd   xmm0,r10d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...c2

  0x000001f5ca2fa331: vpaddd  xmm0,xmm0,xmm1    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c1

  0x000001f5ca2fa335: vmovd   r11d,xmm0         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...c3

  0x000001f5ca2fa33a: vphaddd ymm0,ymm4,ymm4    ;...c4
                                                ;...e2
                                                ;...5d
                                                ;...02
                                                ;...c4

  0x000001f5ca2fa33f: vphaddd ymm0,ymm0,ymm1    ;...c4
                                                ;...e2
                                                ;...7d
                                                ;...02
                                                ;...c1

  0x000001f5ca2fa344: vextracti128 xmm1,ymm0,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...c1
                                                ;...01

  0x000001f5ca2fa34a: vpaddd  xmm0,xmm0,xmm1    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c1

  0x000001f5ca2fa34e: vmovd   xmm1,r11d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...cb

  0x000001f5ca2fa353: vpaddd  xmm1,xmm1,xmm0    ;...c5
                                                ;...f1
                                                ;...fe
                                                ;...c8

  0x000001f5ca2fa357: vmovd   r10d,xmm1         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...7e
                                                ;...ca

  0x000001f5ca2fa35c: vphaddd ymm1,ymm3,ymm3    ;...c4
                                                ;...e2
                                                ;...65
                                                ;...02
                                                ;...cb

  0x000001f5ca2fa361: vphaddd ymm1,ymm1,ymm7    ;...c4
                                                ;...e2
                                                ;...75
                                                ;...02
                                                ;...cf

  0x000001f5ca2fa366: vextracti128 xmm7,ymm1,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...cf
                                                ;...01

  0x000001f5ca2fa36c: vpaddd  xmm1,xmm1,xmm7    ;...c5
                                                ;...f1
                                                ;...fe
                                                ;...cf

  0x000001f5ca2fa370: vmovd   xmm7,r10d         ;...c4
                                                ;...c1
                                                ;...79
                                                ;...6e
                                                ;...fa

  0x000001f5ca2fa375: vpaddd  xmm7,xmm7,xmm1    ;...c5
                                                ;...c1
                                                ;...fe
                                                ;...f9

  0x000001f5ca2fa379: vmovd   ebx,xmm7          ;...c5
                                                ;...f9
                                                ;...7e
                                                ;...fb
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@31 (line 65)

  0x000001f5ca2fa37d: add     r14d,40h          ;...41
                                                ;...83
                                                ;...c6
                                                ;...40
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@33 (line 64)

  0x000001f5ca2fa381: cmp     r14d,r9d          ;...45
                                                ;...3b
                                                ;...f1

  0x000001f5ca2fa384: jl      1f5ca2fa200h      ;...0f
                                                ;...8c
                                                ;...76
                                                ;...fe
                                                ;...ff
                                                ;...ff
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@20 (line 64)

  0x000001f5ca2fa38a: mov     r11d,edi          ;...44
                                                ;...8b
                                                ;...df

  0x000001f5ca2fa38d: add     r11d,0fffffff9h   ;...41
                                                ;...83
                                                ;...c3
                                                ;...f9

  0x000001f5ca2fa391: cmp     edi,r11d          ;...41
                                                ;...3b
                                                ;...fb

  0x000001f5ca2fa394: cmovl   r11d,r8d          ;...45
                                                ;...0f
                                                ;...4c
                                                ;...d8

  0x000001f5ca2fa398: cmp     r14d,r11d         ;...45
                                                ;...3b
                                                ;...f3

  0x000001f5ca2fa39b: jnl     1f5ca2fa3d5h      ;...7d
                                                ;...38

  0x000001f5ca2fa39d: nop                       ;...66
                                                ;...66
                                                ;...90
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@29 (line 65)

  0x000001f5ca2fa3a0: vmovdqu ymm0,ymmword ptr [r13+r14*4+10h]
                                                ;...c4
                                                ;...81
                                                ;...7e
                                                ;...6f
                                                ;...44
                                                ;...b5
                                                ;...10

  0x000001f5ca2fa3a7: vpmulld ymm1,ymm0,ymm2    ;...c4
                                                ;...e2
                                                ;...7d
                                                ;...40
                                                ;...ca

  0x000001f5ca2fa3ac: vphaddd ymm3,ymm1,ymm1    ;...c4
                                                ;...e2
                                                ;...75
                                                ;...02
                                                ;...d9

  0x000001f5ca2fa3b1: vphaddd ymm3,ymm3,ymm0    ;...c4
                                                ;...e2
                                                ;...65
                                                ;...02
                                                ;...d8

  0x000001f5ca2fa3b6: vextracti128 xmm0,ymm3,1h  ;...c4
                                                ;...e3
                                                ;...7d
                                                ;...39
                                                ;...d8
                                                ;...01

  0x000001f5ca2fa3bc: vpaddd  xmm3,xmm3,xmm0    ;...c5
                                                ;...e1
                                                ;...fe
                                                ;...d8

  0x000001f5ca2fa3c0: vmovd   xmm0,ebx          ;...c5
                                                ;...f9
                                                ;...6e
                                                ;...c3

  0x000001f5ca2fa3c4: vpaddd  xmm0,xmm0,xmm3    ;...c5
                                                ;...f9
                                                ;...fe
                                                ;...c3

  0x000001f5ca2fa3c8: vmovd   ebx,xmm0          ;...c5
                                                ;...f9
                                                ;...7e
                                                ;...c3
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@31 (line 65)

  0x000001f5ca2fa3cc: add     r14d,8h           ;...41
                                                ;...83
                                                ;...c6
                                                ;...08
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@33 (line 64)

  0x000001f5ca2fa3d0: cmp     r14d,r11d         ;...45
                                                ;...3b
                                                ;...f3

  0x000001f5ca2fa3d3: jl      1f5ca2fa3a0h      ;...7c
                                                ;...cb
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@20 (line 64)

  0x000001f5ca2fa3d5: cmp     r14d,edi          ;...44
                                                ;...3b
                                                ;...f7

  0x000001f5ca2fa3d8: jnl     1f5ca2fa3f0h      ;...7d
                                                ;...16

  0x000001f5ca2fa3da: nop                       ;...66
                                                ;...90
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@29 (line 65)

  0x000001f5ca2fa3dc: mov     r10d,ebp          ;...44
                                                ;...8b
                                                ;...d5

  0x000001f5ca2fa3df: imul    r10d,dword ptr [r13+r14*4+10h]
                                                ;...47
                                                ;...0f
                                                ;...af
                                                ;...54
                                                ;...b5
                                                ;...10

  0x000001f5ca2fa3e5: add     ebx,r10d          ;...41
                                                ;...03
                                                ;...da
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@31 (line 65)

  0x000001f5ca2fa3e8: inc     r14d              ;...41
                                                ;...ff
                                                ;...c6
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@33 (line 64)

  0x000001f5ca2fa3eb: cmp     r14d,edi          ;...44
                                                ;...3b
                                                ;...f7

  0x000001f5ca2fa3ee: jl      1f5ca2fa3dch      ;...7c
                                                ;...ec
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@20 (line 64)

  0x000001f5ca2fa3f0: cmp     r14d,edi          ;...44
                                                ;...3b
                                                ;...f7

  0x000001f5ca2fa3f3: jnl     1f5ca2fa411h      ;...7d
                                                ;...1c

  0x000001f5ca2fa3f5: nop                       ;...66
                                                ;...66
                                                ;...90
                                                ;*iaload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@29 (line 65)

  0x000001f5ca2fa3f8: cmp     r14d,edi          ;...44
                                                ;...3b
                                                ;...f7

  0x000001f5ca2fa3fb: jnb     1f5ca2fa428h      ;...73
                                                ;...2b

  0x000001f5ca2fa3fd: mov     r11d,ebp          ;...44
                                                ;...8b
                                                ;...dd

  0x000001f5ca2fa400: imul    r11d,dword ptr [r13+r14*4+10h]
                                                ;...47
                                                ;...0f
                                                ;...af
                                                ;...5c
                                                ;...b5
                                                ;...10

  0x000001f5ca2fa406: add     ebx,r11d          ;...41
                                                ;...03
                                                ;...db
                                                ;*iadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@31 (line 65)

  0x000001f5ca2fa409: inc     r14d              ;...41
                                                ;...ff
                                                ;...c6
                                                ;*iinc {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@33 (line 64)

  0x000001f5ca2fa40c: cmp     r14d,edi          ;...44
                                                ;...3b
                                                ;...f7

  0x000001f5ca2fa40f: jl      1f5ca2fa3f8h      ;...7c
                                                ;...e7
                                                ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.scale.Scale::Scale_Int_Dynamic@16 (line 64)

  0x000001f5ca2fa411: mov     eax,ebx           ;...8b
                                                ;...c3

  0x000001f5ca2fa413: vzeroupper                ;...c5
                                                ;...f8
                                                ;...77

  0x000001f5ca2fa416: add     rsp,50h           ;...48
                                                ;...83
                                                ;...c4
                                                ;...50

  0x000001f5ca2fa41a: pop     rbp               ;...5d

  0x000001f5ca2fa41b: test    dword ptr [1f5b68c0000h],eax
                                                ;...85
                                                ;...05
                                                ;...df
                                                ;...5b
                                                ;...5c
                                                ;...ec
                                                ;   {poll_return}
  0x000001f5ca2fa421: ret                       ;...c3

There are two lessons to be learnt here. The first is that what you see is not what you get. The second is about the correctness of asymptotic analysis. If hierarchical cache renders asymptotic analysis bullshit (linear time but cache friendly algorithms can, and do, outperform logarithmic algorithms with cache misses), optimising compilers render the field practically irrelevant.

Zeroing Negative Values in Arrays Efficiently

Replacing negatives with zeroes in large arrays of values is a primitive function of several complex financial risk measures, including potential future exposure (PFE) and the liquidity coverage ratio (LCR). While this is not an interesting operation by any stretch of the imagination, it is useful and there is significant benefit in its performance. This is an operation that can be computed very efficiently using the instruction VMAXPD. For Intel Xeon processors, this instruction requires half a cycle to calculate and has a latency (how long before another instruction can use its result) of four cycles. There is currently no way to trick Java into using this instruction for this simple operation, though there is a placeholder implementation on the current DoubleVector prototype in Project Panama which may do so.

C++ Intel Intrinsics

It’s possible to target instructions from different processor vendors, in my case Intel, by using intrinsic functions which expose instructions as high level functions. The code looks incredibly ugly but it works. Here is a C++ function for 256 bit ymm registers:


void zero_negatives(const double* source, double* target, const size_t length) {
  for (size_t i = 0; i + 3 < length; i += 4) {
    __m256d vector = _mm256_load_pd(source + i);
    __m256d zeroed = _mm256_max_pd(vector, _mm256_setzero_pd());
    _mm256_storeu_pd(target + i, zeroed);
  }
}

The function loads doubles into 256 bit vectors, within each vector replaces the negative values with zero, and writes them back into an array. It generates the following assembly code (which, incidentally, is less of a shit show to access than in Java):


void zero_negatives(const double* source, double* target, const size_t length) {
00007FF746EE5110  mov         qword ptr [rsp+18h],r8  
00007FF746EE5115  mov         qword ptr [rsp+10h],rdx  
00007FF746EE511A  mov         qword ptr [rsp+8],rcx  
00007FF746EE511F  push        r13  
00007FF746EE5121  push        rbp  
00007FF746EE5122  push        rdi  
00007FF746EE5123  sub         rsp,250h  
00007FF746EE512A  mov         r13,rsp  
00007FF746EE512D  lea         rbp,[rsp+20h]  
00007FF746EE5132  and         rbp,0FFFFFFFFFFFFFFE0h  
00007FF746EE5136  mov         rdi,rsp  
00007FF746EE5139  mov         ecx,94h  
00007FF746EE513E  mov         eax,0CCCCCCCCh  
00007FF746EE5143  rep stos    dword ptr [rdi]  
00007FF746EE5145  mov         rcx,qword ptr [rsp+278h]  
  for (size_t i = 0; i + 3 < length; i += 4) {
00007FF746EE514D  mov         qword ptr [rbp+8],0  
00007FF746EE5155  jmp         zero_negatives+53h (07FF746EE5163h)  
00007FF746EE5157  mov         rax,qword ptr [rbp+8]  
00007FF746EE515B  add         rax,4  
00007FF746EE515F  mov         qword ptr [rbp+8],rax  
00007FF746EE5163  mov         rax,qword ptr [rbp+8]  
00007FF746EE5167  add         rax,3  
00007FF746EE516B  cmp         rax,qword ptr [length]  
00007FF746EE5172  jae         zero_negatives+0DDh (07FF746EE51EDh)  
    __m256d vector = _mm256_load_pd(source + i);
00007FF746EE5174  mov         rax,qword ptr [source]  
00007FF746EE517B  mov         rcx,qword ptr [rbp+8]  
00007FF746EE517F  lea         rax,[rax+rcx*8]  
00007FF746EE5183  vmovupd     ymm0,ymmword ptr [rax]  
00007FF746EE5187  vmovupd     ymmword ptr [rbp+180h],ymm0  
00007FF746EE518F  vmovupd     ymm0,ymmword ptr [rbp+180h]  
00007FF746EE5197  vmovupd     ymmword ptr [rbp+40h],ymm0  
    __m256d zeroed = _mm256_max_pd(vector, _mm256_setzero_pd());
00007FF746EE519C  vxorpd      xmm0,xmm0,xmm0  
00007FF746EE51A0  vmovupd     ymmword ptr [rbp+200h],ymm0  
00007FF746EE51A8  vmovupd     ymm0,ymmword ptr [rbp+40h]  
00007FF746EE51AD  vmaxpd      ymm0,ymm0,ymmword ptr [rbp+200h]  
00007FF746EE51B5  vmovupd     ymmword ptr [rbp+1C0h],ymm0  
00007FF746EE51BD  vmovupd     ymm0,ymmword ptr [rbp+1C0h]  
00007FF746EE51C5  vmovupd     ymmword ptr [rbp+80h],ymm0  
    _mm256_storeu_pd(target + i, zeroed);
00007FF746EE51CD  mov         rax,qword ptr [target]  
00007FF746EE51D4  mov         rcx,qword ptr [rbp+8]  
00007FF746EE51D8  lea         rax,[rax+rcx*8]  
00007FF746EE51DC  vmovupd     ymm0,ymmword ptr [rbp+80h]  
00007FF746EE51E4  vmovupd     ymmword ptr [rax],ymm0  
  }
00007FF746EE51E8  jmp         zero_negatives+47h (07FF746EE5157h)  
}
00007FF746EE51ED  lea         rsp,[r13+250h]  
00007FF746EE51F4  pop         rdi  
00007FF746EE51F5  pop         rbp  
00007FF746EE51F6  pop         r13  
00007FF746EE51F8  ret    

This code is noticeably fast. I measured the throughput averaged over 1000 iterations, with an array of 10 million doubles (800MB) uniformly distributed between +/- 1E7, to quantify the throughput in GB/s and iterations/s. This code does between 4.5 and 5 iterations per second, which translates to processing approximately 4GB/s. This seems high, and since I am unaware of best practices in C++, if the measurement is flawed, I would gratefully be educated in the comments.


void benchmark() {
  const size_t length = 1E8;
  double* values = new double[length];
  fill_array(values, length);
  double* zeroed = new double[length];
  auto start = std::chrono::high_resolution_clock::now();
  int iterations = 1000;
  for (int i = 0; i < iterations; ++i) {
    zero_negatives(values, zeroed, length);
  }
  auto end = std::chrono::high_resolution_clock::now();
  auto nanos = std::chrono::duration_cast<std::chrono::nanoseconds>(end - start).count();
  double thrpt_s = (iterations * 1E9) / nanos;
  double thrpt_gbps = (thrpt_s * sizeof(double) * length) / 1E9;
  std::cout << thrpt_s << "/s" << std::endl;
  std::cout << thrpt_gbps << "GB/s" << std::endl;
  delete[] values;
  delete[] zeroed;
}

While I am sure there are various ways an expert could tweak this for performance, this code can’t get much faster unless there are 512 bit zmm registers available, in which case it would be wasteful. While the code looks virtually the same for AVX512 (just replace “256” with “512”), portability and efficiency are at odds. Handling the mess of detecting the best instruction set for the deployed architecture is the main reason for using Java in performance sensitive (but not critical) applications. But this is not the code the JVM generates.

Java Auto-Vectorisation (Play Your Cards Right)

There is currently no abstraction modelling vectorisation in Java. The only access available is if the compiler engineers implement an intrinsic, or auto-vectorisation, which will try, and sometimes succeed admirably, to translate your code to a good vector implementation. There is currently a prototype project for explicit vectorisation in Project Panama. There are a few ways to skin this cat, and it’s worth looking at the code they generate and the throughput available from each approach.

There is a choice between copying the array and zeroing out the negatives, and allocating a new array and only writing the non-negative values. There is another choice between an if statement and branchless code using Math.max. This results in the following four implementations which I measure on comparable data to the C++ benchmark (10 million doubles, normally distributed with mean zero). To be fair to the Java code, as in the C++ benchmarks, the cost of allocation is isolated by writing into an array pre-allocated once per benchmark. This penalises the approaches where the array is copied first and then zeroed wherever the value is negative. The code is online at github.


    @Benchmark
    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    public double[] BranchyCopyAndMask(ArrayWithNegatives state) {
        double[] data = state.data;
        double[] result = state.target;
        System.arraycopy(data, 0, result, 0, data.length);
        for (int i = 0; i < result.length; ++i) {
            if (result[i] < 0D) {
                result[i] = 0D;
            }
        }
        return result;
    }

    @Benchmark
    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    public double[] BranchyNewArray(ArrayWithNegatives state) {
        double[] data = state.data;
        double[] result = state.target;
        for (int i = 0; i < result.length; ++i) {
            result[i] = data[i] < 0D ? 0D : data[i];
        }
        return result;
    }

    @Benchmark
    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    public double[] NewArray(ArrayWithNegatives state) {
        double[] data = state.data;
        double[] result = state.target;
        for (int i = 0; i < result.length; ++i) {
            result[i] = Math.max(data[i], 0D);
        }
        return result;
    }

    @Benchmark
    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    public double[] CopyAndMask(ArrayWithNegatives state) {
        double[] data = state.data;
        double[] result = state.target;
        System.arraycopy(data, 0, result, 0, data.length);
        for (int i = 0; i < result.length; ++i) {
            result[i] = Math.max(result[i], 0D);
        }
        return result;
    }

None of these implementations comes close to the native code above. The best implementation performs 1.8 iterations per second which equates to processing approximately 1.4GB/s, vastly inferior to the 4GB/s achieved with Intel intrinsics. The results are below:

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit
BranchyCopyAndMask thrpt 1 10 1.314845 0.061662 ops/s
BranchyNewArray thrpt 1 10 1.802673 0.061835 ops/s
CopyAndMask thrpt 1 10 1.146630 0.018903 ops/s
NewArray thrpt 1 10 1.357020 0.116481 ops/s

As an aside, there is a very interesting observation to make, worthy of its own post: if the array consists only of positive values, the “branchy” implementations run very well, at speeds comparable to the zero_negatives (when it ran with 50% negatives). The ratio of branch hits to misses is an orthogonal explanatory variable, and the input data, while I often don’t think about it enough, is very important.

I only looked at the assembly emitted for the fastest version (BranchyNewArray) and it doesn’t look anything like zero_negatives, though it does use some vectorisation – as pointed out by Daniel Lemire in the comments, this code has probably not been vectorised and is probably using SSE2 (indeed only quad words are loaded into 128 bit registers):

com/openkappa/simd/positive/PositiveValues.BranchyNewArray(Lcom/openkappa/simd/positive/ArrayWithNegatives;)[D  [0x000002ae309c3ce0, 0x000002ae309c3ff8]  792 bytes
Argument 0 is unknown.RIP: 0x2ae309c3ce0 Code size: 0x00000318
[Entry Point]
[Constants]
  # {method} {0x000002ae4d13ec70} 'BranchyNewArray' '(Lcom/openkappa/simd/positive/ArrayWithNegatives;)[D' in 'com/openkappa/simd/positive/PositiveValues'
  0x000002ae309c3ce0: mov     r10d,dword ptr [rdx+8h]  ;...44
                                                ;...8b
                                                ;...52
                                                ;...08

  0x000002ae309c3ce4: shl     r10,3h            ;...49
                                                ;...c1
                                                ;...e2
                                                ;...03

  0x000002ae309c3ce8: cmp     r10,rax           ;...4c
                                                ;...3b
                                                ;...d0

  0x000002ae309c3ceb: jne     2ae3042c200h      ;...0f
                                                ;...85
                                                ;...0f
                                                ;...85
                                                ;...a6
                                                ;...ff
                                                ;   {runtime_call ic_miss_stub}
  0x000002ae309c3cf1: nop     word ptr [rax+rax+0h]  ;...66
                                                ;...66
                                                ;...66
                                                ;...0f
                                                ;...1f
                                                ;...84
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3cfc: nop                       ;...66
                                                ;...66
                                                ;...66
                                                ;...90

[Verified Entry Point]
  0x000002ae309c3d00: mov     dword ptr [rsp+0ffffffffffff9000h],eax
                                                ;...89
                                                ;...84
                                                ;...24
                                                ;...00
                                                ;...90
                                                ;...ff
                                                ;...ff

  0x000002ae309c3d07: push    rbp               ;...55

  0x000002ae309c3d08: sub     rsp,60h           ;...48
                                                ;...83
                                                ;...ec
                                                ;...60

  0x000002ae309c3d0c: mov     rcx,2ae4d163880h  ;...48
                                                ;...b9
                                                ;...80
                                                ;...38
                                                ;...16
                                                ;...4d
                                                ;...ae
                                                ;...02
                                                ;...00
                                                ;...00
                                                ;   {metadata(method data for {method} {0x000002ae4d13ec70} 'BranchyNewArray' '(Lcom/openkappa/simd/positive/ArrayWithNegatives;)[D' in 'com/openkappa/simd/positive/PositiveValues')}
  0x000002ae309c3d16: mov     esi,dword ptr [rcx+0fch]
                                                ;...8b
                                                ;...b1
                                                ;...fc
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3d1c: add     esi,8h            ;...83
                                                ;...c6
                                                ;...08

  0x000002ae309c3d1f: mov     dword ptr [rcx+0fch],esi
                                                ;...89
                                                ;...b1
                                                ;...fc
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3d25: and     esi,1ff8h         ;...81
                                                ;...e6
                                                ;...f8
                                                ;...1f
                                                ;...00
                                                ;...00

  0x000002ae309c3d2b: cmp     esi,0h            ;...83
                                                ;...fe
                                                ;...00

  0x000002ae309c3d2e: je      2ae309c3ec1h      ;...0f
                                                ;...84
                                                ;...8d
                                                ;...01
                                                ;...00
                                                ;...00
                                                ;*aload_1 {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@0 (line 29)

  0x000002ae309c3d34: mov     edx,dword ptr [r8+0ch]  ;...41
                                                ;...8b
                                                ;...50
                                                ;...0c
                                                ; implicit exception: dispatches to 0x000002ae309c3ee2
  0x000002ae309c3d38: shl     rdx,3h            ;...48
                                                ;...c1
                                                ;...e2
                                                ;...03
                                                ;*getfield data {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@1 (line 29)

  0x000002ae309c3d3c: mov     ecx,dword ptr [r8+10h]  ;...41
                                                ;...8b
                                                ;...48
                                                ;...10

  0x000002ae309c3d40: shl     rcx,3h            ;...48
                                                ;...c1
                                                ;...e1
                                                ;...03
                                                ;*getfield target {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@6 (line 30)

  0x000002ae309c3d44: mov     esi,0h            ;...be
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3d49: jmp     2ae309c3e27h      ;...e9
                                                ;...d9
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;*iload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@13 (line 31)

  0x000002ae309c3d4e: nop                       ;...66
                                                ;...90

  0x000002ae309c3d50: movsxd  rax,esi           ;...48
                                                ;...63
                                                ;...c6

  0x000002ae309c3d53: cmp     esi,dword ptr [rdx+0ch]  ;...3b
                                                ;...72
                                                ;...0c
                                                ; implicit exception: dispatches to 0x000002ae309c3ee7
  0x000002ae309c3d56: jnb     2ae309c3ef1h      ;...0f
                                                ;...83
                                                ;...95
                                                ;...01
                                                ;...00
                                                ;...00

  0x000002ae309c3d5c: vmovsd  xmm0,qword ptr [rdx+rax*8+10h]
                                                ;...c5
                                                ;...fb
                                                ;...10
                                                ;...44
                                                ;...c2
                                                ;...10
                                                ;*daload {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@26 (line 32)

  0x000002ae309c3d62: vxorpd  xmm1,xmm1,xmm1    ;...c5
                                                ;...f1
                                                ;...57
                                                ;...c9

  0x000002ae309c3d66: vucomisd xmm0,xmm1        ;...c5
                                                ;...f9
                                                ;...2e
                                                ;...c1

  0x000002ae309c3d6a: mov     eax,1h            ;...b8
                                                ;...01
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3d6f: jp      2ae309c3d88h      ;...0f
                                                ;...8a
                                                ;...13
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3d75: jnbe    2ae309c3d88h      ;...0f
                                                ;...87
                                                ;...0d
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3d7b: mov     eax,0h            ;...b8
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3d80: je      2ae309c3d88h      ;...0f
                                                ;...84
                                                ;...02
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3d86: dec     eax               ;...ff
                                                ;...c8
                                                ;*dcmpg {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@28 (line 32)

  0x000002ae309c3d88: cmp     eax,0h            ;...83
                                                ;...f8
                                                ;...00

  0x000002ae309c3d8b: mov     rax,2ae4d163880h  ;...48
                                                ;...b8
                                                ;...80
                                                ;...38
                                                ;...16
                                                ;...4d
                                                ;...ae
                                                ;...02
                                                ;...00
                                                ;...00
                                                ;   {metadata(method data for {method} {0x000002ae4d13ec70} 'BranchyNewArray' '(Lcom/openkappa/simd/positive/ArrayWithNegatives;)[D' in 'com/openkappa/simd/positive/PositiveValues')}
  0x000002ae309c3d95: mov     rdi,158h          ;...48
                                                ;...bf
                                                ;...58
                                                ;...01
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3d9f: jnl     2ae309c3dafh      ;...0f
                                                ;...8d
                                                ;...0a
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3da5: mov     rdi,168h          ;...48
                                                ;...bf
                                                ;...68
                                                ;...01
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3daf: mov     rbx,qword ptr [rax+rdi]  ;...48
                                                ;...8b
                                                ;...1c
                                                ;...38

  0x000002ae309c3db3: lea     rbx,[rbx+1h]      ;...48
                                                ;...8d
                                                ;...5b
                                                ;...01

  0x000002ae309c3db7: mov     qword ptr [rax+rdi],rbx  ;...48
                                                ;...89
                                                ;...1c
                                                ;...38

  0x000002ae309c3dbb: jnl     2ae309c3dd5h      ;...0f
                                                ;...8d
                                                ;...14
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;*ifge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@29 (line 32)

  0x000002ae309c3dc1: mov     rax,2ae4d163880h  ;...48
                                                ;...b8
                                                ;...80
                                                ;...38
                                                ;...16
                                                ;...4d
                                                ;...ae
                                                ;...02
                                                ;...00
                                                ;...00
                                                ;   {metadata(method data for {method} {0x000002ae4d13ec70} 'BranchyNewArray' '(Lcom/openkappa/simd/positive/ArrayWithNegatives;)[D' in 'com/openkappa/simd/positive/PositiveValues')}
  0x000002ae309c3dcb: inc     dword ptr [rax+178h]  ;...ff
                                                ;...80
                                                ;...78
                                                ;...01
                                                ;...00
                                                ;...00

  0x000002ae309c3dd1: vxorpd  xmm0,xmm0,xmm0    ;...c5
                                                ;...f9
                                                ;...57
                                                ;...c0
                                                ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@33 (line 32)

  0x000002ae309c3dd5: movsxd  rax,esi           ;...48
                                                ;...63
                                                ;...c6

  0x000002ae309c3dd8: cmp     esi,dword ptr [rcx+0ch]  ;...3b
                                                ;...71
                                                ;...0c

  0x000002ae309c3ddb: jnb     2ae309c3efah      ;...0f
                                                ;...83
                                                ;...19
                                                ;...01
                                                ;...00
                                                ;...00

  0x000002ae309c3de1: vmovsd  qword ptr [rcx+rax*8+10h],xmm0
                                                ;...c5
                                                ;...fb
                                                ;...11
                                                ;...44
                                                ;...c1
                                                ;...10
                                                ;*dastore {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@40 (line 32)

  0x000002ae309c3de7: inc     esi               ;...ff
                                                ;...c6

  0x000002ae309c3de9: mov     rax,2ae4d163880h  ;...48
                                                ;...b8
                                                ;...80
                                                ;...38
                                                ;...16
                                                ;...4d
                                                ;...ae
                                                ;...02
                                                ;...00
                                                ;...00
                                                ;   {metadata(method data for {method} {0x000002ae4d13ec70} 'BranchyNewArray' '(Lcom/openkappa/simd/positive/ArrayWithNegatives;)[D' in 'com/openkappa/simd/positive/PositiveValues')}
  0x000002ae309c3df3: mov     edi,dword ptr [rax+100h]
                                                ;...8b
                                                ;...b8
                                                ;...00
                                                ;...01
                                                ;...00
                                                ;...00

  0x000002ae309c3df9: add     edi,8h            ;...83
                                                ;...c7
                                                ;...08

  0x000002ae309c3dfc: mov     dword ptr [rax+100h],edi
                                                ;...89
                                                ;...b8
                                                ;...00
                                                ;...01
                                                ;...00
                                                ;...00

  0x000002ae309c3e02: and     edi,0fff8h        ;...81
                                                ;...e7
                                                ;...f8
                                                ;...ff
                                                ;...00
                                                ;...00

  0x000002ae309c3e08: cmp     edi,0h            ;...83
                                                ;...ff
                                                ;...00

  0x000002ae309c3e0b: je      2ae309c3f03h      ;...0f
                                                ;...84
                                                ;...f2
                                                ;...00
                                                ;...00
                                                ;...00
                                                ; ImmutableOopMap{rcx=Oop rdx=Oop }
                                                ;*goto {reexecute=1 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@44 (line 31)

  0x000002ae309c3e11: test    dword ptr [2ae24520000h],eax
                                                ;...85
                                                ;...05
                                                ;...e9
                                                ;...c1
                                                ;...b5
                                                ;...f3
                                                ;   {poll}
  0x000002ae309c3e17: mov     rax,2ae4d163880h  ;...48
                                                ;...b8
                                                ;...80
                                                ;...38
                                                ;...16
                                                ;...4d
                                                ;...ae
                                                ;...02
                                                ;...00
                                                ;...00
                                                ;   {metadata(method data for {method} {0x000002ae4d13ec70} 'BranchyNewArray' '(Lcom/openkappa/simd/positive/ArrayWithNegatives;)[D' in 'com/openkappa/simd/positive/PositiveValues')}
  0x000002ae309c3e21: inc     dword ptr [rax+190h]  ;...ff
                                                ;...80
                                                ;...90
                                                ;...01
                                                ;...00
                                                ;...00
                                                ;*goto {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@44 (line 31)

  0x000002ae309c3e27: mov     eax,dword ptr [rcx+0ch]  ;...8b
                                                ;...41
                                                ;...0c
                                                ;*arraylength {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@16 (line 31)
                                                ; implicit exception: dispatches to 0x000002ae309c3f24
  0x000002ae309c3e2a: cmp     esi,eax           ;...3b
                                                ;...f0

  0x000002ae309c3e2c: mov     rax,2ae4d163880h  ;...48
                                                ;...b8
                                                ;...80
                                                ;...38
                                                ;...16
                                                ;...4d
                                                ;...ae
                                                ;...02
                                                ;...00
                                                ;...00
                                                ;   {metadata(method data for {method} {0x000002ae4d13ec70} 'BranchyNewArray' '(Lcom/openkappa/simd/positive/ArrayWithNegatives;)[D' in 'com/openkappa/simd/positive/PositiveValues')}
  0x000002ae309c3e36: mov     rdi,148h          ;...48
                                                ;...bf
                                                ;...48
                                                ;...01
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3e40: jl      2ae309c3e50h      ;...0f
                                                ;...8c
                                                ;...0a
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3e46: mov     rdi,138h          ;...48
                                                ;...bf
                                                ;...38
                                                ;...01
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002ae309c3e50: mov     rbx,qword ptr [rax+rdi]  ;...48
                                                ;...8b
                                                ;...1c
                                                ;...38

  0x000002ae309c3e54: lea     rbx,[rbx+1h]      ;...48
                                                ;...8d
                                                ;...5b
                                                ;...01

  0x000002ae309c3e58: mov     qword ptr [rax+rdi],rbx  ;...48
                                                ;...89
                                                ;...1c
                                                ;...38

  0x000002ae309c3e5c: jl      2ae309c3d50h      ;...0f
                                                ;...8c
                                                ;...ee
                                                ;...fe
                                                ;...ff
                                                ;...ff
                                                ;*if_icmpge {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@17 (line 31)

  0x000002ae309c3e62: mov     rax,rcx           ;...48
                                                ;...8b
                                                ;...c1

  0x000002ae309c3e65: add     rsp,60h           ;...48
                                                ;...83
                                                ;...c4
                                                ;...60

  0x000002ae309c3e69: pop     rbp               ;...5d

  0x000002ae309c3e6a: test    dword ptr [2ae24520000h],eax
                                                ;...85
                                                ;...05
                                                ;...90
                                                ;...c1
                                                ;...b5
                                                ;...f3
                                                ;   {poll_return}
  0x000002ae309c3e70: ret                       ;...c3
                                                ;*areturn {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.simd.positive.PositiveValues::BranchyNewArray@48 (line 34)

I don’t really understand, and haven’t thought about, the intent of the emitted code, but it makes extensive use of the instruction VUCOMISD for comparisons with zero, which has a lower latency but lower throughput than VMAXPD. It would certainly be interesting to see how Project Panama does this. Perhaps this should just be made available as a fail-safe intrinsic like Arrays.mismatch?

Project Panama and Population Count

Project Panama introduces a new interface Vector, where the specialisation for long looks like a promising substrate for an explicitly vectorised bit set. Bit sets are useful for representing composable predicates over data sets. One obvious omission on this interface, required for an adequate implementation of a bit set, is a bit count, otherwise known as population count. Perhaps this is because the vector API aims to generalise across primitive types, whereas population count is only meaningful for integral types. Even so, if Vector can be interpreted as a wider integer, then it would be consistent to add this to the interface. If the method existed, what possible implementation could it have?

In x86, the population count of a 64 bit register is computed by the POPCNT instruction, which is exposed in Java as an intrinsic in Long.bitCount. There is no SIMD equivalent in any extension set until VPOPCNTD/VPOPCNTQ in AVX-512. Very few processors (at the time of writing) support AVX-512, and only the Knights Mill processor supports this extension; there are not even Intel intrinsics exposing these instructions yet.

The algorithm for vectorised population count adopted by the clang compiler is outlined in this paper, which develops on an algorithm designed for 128 bit registers and SSE instructions, presented by Wojciech Muła on his blog in 2008. This approach is shown in the paper to outperform scalar code using POPCNT and 64 bit registers, almost doubling throughput when 256 bit ymm registers are available. The core algorithm (taken from figure 10 in the paper) returns a vector of four 64 bit counts, which can then be added together in a variety of ways to form a population count, proceeds as follows:


// The Muła Function
__m256i count(__m256i v) {
    __m256i lookup = _mm256_setr_epi8(
                 0, 1, 1, 2, 1, 2, 2, 3, 
                 1, 2, 2, 3, 2, 3, 3, 4,
                 0, 1, 1, 2, 1, 2, 2, 3,
                 1, 2, 2, 3, 2, 3, 3, 4);
    __m256i low_mask = _mm256_set1_epi8(0x0f);
    __m256i lo = _mm256_and_si256(v, low_mask);
    __m256i hi = _mm256_and_si256(_mm256_srli_epi32(v, 4), low_mask);
    __m256i popcnt1 = _mm256_shuffle_epi8(lookup, lo);
    __m256i popcnt2 = _mm256_shuffle_epi8(lookup, hi);
    __m256i total = _mm256_add_epi8(popcnt1, popcnt2);
    return _mm256_sad_epu8(total, _mm256_setzero_si256());
}

If you are struggling to read the code above, you are not alone. I haven’t programmed in C++ for several years – it’s amazing how nice the names in civilised languages like Java and python (and even bash) are compared to the black magic above. There is some logic to the naming though: read page 5 of the manual. You can also read an accessible description of some of the functions used in this blog post.

The basic idea starts from storing the population counts for each possible byte value in a lookup table, which can be looked up using bit level parallelism and ultimately added up. For efficiency’s sake, instead of bytes, 4 bit nibbles are used, which is why you only see numbers 0-4 in the lookup table. Various, occasionally obscure, optimisations are applied resulting in the magic numbers at the the top of the function. A large chunk of the paper is devoted to their derivation: if you are interested, go and read the paper – I could not understand the intent of the code at all until reading the paper twice, especially section 2.

The points I find interesting are:

  • This algorithm exists
  • It uses instructions all modern commodity processors have
  • It is fast
  • It is in use

Could this be implemented in the JVM as an intrinsic and exposed on Vector?

Explicit Intent and Even Faster Hash Codes

I wrote a post recently about how disappointed I was that the optimiser couldn’t outsmart some clever Java code for computing hash codes. Well, here’s a faster hash code along the same lines.

The hash code implemented in Arrays.hashCode is a polynomial hash, it applies to any data type with a positional interpretation. It takes the general form \sum_{i=0}^{n}x_{i}31^{n - i} where x_0 = 1. In other words, it’s a dot product of the elements of the array and some powers of 31. Daniel Lemire’s implementation makes it explicit to the optimiser, in a way it won’t otherwise infer, that this operation is data parallel. If it’s really just a dot product it can be made even more obvious at the cost of a loss of flexibility.

Imagine you are processing fixed or limited length strings (VARCHAR(255) or an URL) or coordinates of a space of fixed dimension. Then you could pre-compute the coefficients in an array and write the hash code explicitly as a dot product. Java 9 uses AVX instructions for dot products, so it should be very fast.


public class FixedLengthHashCode {

    private final int[] coefficients;

    public FixedLengthHashCode(int maxLength) {
        this.coefficients = new int[maxLength + 1];
        coefficients[maxLength] = 1;
        for (int i = maxLength - 1; i >= 0; --i) {
            coefficients[i] = 31 * coefficients[i + 1];
        }
    }

    public int hashCode(int[] value) {
        int result = coefficients[0];
        for (int i = 0; i < value.length && i < coefficients.length - 1; ++i) {
            result += coefficients[i + 1] * value[i];
        }
        return result;
    }
}

This is really explicit, unambiguously parallelisable, and the results are remarkable.

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
HashCode.BuiltIn thrpt 1 10 10.323026 0.223614 ops/us 100
HashCode.BuiltIn thrpt 1 10 0.959246 0.038900 ops/us 1000
HashCode.BuiltIn thrpt 1 10 0.096005 0.001836 ops/us 10000
HashCode.FixedLength thrpt 1 10 20.186800 0.297590 ops/us 100
HashCode.FixedLength thrpt 1 10 2.314187 0.082867 ops/us 1000
HashCode.FixedLength thrpt 1 10 0.227090 0.005377 ops/us 10000
HashCode.Unrolled thrpt 1 10 13.250821 0.752609 ops/us 100
HashCode.Unrolled thrpt 1 10 1.503368 0.058200 ops/us 1000
HashCode.Unrolled thrpt 1 10 0.152179 0.003541 ops/us 10000

Modifying the algorithm slightly to support limited variable length arrays degrades performance slightly, but there are seemingly equivalent implementations which do much worse.


public class FixedLengthHashCode {

    private final int[] coefficients;

    public FixedLengthHashCode(int maxLength) {
        this.coefficients = new int[maxLength + 1];
        coefficients[0] = 1;
        for (int i = 1; i >= maxLength; ++i) {
            coefficients[i] = 31 * coefficients[i - 1];
        }
    }

    public int hashCode(int[] value) {
        final int max = value.length;
        int result = coefficients[max];
        for (int i = 0; i < value.length && i < coefficients.length - 1; ++i) {
            result += coefficients[max - i - 1] * value[i];
        }
        return result;
    }
}

Benchmark Mode Threads Samples Score Score Error (99.9%) Unit Param: size
FixedLength thrpt 1 10 19.172574 0.742637 ops/us 100
FixedLength thrpt 1 10 2.233006 0.115285 ops/us 1000
FixedLength thrpt 1 10 0.227451 0.012231 ops/us 10000

The benchmark code is at github.

New Methods in Java 9: Math.fma and Arrays.mismatch

There are two noteworthy new methods in Java 9: Arrays.mismatch and Math.fma.

Arrays.mismatch

This method takes two primitive arrays, and returns the index of the first differing values. This effectively computes the longest common prefix of the two arrays. This is really quite useful, mostly for text processing but also for Bioinformatics (protein sequencing and so on, much more interesting than the sort of thing I work on). Having worked extensively with Apache HBase (where a vast majority of the API involves manipulating byte arrays) I can think of lots of less interesting use cases for this method.

Looking carefully, you can see that the method calls into the internal ArraysSupport utility class, which will try to perform a vectorised mismatch (an intrinsic candidate). Since this will use AVX instructions, this is very fast; much faster than a handwritten loop.

Let’s measure the boost versus a handwritten loop, testing across a range of common prefices and array lengths of byte[].


    @Benchmark
    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    public void Mismatch_Intrinsic(BytePrefixData data, Blackhole bh) {
        bh.consume(Arrays.mismatch(data.data1, data.data2));
    }

    @Benchmark
    @CompilerControl(CompilerControl.Mode.DONT_INLINE)
    public void Mismatch_Handwritten(BytePrefixData data, Blackhole bh) {
        byte[] data1 = data.data1;
        byte[] data2 = data.data2;
        int length = Math.min(data1.length, data2.length);
        int mismatch = -1;
        for (int i = 0; i < length; ++i) {
            if (data1[i] != data2[i]) {
                mismatch = i;
                break;
            }
        }
        bh.consume(mismatch);
    }

The results speak for themselves. Irritatingly, there is some duplication in output because I haven’t figured out how to make JMH use a subset of the Cartesian product of its parameters.

Benchmark (prefix) (size) Mode Cnt Score Error Units
Mismatch_Handwritten 10 100 thrpt 10 22.360 ± 0.938 ops/us
Mismatch_Handwritten 10 1000 thrpt 10 2.459 ± 0.256 ops/us
Mismatch_Handwritten 10 10000 thrpt 10 0.255 ± 0.009 ops/us
Mismatch_Handwritten 100 100 thrpt 10 22.763 ± 0.869 ops/us
Mismatch_Handwritten 100 1000 thrpt 10 2.690 ± 0.044 ops/us
Mismatch_Handwritten 100 10000 thrpt 10 0.273 ± 0.008 ops/us
Mismatch_Handwritten 1000 100 thrpt 10 24.970 ± 0.713 ops/us
Mismatch_Handwritten 1000 1000 thrpt 10 2.791 ± 0.066 ops/us
Mismatch_Handwritten 1000 10000 thrpt 10 0.281 ± 0.007 ops/us
Mismatch_Intrinsic 10 100 thrpt 10 89.169 ± 2.759 ops/us
Mismatch_Intrinsic 10 1000 thrpt 10 26.995 ± 0.501 ops/us
Mismatch_Intrinsic 10 10000 thrpt 10 3.553 ± 0.065 ops/us
Mismatch_Intrinsic 100 100 thrpt 10 83.037 ± 5.590 ops/us
Mismatch_Intrinsic 100 1000 thrpt 10 26.249 ± 0.714 ops/us
Mismatch_Intrinsic 100 10000 thrpt 10 3.523 ± 0.122 ops/us
Mismatch_Intrinsic 1000 100 thrpt 10 87.921 ± 6.566 ops/us
Mismatch_Intrinsic 1000 1000 thrpt 10 25.812 ± 0.442 ops/us
Mismatch_Intrinsic 1000 10000 thrpt 10 4.177 ± 0.059 ops/us

Why is there such a big difference? Look at how the score decreases as a function of array length, even when the common prefix, and therefore the number of comparisons required, is small: clearly the performance of this algorithm depends on the efficiency of memory access. Arrays.mismatch optimises this, reading qwords of the array into SIMD registers. Working one long at a time, it is possible to compute the XOR in a single instruction to determine if it’s even necessary to look at each byte.

java/util/ArraysSupport.vectorizedMismatch(Ljava/lang/Object;JLjava/lang/Object;JII)I  [0x000002bd9215a820, 0x000002bd9215aa78]  600 bytes
Argument 0 is unknown.RIP: 0x2bd9215a820 Code size: 0x00000258
[Entry Point]
[Verified Entry Point]
[Constants]
  # {method} {0x000002bda79cbf68} 'vectorizedMismatch' '(Ljava/lang/Object;JLjava/lang/Object;JII)I' in 'java/util/ArraysSupport'
  # parm0:    rdx:rdx   = 'java/lang/Object'
  # parm1:    r8:r8     = long
  # parm2:    r9:r9     = 'java/lang/Object'
  # parm3:    rdi:rdi   = long
  # parm4:    rsi       = int
  # parm5:    rcx       = int
  #           [sp+0x60]  (sp of caller)
  0x000002bd9215a820: mov     dword ptr [rsp+0ffffffffffff9000h],eax
                                                ;...89
                                                ;...84
                                                ;...24
                                                ;...00
                                                ;...90
                                                ;...ff
                                                ;...ff

  0x000002bd9215a827: push    rbp               ;...55

  0x000002bd9215a828: sub     rsp,50h           ;...48
                                                ;...83
                                                ;...ec
                                                ;...50
                                                ;*synchronization entry
                                                ; - java.util.ArraysSupport::vectorizedMismatch@-1 (line 120)

  0x000002bd9215a82c: mov     r10,rdi           ;...4c
                                                ;...8b
                                                ;...d7

  0x000002bd9215a82f: vmovq   xmm2,r9           ;...c4
                                                ;...c1
                                                ;...f9
                                                ;...6e
                                                ;...d1

  0x000002bd9215a834: vmovq   xmm1,rdx          ;...c4
                                                ;...e1
                                                ;...f9
                                                ;...6e
                                                ;...ca

  0x000002bd9215a839: mov     r14d,ecx          ;...44
                                                ;...8b
                                                ;...f1

  0x000002bd9215a83c: vmovd   xmm0,esi          ;...c5
                                                ;...f9
                                                ;...6e
                                                ;...c6

  0x000002bd9215a840: mov     r9d,3h            ;...41
                                                ;...b9
                                                ;...03
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002bd9215a846: sub     r9d,ecx           ;...44
                                                ;...2b
                                                ;...c9
                                                ;*isub {reexecute=0 rethrow=0 return_oop=0}
                                                ; - java.util.ArraysSupport::vectorizedMismatch@5 (line 120)

  0x000002bd9215a849: mov     edx,esi           ;...8b
                                                ;...d6

  0x000002bd9215a84b: mov     ecx,r9d           ;...41
                                                ;...8b
                                                ;...c9

  0x000002bd9215a84e: sar     edx,cl            ;...d3
                                                ;...fa
                                                ;*ishr {reexecute=0 rethrow=0 return_oop=0}
                                                ; - java.util.ArraysSupport::vectorizedMismatch@17 (line 122)

  0x000002bd9215a850: mov     eax,1h            ;...b8
                                                ;...01
                                                ;...00
                                                ;...00
                                                ;...00

  0x000002bd9215a855: xor     edi,edi           ;...33
                                                ;...ff

  0x000002bd9215a857: test    edx,edx           ;...85
                                                ;...d2

  0x000002bd9215a859: jle     2bd9215a97ah      ;...0f
                                                ;...8e
                                                ;...1b
                                                ;...01
                                                ;...00
                                                ;...00

The code for this benchmark is at github.

Math.fma

In comparison to users of some languages, Java programmers are lackadaisical about floating point errors. It’s a good job that historically Java hasn’t been considered suitable for the implementation of numerical algorithms. But all of a sudden there is a revolution of data science on the JVM, albeit mostly driven by the Scala community, with JVM implementations of structures like recurrent neural networks abounding. It matters less for machine learning than root finding, but how accurate can these implementations be without JVM level support for minimising the propagation floating point errors? With Math.fma this is improving, by allowing two common operations to be performed before rounding.

Math.fma fuses a multiplication and an addition into a single floating point operation to compute expressions like ab + c. This has two key benefits:

  1. There’s only one operation, and only one rounding error
  2. This is explicitly supported in AVX2 by the VFMADD* instructions

Newton’s Method

To investigate any superior suppression of floating point errors, I use a toy implementation of Newton’s method to compute the root of a quadratic equation, which any teenager could calculate analytically (the error is easy to quantify).

I compare these two implementations for 4x^2 - 12x + 9 (there is a repeated root at 1.5) to get an idea for the error (defined by |1.5 - x_n|) after a large number of iterations.

I implemented this using FMA:


public class NewtonsMethodFMA {

    private final double[] coefficients;

    public NewtonsMethodFMA(double[] coefficients) {
        this.coefficients = coefficients;
    }

    public double evaluateF(double x) {
        double f = 0D;
        int power = coefficients.length - 1;
        for (int i = 0; i < coefficients.length; ++i) {
            f = Math.fma(coefficients[i], Math.pow(x, power--), f);
        }
        return f;
    }

    public double evaluateDF(double x) {
        double df = 0D;
        int power = coefficients.length - 2;
        for (int i = 0; i < coefficients.length - 1; ++i) {
            df = Math.fma((power + 1) * coefficients[i],  Math.pow(x, power--), df);
        }
        return df;
    }

    public double solve(double initialEstimate, int maxIterations) {
        double result = initialEstimate;
        for (int i = 0; i < maxIterations; ++i) {
            result -= evaluateF(result)/evaluateDF(result);
        }
        return result;
    }
}

And an implementation with normal operations:



public class NewtonsMethod {

    private final double[] coefficients;

    public NewtonsMethod(double[] coefficients) {
        this.coefficients = coefficients;
    }

    public double evaluateF(double x) {
        double f = 0D;
        int power = coefficients.length - 1;
        for (int i = 0; i < coefficients.length; ++i) {
            f += coefficients[i] * Math.pow(x, power--);
        }
        return f;
    }

    public double evaluateDF(double x) {
        double df = 0D;
        int power = coefficients.length - 2;
        for (int i = 0; i < coefficients.length - 1; ++i) {
            df += (power + 1) * coefficients[i] * Math.pow(x, power--);
        }
        return df;
    }

    public double solve(double initialEstimate, int maxIterations) {
        double result = initialEstimate;
        for (int i = 0; i < maxIterations; ++i) {
            result -= evaluateF(result)/evaluateDF(result);
        }
        return result;
    }
}

When I run this code for 1000 iterations, the FMA version results in 1.5000000083575202, whereas the vanilla version results in 1.500000017233207. It’s completely unscientific, but seems plausible and confirms my prejudice so… In fact, it’s not that simple, and over a range of initial values, there is only a very small difference in FMA’s favour. There’s not even a performance improvement – clearly this method wasn’t added so you can start implementing numerical root finding algorithms – the key takeaway is that the results are slightly different because a different rounding strategy has been used.

Benchmark (maxIterations) Mode Cnt Score Error Units
NM_FMA 100 thrpt 10 93.805 ± 5.174 ops/ms
NM_FMA 1000 thrpt 10 9.420 ± 1.169 ops/ms
NM_FMA 10000 thrpt 10 0.962 ± 0.044 ops/ms
NM_HandWritten 100 thrpt 10 93.457 ± 5.048 ops/ms
NM_HandWritten 1000 thrpt 10 9.274 ± 0.483 ops/ms
NM_HandWritten 10000 thrpt 10 0.928 ± 0.041 ops/ms

Can Java 9 Leverage AVX-512 Yet?

Advanced Vector Extensions (AVX) has gone through several iterations, doubling the size of SIMD registers available several times. The state of the art is currently AVX-512 which offers 512-bit registers. In parallel, support for auto-vectorisation of handwritten Java has been improving. With an early access JDK9 on an Intel(R) Core(TM) i7-6700HQ CPU I have observed usage of both 256-bit (vector addition) and 128-bit registers (sum of squares). I was very kindly granted access to a Knights Landing server where I ran the same code. I was hoping to see evidence of usage of 512-bit registers, though I didn’t see it directly I did see a strong indication that this occurs.

SIMD Registers

There is a hierarchy of register types from several iterations of SIMD instruction sets: 128 bits, named xmm, originally from Streaming SIMD Extensions (SSE), 256-bits, named ymm, originally from AVX, and 512-bits, named zmm, introduced with AVX-512. A 512-bit register is enormous; it can contain 16 ints and can be used as a single operand to various vector instructions allowing up to 16 ints to be operated on in parallel. Each of these registers can be used as if it is the predecessor by masking the upper bits.

On my laptop I have seen Java 9 use 128-bit xmm registers in a sum of squares calculation on doubles:

0x000001aada783fca: vmovsd  xmm0,qword ptr [rbp+r13*8+10h] ...
0x000001aada783fd1: vmulsd  xmm0,xmm0,xmm0 ...
0x000001aada783fd5: vmovq   xmm1,rbx ...
0x000001aada783fda: vaddsd  xmm0,xmm1,xmm0 ...

As an aside, the weird thing about this code is that only a 64-bit qword, as opposed to an xmmword, is being loaded into xmm0 here. The code generated for floats is very interesting in this respect because it seems to load a full 256-bit ymmword of the array and supplies it as an operand to vmulps.

  0x00000245b32aa0dc: vmovdqu ymm1,ymmword ptr [r8+r9*4+10h]                                           
  0x00000245b32aa0e3: vmulps  ymm1,ymm1,ymm1
  0x00000245b32aa0e7: vaddss  xmm0,xmm0,xmm1

I have also seen 256-bit ymm registers in use in the simpler float vector addition:

  0x000002759215a17a: vmovdqu ymm0,ymmword ptr [rcx+rax*4+50h]
  0x000002759215a180: vaddps  ymm0,ymm0,ymmword ptr [rbp+rax*4+50h]
  0x000002759215a186: vmovdqu ymmword ptr [rbx+rax*4+50h],ymm0

Where a 256-bit ymmword is being used to load chunks of the array into the register. The interesting thing about running the float vector addition case is that you can trace the optimisations being applied as the code runs; starting off with scalar instructions, graduating to AVX instructions on xmm registers, finally learning to utilise the ymm registers. This is quite an impressive feat of the optimiser.

What’s next? 512-bit zmm registers!

Knights Landing

Knights Landing has 32 512-bit zmm registers, I try to use them by running the code at github, and observing the assembly code emitted.

About the Box

The Knights Landing server has 256 processors, each with a modest clock speed, but support for AVX-512 core and several extensions (see the flags). It is designed for massively parallel workloads, and, for single threaded code, would not compare favourably with a commodity desktop. But if you can write code to keep 256 cores busy concurrently, it will likely profit from running on a Knights Landing.

processor       : 255
vendor_id       : GenuineIntel
cpu family      : 6
model           : 87
model name      : Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz
stepping        : 1
microcode       : 0x130
cpu MHz         : 1299.695
cache size      : 1024 KB
physical id     : 0
siblings        : 256
core id         : 73
cpu cores       : 64
apicid          : 295
initial apicid  : 295
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms avx512f rdseed adx avx512pf avx512er avx512cd xsaveopt
bogomips        : 2600.02
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

I am running on this server simply because it supports AVX-512, and am interested to see if Java can use it. The code is purposefully single threaded, and the point is not to confirm that one of 256 1.3Ghz processors is twice as slow as one of my eight 2.7GHz processors when measured on a single threaded workload, the interesting aspect is the instructions generated.

The operating system is CentOS Linux release 7.2.1511 (Core), the Java version used was Java HotSpot(TM) 64-Bit Server VM (build 9+177, mixed mode).

Observations

My observations were mixed: I didn’t see any 512-bit registers being used, some code expected to vectorise didn’t, but, critically, also saw evidence that the hsdis-amd64.so disassembler did not understand some of the assembly used. I hope, just because I like things to work, that this is hiding evidence of 512-bit register use. As you can see, there are a lot of new instructions in AVX-512 which may explain these black holes in the output.

I ran vector addition and sum of squares across a range of vector sizes and primitive types and saw little use of AVX instructions in my handwritten code, let alone evidence of 512-bit zmm register operands, yet extensive use of 256-bit registers for String intrinsics.

The first data point to look at is the relative timings (remember: absolute timings are meaningless because of Knights Landing design motivations, even more so because assembly was being printed during this run) which are interesting: some kind of optimisation is being applied to sum of squares only for ints.

Benchmark                          (size)   Mode  Cnt   Score    Error   Units
c.o.s.add.Addition.Add_Doubles     100000  thrpt   10   3.022 ±  0.262  ops/ms
c.o.s.add.Addition.Add_Doubles    1000000  thrpt   10   0.284 ±  0.006  ops/ms
c.o.s.add.Addition.Add_Floats      100000  thrpt   10   6.212 ±  0.521  ops/ms
c.o.s.add.Addition.Add_Floats     1000000  thrpt   10   0.572 ±  0.019  ops/ms
c.o.s.add.Addition.Add_Ints        100000  thrpt   10   6.383 ±  0.445  ops/ms
c.o.s.add.Addition.Add_Ints       1000000  thrpt   10   0.573 ±  0.019  ops/ms
c.o.s.add.Addition.Add_Longs       100000  thrpt   10   3.022 ±  0.241  ops/ms
c.o.s.add.Addition.Add_Longs      1000000  thrpt   10   0.281 ±  0.025  ops/ms
c.o.s.ss.SumOfSquares.SS_Doubles   100000  thrpt   10   2.145 ±  0.004  ops/ms
c.o.s.ss.SumOfSquares.SS_Doubles  1000000  thrpt   10   0.206 ±  0.001  ops/ms
c.o.s.ss.SumOfSquares.SS_Floats    100000  thrpt   10   2.150 ±  0.002  ops/ms
c.o.s.ss.SumOfSquares.SS_Floats   1000000  thrpt   10   0.212 ±  0.001  ops/ms
c.o.s.ss.SumOfSquares.SS_Ints      100000  thrpt   10  16.960 ±  0.043  ops/ms
c.o.s.ss.SumOfSquares.SS_Ints     1000000  thrpt   10   1.015 ±  0.019  ops/ms
c.o.s.ss.SumOfSquares.SS_Longs     100000  thrpt   10   6.379 ±  0.014  ops/ms
c.o.s.ss.SumOfSquares.SS_Longs    1000000  thrpt   10   0.429 ±  0.033  ops/ms

I looked at how SS_Ints was being executed and saw the usage of the vpaddd (addition of packed integers) instruction with xmm register operands, in between two instructions the disassembler (hsdis) seemingly could not read. Maybe these unprintable instructions are from AVX-512 or extensions? It’s impossible to say. The same mechanism, using vpaddq, was not used for longs, but is used on my laptop.

  0x00007fc31576edf1: (bad)                     ;...0e

  0x00007fc31576edf2: vpaddd %xmm4,%xmm1,%xmm1  ;...c5f1fecc

  0x00007fc31576edf6: (bad)                     ;...62

The case of vector addition is less clear; there are many uninterpreted instructions in the output, while it clearly vectorises on my laptop using the largest registers available. The only vector instruction present was vzeroupper, which zeroes the upper 128 bits of a 256-bit register, and is usually used for optimising use of SSE on more modern architectures. Incidentally, there is explicit advice from Intel against using this instruction on Knights Landing. I saw assembly for the scalar implementation of this code (this will always happen prior to optimisation), but there’s no evidence the code was vectorised. However, there were 110 (bad) instructions in the output for float array addition alone.

An interesting presentation, by one of the key Hotspot compiler engineers, outlines some of the limits of SIMD support in Java. It neither includes examples of 512-bit register usage nor states that this is unsupported. Possibly Project Panama‘s Long8 type will utilise 512-bit registers. I will recompile the disassembler and give JDK9 the benefit of the doubt for now.

Comparing Streams with Arrays

Java 8 added some great features which, combined with the vigilance of a responsible adult, make it easier to keep a Java code base civilised. Streams and lambdas, especially the limited support offered for primitive types, are a fantastic addition. Is there an observable performance cost to using these features? For a simple calculation amenable to instruction level parallelism, I compare modern and traditional implementations and observe the differences in instructions generated.

Sum of Squares

The sum of squares is the building block of a linear regression analysis so is ubiquitous in statistical computing. It is associative and therefore data-parallel. I compare four implementations: a sequential stream wrapping an array, a parallel stream wrapping an array, a generative sequential stream and a traditional for loop. The benchmark code is on github.


    @Benchmark
    public void SS_SequentialStream(Data state, Blackhole bh) {
        bh.consume(DoubleStream.of(state.data)
                               .map(x -> x * x)
                               .reduce((x, y) -> x + y));
    }

    @Benchmark
    public void SS_ParallelStream(Data state, Blackhole bh) {
        bh.consume(DoubleStream.of(state.data)
                               .parallel()
                               .map(x -> x * x)
                               .reduce((x, y) -> x + y));
    }

    @Benchmark
    public void SS_ForLoop(Data state, Blackhole bh) {
        double result = 0D;
        final double[] data = state.data;
        for (int i = 0; i < data.length; ++i) {
            result += data[i] * data[i];
        }
        bh.consume(result);
    }

    @Benchmark
    public void SS_GenerativeSequentialStream(Data state, Blackhole bh) {
        double[] data = state.data;
        bh.consume(IntStream.iterate(0, i -> i + 1)
                            .limit(1_000_000)
                            .mapToDouble(i -> data[i])
                            .map(x -> x * x)
                            .reduce((x, y) -> x + y));
    }

I must admit I prefer the readability of the stream versions, but let’s see if there is a comedown after the syntactic sugar rush.

Running a Benchmark

I compare the three implementations on an array of one million doubles. I am using JDK 9-ea, VM 9-ea+166 on a fairly powerful laptop with 8 processors:

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 94
model name      : Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
stepping        : 3
cpu MHz         : 2592.000
cache size      : 256 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pni dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt aes xsave osxsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

Before running the benchmark we might expect the for loop and stream to have similar performance, and the parallel version to be about eight times faster. The generative version is very similar to the for loop so a slow down might not be expected. To isolate vectorised optimisations, I first run with SuperWord disabled.

Benchmark Mode Count Score Units
SS_ForLoop thrpt 10 0.540 ± 0.072 ops/ms
SS_ParallelStream thrpt 10 2.336 ± 0.412 ops/ms
SS_SequentialStream thrpt 10 0.565 ± 0.052 ops/ms
SS_GenerativeSequentialStream thrpt 10 0.247 ± 0.036 ops/ms

The for loop and stream are similar but the stream is better. The parallel version is faster but is either not utilising all of the cores or incurring a cost; the improvement in throughput is only a factor of 4.3. It has never felt like a good idea to execute code blindly on a default fork join pool without safe guards. There is a huge difference between streams which wrap arrays and a stream which iterates over an array, most likely related to cache locality.

What happens when SuperWord optimisations are enabled? Throughput improves except for the generative stream (suggesting no SIMD execution at all), but the for loop improves the most.

Benchmark Mode Count Score Units
SS_ForLoop thrpt 10 0.614 ± 0.035 ops/ms
SS_ParallelStream thrpt 10 2.551 ± 0.095 ops/ms
SS_SequentialStream thrpt 10 0.596 ± 0.043 ops/ms
SS_GenerativeSequentialStream thrpt 10 0.240 ± 0.049 ops/ms

The three nines sample is actually worse than the for loop in the parallel stream version, whereas the generative stream is off the chart.

Benchmark Mode Score Units
SS_ForLoop·p0.999 sample 5.424 ms/op
SS_ParallelStream·p0.999 sample 6.077 ms/op
SS_SequentialStream·p0.999 sample 6.340 ms/op
SS_GenerativeSequentialStream·p0.999 sample 14.516 ms/op

Inspecting the assembly code generated, it is clear that the for loop body is actually only compiling down to SSE here: it’s only loading one double at a time into registers.

0x000001aada783fca: vmovsd  xmm0,qword ptr [rbp+r13*8+10h] ...
0x000001aada783fd1: vmulsd  xmm0,xmm0,xmm0 ...
0x000001aada783fd5: vmovq   xmm1,rbx ...
0x000001aada783fda: vaddsd  xmm0,xmm1,xmm0 ...

But so is the sequential stream – here is the lambda implementing the square:

0x00000170c11228be: vmulsd  xmm1,xmm0,xmm0    ;...c5
                                                ;...fb
                                                ;...59
                                                ;...c8
                                                ;*dmul {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.mythbusters.streams.StreamReduction::lambda$SS_SequentialStream$0@2 (line 66)
                                                ; - com.openkappa.mythbusters.streams.StreamReduction$$Lambda$40/1123395594::applyAsDouble@1
                                                ; - java.util.stream.DoublePipeline$2$1::accept@12 (line 213)
                                                ; - java.util.Spliterators$DoubleArraySpliterator::forEachRemaining@53 (line 1198)
                                                ; - java.util.Spliterator$OfDouble::forEachRemaining@12 (line 828)
                                                ; - java.util.stream.AbstractPipeline::copyInto@32 (line 484)
                                                ; - java.util.stream.AbstractPipeline::wrapAndCopyInto@13 (line 474)
                                                ; - java.util.stream.ReduceOps$ReduceOp::evaluateSequential@6 (line 913)
                                                ; - java.util.stream.AbstractPipeline::evaluate@88 (line 234)

And here is the reduce:

  0x00000170c1118610: vaddsd  xmm1,xmm1,xmm2    ;...c5
                                                ;...f3
                                                ;...58
                                                ;...ca
                                                ;*dadd {reexecute=0 rethrow=0 return_oop=0}
                                                ; - com.openkappa.mythbusters.streams.StreamReduction::lambda$SS_SequentialStream$1@2 (line 66)
                                                ; - com.openkappa.mythbusters.streams.StreamReduction$$Lambda$41/1355169748::applyAsDouble@2
                                                ; - java.util.stream.ReduceOps$12ReducingSink::accept@30 (line 693)
                                                ; - java.util.stream.DoublePipeline$2$1::accept@17 (line 213)
                                                ; - java.util.Spliterators$DoubleArraySpliterator::forEachRemaining@53 (line 1198)

Is it mechanically sympathetic to separate these instructions like this? Probably not, which explains the degradation in performance relative to the for loop, but it’s not so drastic that a completely different instruction set is being used. Use of the same SSE instructions can be seen with Stream.parallelStream.

Conclusion

The same optimisations are applied to streams when they are countable. Arrays seems to do slightly better, but not well enough to eschew streams. Uncountable or opaquely countable streams perform significantly worse than countable streams or traditional code. Stream.parallelStream can improve performance but does not make the best use of the cores available and using a global thread pool leads to unpredictability. Balancing performance and readability leads to an approach where data is stored in paginated arrays and accessed and transformed via the streams API.

Targeting SIMD in Java

Vectorised instruction execution can be targeted directly in C++ but with Java there are extra layers of abstraction to go through. Folklore aside, when does vectorisation or SIMD execution actually happen in Java? Skeptical of old wives’ tales, I investigate when SIMD instructions are actually used in Java 9, and how to disable it by programming badly.

Building a Benchmark Harness

I use JMH to write benchmarks. For the uninitiated, it is the standard Java micro-benchmarking framework and handles various pitfalls, the most salient of which being that it ensures that your code actually executes, and ensures measurement is performed against a monotonic measure of time. Benchmarks produced without JMH are unlikely to be correct and should arouse suspicion, especially if used during a sales process.

Averages always lie, so minimisation of 99.9th percentile execution time is a better objective to have in mind. As a general performance indicator I measure throughput in operations per time unit, which is useful to correlate with JVM compilation events.

To tune performance, before even worrying about achieving SIMD execution, we need to be aware of recompilation and failures to inline. These features are enabled by the arguments below and my simple harness has a specific mode to support these diagnostics:

-XX:+PrintCompilation 
-XX:+UnlockDiagnosticVMOptions 
-XX:+PrintInlining

The proof that code is actually being vectorised is to observe the emission of AVX instructions. To prove this has happened (and if it does, it will be correlated with astonishing performance statistics), we need to see the assembly code so I run the benchmark in a mode that will print out the generated assembly via the arguments:

-XX:+UnlockDiagnosticVMOptions
-XX:+PrintAssembly
-XX:PrintAssemblyOptions=hsdis-print-bytes 
-XX:CompileCommand=print

However, SIMD execution only happens when SuperWord parallelism is enabled (and by default, it is) so we won’t even need to look at the assembly unless we see a clear difference when running the benchmark without the UseSuperWord option:

-XX:-UseSuperWord

What Gains Can Be Expected?

Is vectorised code execution a panacea? Assuming we can force the JVM to use SIMD, how much performance can be expected? It turns out that java.util.Arrays.fill can be vectorised, so we can get a taste of the difference it makes. We can observe its impact by benchmarking throughput with and without SuperWord instruction level parallelism.


    private void benchmark(String... jvmArgs) throws RunnerException {
            Options opt = new OptionsBuilder()
                    .include(this.getClass().getName() + ",*")
                    .mode(Mode.Throughput)
                    .timeUnit(TimeUnit.MILLISECONDS)
                    .warmupIterations(5)
                    .measurementIterations(5)
                    .forks(1)
                    .shouldFailOnError(true)
                    .shouldDoGC(true)
                    .operationsPerInvocation(1_000_000)
                    .jvmArgs(jvmArgs)
                    .build();
            new Runner(opt).run();
    }

    @Benchmark
    public void fillArray(Blackhole bh)  {
        double[] array = new double[1 << 10];
        for(int i = 0; i << 1_000_000; ++i) {
            Arrays.fill(array, i);
            bh.consume(array);
        }
    }

# VM version: JDK 9-ea, VM 9-ea+166
...
# VM options: -XX:-UseSuperWord
...

Benchmark                 Mode  Cnt    Score     Error   Units
TestBenchmark.fillArray  thrpt    5  966.947 ± 596.705  ops/ms

# VM version: JDK 9-ea, VM 9-ea+166
...
# VM options: -XX:+UseSuperWord
...

Benchmark                 Mode  Cnt     Score      Error   Units
TestBenchmark.fillArray  thrpt    5  4230.802 ± 5311.001  ops/ms

The difference in throughput is palpable and astonishing.

Intrinsics

Various intrinsics, such as Arrays.fill above, compile down to vectorised instructions. When SuperWord parallelism is enabled, they will always run faster. The JIT compiler will also target simple hand-written code, but intrinsics are a safer bet.

Adding Arrays Together

Addition of arrays will either use SIMD or not; it depends how you go about writing your code. If your code is too complicated then you will not achieve vectorised execution. I will start from a method which does not vectorise, transform it to one that does by a process of simplification, and then break it again by trying to be too clever. The code which does vectorise is the same naive code I would have written in C++ as a student, ignorant of JVM internals. All of the code is available at github.

To start off with I created a highly contrived class called BadCode which is actually inspired, albeit seriously exacerbated, by a generic function in an API I have seen in a professional setting. The so called object oriented (sic) API seeks to be able to operate on any type of primitive array, so takes the lowest common denominator (java.lang.Object) as parameters and performs instanceof checks to get the correctly typed array instance. While the code is bloated, this provides as much flexibility as possible, which suits the supplier of the API – which has many clients with disparate use cases – but not necessarily the performance constrained caller.

The code is too bloated to include here but can be seen on github. It has two major performance bugs which will be addressed in turn:

  1. It’s verbose enough not to inline
  2. It has more than one exit point

Because the method supports so many use cases, the code size is very large (2765 bytes). By default, any method with a code size of greater than 2000 bytes will fail to inline. Inlining is at the very base of our hierarchy of performance needs. We can see that the JIT compiler has failed to inline the method by printing inlining and compilation:

   2193  715 %     4       com.openkappa.simd.generated.TestBenchmark_badClass_jmhTest::badClass_thrpt_jmhStub @ 13 (55 bytes)
                              @ 19   com.openkappa.simd.TestBenchmark::badClass (9 bytes)   force inline by CompileCommand
                                @ 2   com.openkappa.simd.TestBenchmark$BadClassState::compute (16 bytes)   inline (hot)
                                  @ 12   com.openkappa.simd.BadCode::bigMethod (2765 bytes)   hot method too big

Taking a look at our throughput, we can see there is a problem. The code isn’t getting any faster, so clearly no optimisations are being applied.
# Run progress: 0.00% complete, ETA 00:00:40
# Fork: 1 of 1
# Warmup Iteration   1: 0.193 ops/us
# Warmup Iteration   2: 0.176 ops/us
# Warmup Iteration   3: 0.216 ops/us
# Warmup Iteration   4: 0.183 ops/us
# Warmup Iteration   5: 0.294 ops/us
Iteration   1: 0.260 ops/us
Iteration   2: 0.224 ops/us
Iteration   3: 0.170 ops/us
Iteration   4: 0.169 ops/us
Iteration   5: 0.169 ops/us

Our three nines is terrible, taking up to six microseconds to add two arrays:
TestBenchmark.badClass:badClass·p0.999         sample       6.296           us/op

Regardless of the large code size, the approach taken in the API has enforced multiple exit points in the method, which will always disable SIMD execution. Running with SuperWord disabled does not worsen performance, implying our code is not getting vectorised.

Smaller Code

Having noticed that the method is not getting inlined, let alone compiling to AVX instructions, we need to make the code smaller first. In this scenario our values are always double[] so the API provider effectively forces us to pay for the disparate use cases they must support, and this taxation without representation harms performance. We can rewrite it to be smaller, targeting our own specific use case. The code is concise enough to include here, and is the code any student would write to perform element-wise addition of two arrays. Notice the loop condition.


public class SmallerCode {

    public double[] smallMethod(double[] left, double[] right) {
        double[] result = new double[left.length];
        for(int i = 0; i < left.length && i < right.length; ++i) {
            result[i] = left[i] + right[i];
        }
        return result;
    }
}

Let’s benchmark SmallerCode, where the same code has a size of 43B. The method is indeed inlined.

   1374  647       3       com.openkappa.simd.TestBenchmark$SmallerCodeState::compute (16 bytes)   made not entrant
                              @ 12   com.openkappa.simd.SmallerCode::smallMethod (43 bytes)   inline (hot)

Throughput is doubled and we see evidence of dynamic optimisation.
# Run progress: 25.00% complete, ETA 00:02:42
# Fork: 1 of 1
# Warmup Iteration   1: 0.372 ops/us
# Warmup Iteration   2: 0.340 ops/us
# Warmup Iteration   3: 0.432 ops/us
# Warmup Iteration   4: 0.497 ops/us
# Warmup Iteration   5: 0.499 ops/us
Iteration   1: 0.398 ops/us
Iteration   2: 0.364 ops/us
Iteration   3: 0.408 ops/us
Iteration   4: 0.544 ops/us
Iteration   5: 0.401 ops/us

This code is twice as fast, and our three nines is better, just by virtue of keeping the code simple.
TestBenchmark.smallerCode:smallerCode·p0.999             sample       2.164          us/op

But are we getting SIMD execution? Possibly – disabling SuperWord yields noticeably worse results.
# Run progress: 25.00% complete, ETA 00:02:22
# Fork: 1 of 1
# Warmup Iteration   1: 0.261 ops/us
# Warmup Iteration   2: 0.343 ops/us
# Warmup Iteration   3: 0.294 ops/us
# Warmup Iteration   4: 0.320 ops/us
# Warmup Iteration   5: 0.316 ops/us
Iteration   1: 0.293 ops/us
Iteration   2: 0.276 ops/us
Iteration   3: 0.304 ops/us
Iteration   4: 0.291 ops/us
Iteration   5: 0.279 ops/us

It’s worth inspecting the assembly to see if we can observe the emission of AVX instructions once we reenable UseSuperWord. Assembly is easier to read if inlining is disabled, this can be controlled in JMH with this annotation: @CompilerControl(CompilerControl.Mode.DONT_INLINE). Applying the annotation, the assembly is printed using the appropriate JVM args:
-XX:+UnlockDiagnosticVMOptions 
-XX:+PrintAssembly
-XX:PrintAssemblyOptions=hsdis-print-bytes 
-XX:CompileCommand=print

0x00000154edb691e0: vmovdqu ymm0,ymmword ptr [rbp+r10*8+10h]

0x00000154edb691e7: vaddpd  ymm0,ymm0,ymmword ptr [rdx+r10*8+10h]

0x00000154edb691ee: vmovdqu ymmword ptr [r8+r10*8+10h],ymm0

0x00000154edb691f5: add     r10d,4h           

0x00000154edb691f9: cmp     r10d,ebx          

0x00000154edb691fc: jl      154edb691e0h      

It’s true – this code is indeed vectorised – see the AVX instructions in bold!

Ruining It

Having witnessed vectorisation when adding arrays together, let’s try and break it. There are a few patterns we can apply to break our vectorised code:

  • Putting an OR condition as the loop condition
  • Putting a non-inlined method inside the loop
  • Putting an arbitrary method as the loop condition
  • Manually unrolling the loop
  • Using a long as the loop variable
  • Multiple exit points

The list goes on but now let’s really fuck it up. We were using double[] so let’s see what happens if we use a DirectByteBuffer as the backing for a homegrown vector construct instead. Instead of returning a new heap allocated array, we will write our doubles into a byte buffer, and use an offset and length to delineate the arrays. For instance, the code below will write the sum of two arrays stored in a byte buffer back into the same byte buffer. We can abstract vectors by creating small on-heap objects for each array which know the offset and length of each array in the buffer.


public int add(ByteBuffer byteBuffer,
               int leftOffset, int leftLength,
               int rightOffset, int rightLength,
               int resultOffset) {
        int resultIndex = resultOffset;
        for(int l = leftOffset, r = rightOffset;
            l < leftOffset + leftLength && r < rightOffset + rightLength;
            l += 8, r += 8, resultIndex += 8) {
            byteBuffer.putDouble(resultIndex, byteBuffer.getDouble(l) + byteBuffer.getDouble(r));
        }
        return resultIndex;
    }

Is this clever? We’ve beaten the garbage collector. It certainly feels like a clever engineering story. No, performance wise, we are back to where we were with the remarkably convoluted and conflated catch all function. Entertainingly, had we only ever benchmarked against BadCode::bigMethod we may not have noticed this performance regression.

# Run progress: 0.00% complete, ETA 00:00:40
# Fork: 1 of 1
# Warmup Iteration   1: 0.156 ops/us
# Warmup Iteration   2: 0.160 ops/us
# Warmup Iteration   3: 0.198 ops/us
# Warmup Iteration   4: 0.190 ops/us
# Warmup Iteration   5: 0.272 ops/us
Iteration   1: 0.220 ops/us
Iteration   2: 0.242 ops/us
Iteration   3: 0.216 ops/us
Iteration   4: 0.248 ops/us
Iteration   5: 0.351 ops/us

TestBenchmark.byteBuffer:byteBuffer·p0.999     sample       6.552           us/op

The obvious indicator that this is not being vectorised is that performance does not degrade when setting
-XX:-UseSuperWord

And to be sure we can inspect the assembly code emitted (without disabling UseSuperWord!).
  0x000001869216cb67: mov     edx,ecx

  0x000001869216cb69: add     edx,r9d

  0x000001869216cb6c: cmp     ecx,edx

  0x000001869216cb6e: jnl     1869216cb1dh

  0x000001869216cb70: mov     r13d,dword ptr [r12+r11*8+8h]
                                                
  0x000001869216cb75: cmp     r13d,0f8007ed8h
                                      
  0x000001869216cb7c: jne     1869216d015h

  0x000001869216cb82: lea     r13,[r12+r11*8]

  0x000001869216cb86: movzx   eax,byte ptr [r13+29h]

  0x000001869216cb8b: test    eax,eax

  0x000001869216cb8d: je      1869216d015h    

The reality is that whenever sun.misc.Unsafe is used, directly or indirectly, we lose access to SIMD. The bottom line is if you want to exploit instruction level parallelism, then be prepared to write code even a beginner could understand instantly. Bizarre off-heap data structures and vectorised execution? Unlikely.