testing

Big Data

Tuning Spark Back Pressure by Simulation

Spark back pressure, which can be enabled by setting spark.streaming.backpressure.enabled=true, will dynamically resize batches so as to avoid queue build up. It is implemented using a Proportional Integral¬†Derivative (PID) algorithm. This algorithm¬†has some interesting properties,¬†including the lack of guarantee of a stable fixed point. This can manifest itself not just in transient overshoot, but in […]

Read More