joshuaboniface
/
blog
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Fix wording in random write

This commit is contained in:
Joshua Boniface 2023-07-29 16:07:33 -04:00
parent 3f98498413
commit 09359c21cf
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
content/pvc-ceph-tuning-adventures-part-3.md
View File

@ -129,7 +129,7 @@ System load continues to show almost no correlation at all with performance, and
Random writes bring back the strange anomaly that we saw with sequential reads in the previous post. Namely, that for some reason, the no-limit configuration performs significantly better than all limits. After that, the performance seems to scale roughly linearly with each increase in CPU core count, exactly as was seen with the SATA SSDs in the previous post.
The system load here shows a possibly explanation for the anomalous results though. Random writes seem to hit the CPU much harder than the other tests, and the baseline load of all nodes with the no-limit configuration is about 8, which would indicate that the OSD processes want about 8 CPU cores per OSD here. This is double even our highest limited configuration, and thus seems to indicate that any OSD CPU limit below 4+8+20 will hurt write performance. Adding in the 4+8+20 configuration, we can see that this is definitely higher than all the other limit configurations, but is still less than the no-limit configuration.
The system load here shows a possibly explanation for the anomalous results though. Random writes seem to hit the CPU much harder than the other tests, and the baseline load of all nodes with the no-limit configuration is about 8, which would indicate that the OSD processes want about 8 CPU cores per OSD here. Adding in the 4+8+20 configuration, we can see that this is definitely higher than all the other limit configurations, but is still less than the no-limit configuration. It does appear that the scaling is not linear as well, since doubling the cores only brought us about half-way up to the no-limit performance, thus giving us a pretty conclusively "yes" answer to our first main question.
For write-heavy workloads, this is a very important takeaway. This test clearly shows that the no-limit configuration is ideal for random writes on NVMe drives, as the Linux scheduler seems better able to distribute the load among many cores. I'd be interested to see how this is affected by many CPU-heavy noisy-neighbour VMs, but testing this is extremely difficult and thus is not in scope for this series.