Framework performance
Is there some sort of throttling mechanism build into the framework? I ask because I’ve created a small app to test performance and when it runs it does not consume all available CPU cycles, or even more than three-quarters for that matter.
What I’ve done is modified one of the evaluation apps to in the active write an incrementing counter value to the COR as quickly as possible (no sleep in the while(!exiting) loop). The standby application sleeps for 1 second and then wakes up to read the counter from the COR then sleeps again.
What I see I that the counts per iteration on my box only get to at max 400 or so dipping to under 200 at times. When I use top to inspect the processes I see that the sisp_cor and sisp_txn are working pretty hard followed by the eval applications but I still see the system has idle cycles.
I’m trying to get a feel for the performance of the framework/system to see if it will meet the needs we have but at a maximum 400 transactions per second the performance isn’t even close to what we’ve spec’ed.
Anyone have any ideas on if this type of performance is expected or should it be better?
Remember that transaction completion entails a message being sent to the secondary, the secondary persisting the transaction, a reply back to the primary, and processing of the reply. Your inability to achieve 100% of the CPU is probably due to network latency. In fact, network latency is probably also your primary limiting factor for performance because it sounds like you are issuing the transactions serially.
If you compare these results to other DB's white papers please read the fine print; I have seen them put both the primary and secondary on the same blade and justify it by arguing that the DB software has not control over network latency. Ideally, you should test all contenders on the same hardware with the same write pattern; my experience is that DB performance white papers achieve exaggerated results through techniques like cache preloading, asynchronous operations, relaxing the "Durability" constraint and so forth.
To improve performance in your test, try doing more then 1 operation (write to 2 different items for example) in a single transaction, and try spawning multiple copies of your test in separate threads of control. Unless you have a particular use in mind, having 5 to 10 write threads that do 3-10 operations per transaction is probably a more realistic access pattern then sequentially writing the same single item.
If you would like to contribute your performance test to our benchmark model (in development) that would be highly appreciated & I'll be able to help you tweak for performance.
Good luck,
Andrew Stone
stone@openclovis.com