ckpt write performance problem

Hello all,

When benchmarking ckpt service, I find saCkptCheckpointWrite's performance is
substantially lower than saCkptSectionOverwrite's (2~4 times slower depending
on the data size per write).

You can trigger the problem by saCkptCheckpointWrite 2 iov elements, both iov
element referencing the same ckpt section, but writes to different offset.

Problem lies in _ckptCheckpointWrite, currently I circumvent the problem using
the patch below.

Another three problems found during benchmarking (benchmark code borrowed from
openais's ckptbench.c)

* When write large chunk, kernel coredumps. In my benchmark case, >2.5M Bytes per
write will trigger the problem.

* Ckpt write performance drops substantially after reach certain chunk size. In my
benchmark case, ckpt performance number looks like below

chunk size (Bytes per write), ckpt throughput (bps)
- 30k, 206M
- 40k, 229M
- 50k, 244M
- 60k, 260M
- 70k, 23M<----
- 100k, 30M

* Openclovis's ckpt performance don't perform well on small chunk size as compared
to openais. Eg. for 1k Bytes per write, openclovis's thruput is 15Mbps, while
openais's is 33Mbps.

I don't dig into the above problems (app won't touch them at present). Pls drop
message if you get solution.

Thanks
Eddy

*** clCkptSaf.c.orig 2007-01-05 17:06:09.120859952 +0800
--- clCkptSaf.c 2007-01-05 17:05:46.050367200 +0800
***************
*** 1371,1383 ****
CKPT_ERR_CHECK(CL_CKPT_SVR,CL_DEBUG_ERROR,
("Failed to overwrite on remote server %s rc[0x %x]",
pCkpt->ckptName.value,rc), rc);
! checkFlag = CKPT_SEC_OVERWRITE;
! rc = clXdrMarshallClInt32T(&checkFlag,msg,0);
! CKPT_ERR_CHECK(CL_CKPT_SVR,CL_DEBUG_ERROR,
! ("Failed: Marshalling rc[0x %x]\n",rc), rc);
! rc = ckptSectionDetailedPack(msg, pCkpt, pSec);
! CKPT_ERR_CHECK(CL_CKPT_SVR,CL_DEBUG_ERROR,
! ("Failed to Pack section rc[0x %x]\n", rc), rc);
pIoVector++;
}
if(peerCount > 1)
--- 1371,1400 ----
CKPT_ERR_CHECK(CL_CKPT_SVR,CL_DEBUG_ERROR,
("Failed to overwrite on remote server %s rc[0x %x]",
pCkpt->ckptName.value,rc), rc);
! /*
! * performance optimization:
! *
! * When vector array contains multiple vector entries point to the same
! * ckpt section, msg ends up with redundant ckpt section overwrite info
! * (which will hughly degrade ckpt performance when copying section data
! * to msg).
! *
! * We optimize this scenario by allowing user to group vector entries
! * point to the same ckpt section as contiguous vector entries block,
! * thus we only write one ckpt section overwrite info to msg per updated
! * ckpt section.
! */
! if ((count == numberOfElements - 1) ||
! ((pIoVector + 1)->sectionId.idLen != pIoVector->sectionId.idLen) ||
! (memcmp((pIoVector + 1)->sectionId.id, pIoVector->sectionId.id, pIoVector->sectionId.idLen) != 0)) {
! checkFlag = CKPT_SEC_OVERWRITE;
! rc = clXdrMarshallClInt32T(&checkFlag,msg,0);
! CKPT_ERR_CHECK(CL_CKPT_SVR,CL_DEBUG_ERROR,
! ("Failed: Marshalling rc[0x %x]\n",rc), rc);
! rc = ckptSectionDetailedPack(msg, pCkpt, pSec);
! CKPT_ERR_CHECK(CL_CKPT_SVR,CL_DEBUG_ERROR,
! ("Failed to Pack section rc[0x %x]\n", rc), rc);
! }
pIoVector++;
}
if(peerCount > 1)