Here’s the steps I followed to build the kernel on CentOS4. Writing it here so I dont forget next time, and it helps some other distressed soul.

Download kernel-xxx.src.rpm from any of centos mirrors.
groupadd buildcentos
useradd -g buildcentos buildcentos
rpm -ivh kernel-xxx-src.rpm
cd /usr/src/redhat/SPECS
rpmbuild -bp –target=`uname -p` kernel-xxx.spec
cd /usr/src/redhat/BUILD/kernel-xxx/linux-xxx
make menuconfig
cp -f .config /usr/src/redhat/SOURCES/kernel-xxx-ARCH.config
cd /usr/src/redhat/SPECS
rpmbuild -ba –target=`uname -p` kernel-xxx.spec
go get [...]