Installing Nvidia Tesla C1060 on a Mac Pro with CentOS

After my last post on getting CUDA 2.0 to run on my Mac, I will go through the steps for installing a Nvidia Tesla C1060 on a early 2008 Mac Pro with CentOS or equivalent RHEL 5.xx Linux distro. The Tesla C1060 has 240 streaming processor cores (1.3 GHz), 4GB onboard memory, requires PCIe x16, occupies two slots and has a 6-pin and 8-pin connector.

Instructions:

1. Physical install: Please download the C1060 User Manual and go through it before installing the C1060 physically. The C1060 has two connectors 6-pin and 8-pin, for a Mac Pro please please order the Booster cable (that's the correct Apple jargon). The booster cable is mentioned in the Mac Pro documentation to connect the Nvidia Quadro FX 5600. After inserting the C1060, refer to page 16 of the manual to see the LED status. Red indicates something wrong in connecting the booster cables. (As an afterthought, if you have a PC, you can buy the standard 6-pin/8-pin connector from Newegg for about $6.99)

2. Software install: Step 1: Get the latest CUDA drivers, toolkit and SDK from here. I selected the CUDA 2.1 drivers for Linux 64-bit for RHEL 5.xx. (You can paste the code directly to your terminal)

wget http://developer.download.nvidia.com/compute/cuda/2_1/drivers/NVIDIA-Linux-x86_64-180.22-pkg2.run
wget http://developer.download.nvidia.com/compute/cuda/2_1/toolkit/cudatoolkit_2.1_linux64_rhel5.2.run
wget http://developer.download.nvidia.com/compute/cuda/2_1/SDK/cuda-sdk-linux-2.10.1215.2015-3233425.run

Change permissions on the downloaded files

chmod +x NVIDIA-Linux-x86_64-180.22-pkg2.run
chmod +x cuda-sdk-linux-2.10.1215.2015-3233425.run
chmod +x cudatoolkit_2.1_linux64_rhel5.2.run

Step 2: Install the following packages or see if you have them installed on your system already.

sudo yum install freeglut-devel
sudo yum install libXi-devel

Step 3: Install the CUDA driver first, the toolkit and the SDK finally.

sudo ./NVIDIA-Linux-x86_64-180.22-pkg2.run
sudo ./cudatoolkit_2.1_linux64_rhel5.2.run
./cuda-sdk-linux-2.10.1215.2015-3233425.run

The Nvidia driver build the kernel for the Linux distro, if it cannot find the precompiled kernel. Click yes and go on. The toolkit installs cuda to /usr/local/cuda by default. The SDK installs into the NVIDIA_CUDA_SDK directory in your home folder by default.

Paste the following in your terminal (thanks to the Ubuntu instructions here)

echo "# CUDA stuff
PATH=\$PATH:/usr/local/cuda/bin
LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/usr/local/cuda/lib
export PATH
export LD_LIBRARY_PATH" >> ~/.bashrc

Step 4: On a RHEL Linux distro, a startup script is required to turn on and off the Nvidia device. This startup script is posted on the Nvidia forums.

#!/bin/bash
#
# Startup/shutdown script for nVidia CUDA
#
# chkconfig: 345 80 20
# description: Startup/shutdown script for nVidia CUDA

# Source function library.
. /etc/init.d/functions

DRIVER=nvidia
RETVAL=0

# Create /dev nodes for nvidia devices
function createnodes() {
   # Count the number of NVIDIA controllers found.
  N3D=`/sbin/lspci | grep -i NVIDIA | grep "3D controller" | wc -l`
  NVGA=`/sbin/lspci | grep -i NVIDIA | grep "VGA compatible controller" | wc -l`

  N=`expr $N3D + $NVGA - 1`
  for i in `seq 0 $N`; do
       mknod -m 666 /dev/nvidia$i c 195 $i
      RETVAL=$?
      [ "$RETVAL" = 0 ] || exit $RETVAL
   done

  mknod -m 666 /dev/nvidiactl c 195 255
  RETVAL=$?
  [ "$RETVAL" = 0 ] || exit $RETVAL
}

# Remove /dev nodes for nvidia devices
function removenodes() {
   rm -f /dev/nvidia*
}

# Start daemon
function start() {
   echo -n $"Loading $DRIVER kernel module: "
  modprobe $DRIVER && success || failure
  RETVAL=$?
  echo
  [ "$RETVAL" = 0 ] || exit $RETVAL

  echo -n $"Initializing CUDA /dev entries: "
  createnodes && success || failure
  RETVAL=$?
  echo
  [ "$RETVAL" = 0 ] || exit $RETVAL
}

# Stop daemon
function stop() {
   echo -n $"Unloading $DRIVER kernel module: "
  rmmod -f $DRIVER && success || failure
  RETVAL=$?
  echo
  [ "$RETVAL" = 0 ] || exit $RETVAL

  echo -n $"Removing CUDA /dev entries: "
  removenodes && success || failure
  RETVAL=$?
  echo
  [ "$RETVAL" = 0 ] || exit $RETVAL
}

# See how we were called
case "$1" in
   start)
       start
      ;;
   stop)
       stop
      ;;
   restart)
       stop
      start
      ;;
   *)
       echo $"Usage: $0 {start|stop|restart}"
      RETVAL=1
esac
exit $RETVAL

Copy this startup script into a file, "cuda_startup_script".

cp $HOME/cuda_drivers_2.1/cuda_startup_script cuda
cp cuda /etc/init.d/cuda
chmod 755 /etc/init.d/cuda
chkconfig --add cuda
chkconfig cuda on
service cuda start

After this, check if the device is recognized by executing

ls /dev/nv*

You should see the following in your terminal

/dev/nvidia0  /dev/nvidiactl  /dev/nvram

The "/dev/nvidiactl" indicates that the Nvidia card is recognized on your system. This is necessary when you need to compile your projects in the SDK/projects folder.

Also, on executing a lspci, you should see the following:

sudo /sbin/lspci -d "10de:*" -v -xxx
02:00.0 3D controller: nVidia Corporation Unknown device 05e7 (rev a1)
 Subsystem: nVidia Corporation Unknown device 066a
 Flags: bus master, fast devsel, latency 0, IRQ 193
 Memory at 96000000 (32-bit, non-prefetchable) [size=16M]
 Memory at 90000000 (64-bit, prefetchable) [size=64M]
 Memory at 94000000 (64-bit, non-prefetchable) [size=32M]
 I/O ports at 3000 [size=128]
 [virtual] Expansion ROM at 97000000 [disabled] [size=512K]
 Capabilities: [60] Power Management version 3
 Capabilities: [68] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
 Capabilities: [78] Express Endpoint IRQ 0
00: de 10 e7 05 07 00 10 00 a1 00 02 03 00 00 00 00
10: 00 00 00 96 0c 00 00 90 00 00 00 00 04 00 00 94
20: 00 00 00 00 01 30 00 00 00 00 00 00 de 10 6a 06
30: 00 00 00 00 60 00 00 00 00 00 00 00 0b 01 00 00
40: de 10 6a 06 00 00 00 00 00 00 00 00 00 00 00 00
50: 01 00 00 00 01 00 00 00 ce d6 23 00 00 00 00 00
60: 01 68 03 00 08 00 00 00 05 78 80 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 10 00 01 00 e0 84 00 00
80: 10 29 00 00 01 3d 00 08 08 00 01 01 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

Step 5:

cd NVIDIA_CUDA_SDK/
make

This should make all your projects in the following directory

$HOME/NVIDIA_CUDA_SDK/bin/linux/release

Execute the deviceQuery executable should show this:

./deviceQuery
There is 1 device supporting CUDA

Device 0: "Tesla C1060"
 Major revision number:                         1
 Minor revision number:                         3
 Total amount of global memory:                 4294705152 bytes
 Number of multiprocessors:                     30
 Number of cores:                               240
 Total amount of constant memory:               65536 bytes
 Total amount of shared memory per block:       16384 bytes
 Total number of registers available per block: 16384
 Warp size:                                     32
 Maximum number of threads per block:           512
 Maximum sizes of each dimension of a block:    512 x 512 x 64
 Maximum sizes of each dimension of a grid:     65535 x 65535 x 1
 Maximum memory pitch:                          262144 bytes
 Texture alignment:                             256 bytes
 Clock rate:                                    1.30 GHz
 Concurrent copy and execution:                 Yes

Test PASSED

Press ENTER to exit...

Congratulations !!! Your Nvidia Tesla C1060 is configured on your system running CentOS or a similar RHEL distro. I shall post some exercises soon. For further questions, refer to the Nvidia forums.