Compute Resources

Rapid advancements in imaging technology have provided researchers with the ability to produce very high-resolution, time-varying, multidimensional data sets of the human brain. The complexity of the new data, however, requires immense computing and storage capabilities. To meet this requirement, LONI uses a 306-node, dual-processor SUN Microsystems V20z cluster, one of the largest V20z installations in the nation. Each compute node has dual 64-bit 2.4 gigahertz AMD Opteron processors with four gigabytes of memory. In addition, LONI has a 58-node Dell development cluster, with each node using dual 64-bit 3.6 gigahertz Intel EM64T processors and four gigabytes of memory. To augment the facility's cluster resources, LONI has a sixty-four 400MHz MIPS-R12000 SGI Origin 3800 SMP supercomputer with thirty-two gigabytes of memory and a thirty-two 800MHz IA-32 processor SGI 1200 cluster with eight gigabytes of memory. A 32-processor SGI Onyx2 Reality Monster with sixteen gigabytes of memory and a 6-processor SGI Onyx2 with eight gigabytes of memory are utilized to drive graphics-intensive applications and interactive real-time multidimensional visualization of structural brain models and volumetric datasets. The separation of the graphics and computation systems allows for function-specific optimizations and the redundancy provides fault tolerance by eliminating any single point of failure.

To facilitate the submission and execution of compute jobs in this heterogeneous compute environment, SUN's Grid Engine (SGE) is used to virtualize the resources above into a compute service. A grid layer sits atop the compute resources and submits jobs to available resources according to user-defined criteria such as CPU type, processor count, etc. The laboratory has successfully integrated the latest version of the LONI Pipeline (http://pipeline.loni.ucla.edu) with SGE using SUN’s Java DRMAA bindings. The bindings allow jobs to be submitted natively from the LONI Pipeline to the grid without the need for external scripts. Furthermore, the LONI Pipeline can directly control the grid with DRMAA, significantly increasing the operating environment’s versatility and efficacy, and improving overall end-user experience.

Numerous desktop workstations are available. These include various SUN, SGI, Linux, as well as Windows and Macintosh machines. These workstations are networked to the SAN via switched 100baseT and Gigabit ethernet. This configuration allows the sharing of storage and other peripherals between machines on the LONI network. A queuing system, SGE, delivers effective use of the many processors available, whether in the supercomputers or in the workstations to load balance and distribute compute-intensive jobs. Document preparation and production are supported with a variety of HP laser printers, a Tektronix Phaser 7700 color printer, an HP DesignJet 3500CP poster printer and a publication-quality Fujix Pictography 3000 color printer.

LONI’s efforts to implement redundancy extend beyond the hardware detailed above. The laboratory has configured two Linux Virtual Servers (LVS), primary and backup, for high-availability web and database service provision. In conjunction with dual HTTP, application and MySQL servers, this LVS solution provides two critical features—load balancing and high availability. A round-robin load balancing implementation is currently used such that if the processing load on one server is heavy, incoming requests, be it HTTP, JSP or MySQL, are forwarded to the next available server by the LVS nodes. Listeners on one virual server monitor the status and responsiveness of its counterpart. If a failure is detected, the available server assumes control and request forwarding for the entire LVS environment.