OUP user menu

An Integrated Software Suite for Surface-based Analyses of Cerebral Cortex

David C. Van Essen PhD, Heather A. Drury MS, James Dickson MS, John Harwell MS, Donna Hanlon MS, Charles H. Anderson PhD
DOI: http://dx.doi.org/10.1136/jamia.2001.0080443 443-459 First published online: 1 September 2001


The authors describe and illustrate an integrated trio of software programs for carrying out surface-based nalyses of cerebral cortex. The first component of this trio, SureFit (Surface Reconstruction by Filtering and ntensity Transformations), is used primarily for cortical segmentation, volume visualization, surface generation, and the mapping of functional neuroimaging data onto surfaces. The second component, Caret (Computerized Anatomical Reconstruction and Editing Tool Kit), provides a wide range of surface visualization and analysis options as well as capabilities for surface flattening, surface-based deformation, and other surface manipulations. The third component, SuMS (Surface Management System), is a database and associated user interface for surface-related data. It provides for efficient insertion, searching, and extraction of surface and volume data from the database.

The cerebral cortex is the dominant structure of the mammalian brain and is responsible for an impressively diverse range of sensory, motor, and cognitive functions. Anatomically, the cortex is a sheet-like structure whose surface area greatly exceeds the surface area of a smooth, solid shape containing it. In large-brained mammals, including humans, the cortex is extensively folded, and the pattern of convolutions varies considerably from one individual to the next. The irregularity and variability of these convolutions pose major challenges in analyzing and visualizing cortical structure, function, and development. These problems are particularly acute in view of the explosion of high-resolution data derived from many different experimental methods, particularly non-invasive neuroimaging methods such as structural and functional magnetic resonance imaging (MRI) applied to human beings and nonhuman primates.

Computational cortical cartography represents a powerful general approach to dealing with these problems by using surface-based visualization and analysis methods. The essence of the approach is to represent the cortex by explicit surface reconstructions onto which various types of experimental information are mapped. The advantages of surface reconstructions can be grouped into four main categories:

  • Visualization using multiple configurations. Once generated, surface reconstructions can be manipulated in shape to improve visualization. Commonly used configurations, besides the initial (fiducial) three-dimensional shape of the cortex, include inflated (extensively smoothed) surfaces, spherical surfaces (used for surface-based coordinates, as discussed below), and flat maps, which allow the entire hemisphere to be viewed with only modest distortions, albeit at the price of artificial cuts (akin to those used in maps of the earth's surface).

  • Localization using surface-based coordinates. It is often important to be precise about exact locations in the cortex. Surface-based coordinates (latitude and longitude on a sphere) provide a concise, precise, and objective metric that respects the topology of the cortical surface. In this respect, they offer important advantages over standard stereotaxic coordinates or geographically referenced descriptions.

  • Compensation for individual variability. Comparison of results obtained from different individuals is inherently challenging because of the combination of geographic differences (in the pattern of folding) and functional differences in the size of each cortical area (which can vary two-fold or more) and in their shape and location. Surface-based warping provides a natural and powerful way to compensate for these differences while respecting the topology of the cortical sheet.

  • Consolidation onto surface-based atlases. With the explosion of information about cortical structure and function, surface-based atlases will become increasingly important, just as atlases of the earth's surface are invaluable repositories of information about countless aspects of geography. Information from many different individuals can be brought into a common reference frame.

The many steps involved in surface-based cortical analysis can be subdivided into four main stages, each dealing with distinct types of data and having different analysis objectives. Historically, the methods used to carry out each of these processing stages initially involved manual or other noncomputerized methods that have only recently been supplanted by automated or semi-automated methods of computerized neuroanatomy.

  • Acquisition of structural data. Surface-based analysis starts with a source of structural information that can be used to infer the shape of the cortical sheet. Until the 1990s, surface reconstructions were generally based on postmortem histology, using photographs of histologic sections. The situation has changed dramatically with the advent of noninvasive imaging methods (particularly structural MRI) that can routinely be used to image the structure of cerebral cortex in human beings as well as laboratory animals.

  • Segmentation and surface reconstruction. Generating accurate surface reconstructions of cerebral cortex is a challenging problem in general, because the cortex is highly complex in shape and because the image data on which reconstructions are based are typically noisy or contain artifactual irregularities. The earliest explicit surface representations were generated by physical models.1,2 A variety of methods are now available that allow cortical segmentation and surface reconstruction with reasonable accuracy and robustness (see below).36

  • Surface reconfiguration and flattening. The earliest approaches to reconfiguring cortical shape involved generating cortical flat maps by manual methods, such as tracing contours with a pencil and tracing paper.7,8 A variety of methods are now available for inflating the cortex and for making computer-generated flat maps using various metrics for minimizing distortions.4,913

  • Mapping to a surface-based atlas. Early surface-based atlases were based on manually generated flat maps to which data were transposed manually, on the basis of geographic landmarks and approximate distances.8,14 The emergence of surface-based deformation algorithms1517 has allowed an objective approach to bringing data into register with an atlas while respecting the topology of the cortical surface.16,1819

Although the use of surface reconstructions has increased rapidly in recent years, enormous potential remains untapped in terms of ongoing neuroimaging and systems neuroscience studies that do not yet capitalize on the power of surface-based analyses. This is largely because key elements of the enabling technology have only recently become available.

The contribution of our laboratory to this effort involves the development of an integrated software suite for surface-based analyses of cerebral cortex. Our objective has been to provide software that is freely available to the neuroscience community, runs on multiple hardware platforms, and can be used to carry out the major stages of surface reconstruction and analysis efficiently and in as automated a manner as possible.

The integrated system has three main software components. The first component, SureFit (Surface Reconstruction by Filtering and Intensity Transformations), is used primarily for cortical segmentation, volume visualization, and initial surface generation. The second component, Caret (Computerized Anatomical Reconstruction and Editing Tool Kit), provides a wide range of surface visualization and analysis options as well as capabilities for surface flattening, surface-based deformation, and a number of other surface manipulations. The third component, SuMS (Surface Management System), is a database and associated user interface for dealing efficiently with the increasing number of complex data sets associated with surface-based analyses. It provides an organized framework for efficiently inserting, searching, and extracting surface and volume data from a database.

Here we provide a general introduction to and overview of the surface-based analyses that can be carried out using SureFit, Caret, and SuMS. Key algorithmic principles underlying the main processing steps are described, and selected examples are used to illustrate the methods and the utility of the overall approach. Additional information is available in the user's guide to SureFit and Caret (http://stp.wustl.edu/resources/cortcart.html), in the on help menu for Caret (http://stp.wustl.edu/caret/html4.3/reference_manual_toc.html), and in related publications.11,2022

Overview of Processing Stages in Surface-based Analysis

Surface Reconstruction

Some of the key stages of surface-based analyses are shown in a generic flow chart (Figure 1) and in an illustrative structural and functional MRI (fMRI) data set processed in SureFit and Caret (Figure 2). For each experimental hemisphere under investigation, a key initial objective is to obtain a fiducial surface reconstruction—one that represents the shape of the cortex as accurately as possible.

Figure 1

Processing stages in surface-based analysis. The upper portion shows the primary stages involved in generating an initial (“raw”) surface reconstruction from a primary source of structural data. Entries on the upper left indicate stages associated with extracting a cortical segmentation from volumetric structural data. Entries on the upper right indicate analogous stages associated with extracting and aligning section contours to represent cortical shape. The lower half shows transformations from the raw cortical surface to various alternative configurations of the same surface as well as deformations to a surface-based atlas.

Figure 2

Illustrative data set taken through the processing sequence illustrated in Figure 1. A, Coronal slice through a structural MRI volume of human left hemisphere. B, Cortical segmentation through the same coronal slice, generated using the SureFit method. C, The fiducial surface generated from this segmentation. D, Functional MRI (fMRI) data overlaid on the same structural MRI data shown in A, generated in a behavioral paradigm involving eye movements.23 E, The fMRI data painted on the cortical surface and displayed in white and light shades. F, The same fMRI data displayed on an inflated surface, with cortical geography (sulcal regions) shown in darker shades. G, Spherical map that shows fMRI data, cortical geography, and latitude-longitude isocontours. H, The same data shown on a cortical flat map. I, Spherical map of the Visible Man atlas, with fMRI data deformed to the atlas. J,.Flat map of the Visible Man atlas that includes cortical geography, deformed fMRI data projected from the sphere to the flat map, and boundaries of Brodmann's architectonic areas as mapped by Drury et al.20 The data for the individual case can be downloaded from SuMS via a hyperlink connection to http://stp.wustl.edu/sums/sums.cgi?specfile=Demo.L.full.jamia.Fig2.

The strategy for achieving this objective depends on the nature of the primary structural data. If structural MRI or cryosection data are available, the preferred strategy is to use a volume-based method such as SureFit to carry out automated segmentation and surface reconstruction (Figure 1, left side) . The segmentation process, applied to structural MRI image data of the type shown in Figure 2 A, results in a segmentation (a binary volume) whose boundary represents the shape of the cortex, as in the slice shown in Figure 2 B. This segmentation (or “reconstruction substrate”) is used to automatically generate an explicit surface representation, i.e., a wire-frame tessellation whose nodes lie on the boundary of the segmentation (typically 50,000 to100,000 nodes for a human hemisphere) and whose surface topology is defined by the links between nodes. The initial (raw) surface has a blocky appearance associated with the cubical voxels (typically 1 mm3) of the segmentation. Hence, it is slightly smoothed to generate a satisfactory fiducial surface (Figure 2 C).

If only histologic sections are available, surfaces can be reconstructed by an alternative contour-based reconstruction strategy, using options available in Caret (Figure 1, right side). This entails drawing contours along a particular cortical layer (e.g., layer 4) in individual sections, bringing contours from adjacent sections into register, and generating an explicit surface reconstruction using an automated tessellation method. Although the method for surface reconstruction is different, the outcome of segmentation-based vs. contour-based processes is fundamentally the same, that is, an explicit surface that represents the shape of the convoluted cortex.

Visualizing Experimental Data

Surface reconstructions are invaluable for visualizing many types of experimental data, ranging from minimally processed representations of primary data to highly abstracted representations. One particularly important class of experimental data involves functional activation patterns obtained using fMRI. For example, Figure 2 D shows an fMRI activation focus in a coronal slice through the intraparietal sulcus (overlaid on the structural MRI slice), generated in a behavioral paradigm involving saccadic eye movements.23 These data were then mapped from the volume onto a surface representation using a SureFit algorithm that takes local surface orientation into account when mapping volumetric fMRI data onto the surface.

Surface Reconfiguration

Once a fiducial surface has been generated, its shape can be modified into several alternative configurations that are widely used in surface-based analysis. Standard surface configurations include inflated maps (Figure 2 F), spherical maps (Figure 2 G), and flat maps (Figure 2 H). These exemplar surfaces are painted with a combination of functional and geographic data, in which darker shading indicates buried cortex; fMRI data are represented in the figure by white or light for surface nodes above a threshold activation level.

Each alternative configuration has advantages for visualization and analysis. The inflated configuration looks like a lissencephalic hemisphere and gives an intuitive sense of geographic location. The spherical configuration (Figure 2 G) allows for determination of spherical coordinates (latitude and longitude), which constitute a precise and objective localization metric that respects surface topology. The flat map shows the entire pattern in a single view (Figure 2 H), with distortions reduced by artificial cuts like those used on flat maps of the earth's surface.

Notice that the polar coordinates, while generated on the sphere, can be readily viewed and interpreted after projection to the flat map. All the surface maps, but particularly the flat map, reveal that the behavioral task (saccadic eye movements) was associated with numerous activation foci that are more efficiently and accurately visualized on a single surface than in a series of volume slices.

Surface-based Atlases

The final stage of surface-based analysis illustrated in Figures 1 and 2 involves mapping data from an individual hemisphere to a surface-based atlas. To ensure a mapping that respects the topology of the cortical surface, it is desirable to use a surface-based deformation algorithm applied to spherical maps, which are unaffected by cuts in the surface. In Figure 2, the source hemisphere (G) was deformed to the target hemisphere (I) using geographically defined landmarks in the source and target maps and a differeomorphic deformation algorithm developed by Bakircioglu et al.17 In Figure 2 J, the results of the deformation are shown by projecting the data onto a flat map of the surface-based atlas. This allows the data from a single subject to be viewed in relation to any other data registered with the atlas, such the boundaries of Brodmann's architectonic subdivisions shown on the flat map.20

Database Entry and Retrieval

Each of the display formats and the ancillary data shown in Figure 2 are useful for specific visualization and analysis options. Consequently, it is desirable to be able to access any or all of the data in an efficient, organized way, even though the data comprise more than a dozen files that represent a variety of volume data, surface geometry data, and ancillary experimental data. This access was achieved by storing the data in the SuMS database, which provides several options for searching and retrieving data. Data entry and retrieval from SuMS as well as data visualization in SureFit and Caret are greatly facilitated by grouping common families of files into organized sets that are listed in “specification files.” By selecting the appropriate volume or surface specification file (or files) of interest, a family of related files can be downloaded as a group. An especially easy and direct way to do this is by direct hyperlink connections to particular specification files in SuMS, as exemplified by the hyperlinks specified in the legend to Figure 2.

We now describe SureFit, Caret, and SuMS each in greater detail, to illustrate their functionality and the algorithms on which they are based.



SureFit includes general-purpose capabilities for volume visualization (viewing one or two volumes concurrently) and surface visualization (viewing up to three surfaces concurrently). For example, Figure 3 A shows the main SureFit window, with a coronal slice through a cortical segmentation overlaid on the structural MRI volume. The overlay option allows the relationship between the segmentation (solid shading; red in the actual view) and the intensity data to be compared directly. Two surface viewing windows are shown, one with the fiducial surface (Figure 3 B) and the other with an inflated surface (Figure 3 C). The spatial relationships among different volume and surface representations can be visualized using a “pick” option, which highlights points in each of the surface views (arrows) that correspond to the cross-hairs in the slice viewer.

Figure 3

Volume and surface visualization in SureFit. A, The main SureFit window, which can be used as a general volumetric slice viewer and in connection with the SureFit segmentation process. B, Fiducial surface reconstruction displayed in the SureFit surface viewer. C, Inflated cortical surface displayed simultaneously in a second surface viewer.

Segmentation and Surface Reconstruction

The main functionality of SureFit is to generate segmentations and surface reconstructions of cerebral cortex (like those in Figure 3) from structural MRI or other scale image data. This involves an automated sequence of image-processing operations and other functions, with only two stages (volume preparation and interactive error correction) that entail significant user interaction.

Volume Preparation

Prior to launching the automated segmentation process, several preparatory steps are carried out. These include:

  • Tagging each data set with appropriate general information (e.g., case name, region, and specific comments)

  • Orienting the volume and cropping it to the desired spatial extent

  • Identifying the anterior commissure (AC) as a key geographic landmark

  • Setting two parameters (the matter and white matter peaks in the intensity histogram) that are needed for automatic segmentation.

Automated Segmentation

Automated segmentation in SureFit is based on several structural characteristics of cerebral cortex that are schematized in Figure 4. The cortex is a sheet-like tissue, approximately constant in thickness, adjoined on its inner side by subcortical white matter. On its outer side, the pial surface adjoins cerebrospinal fluid in gyral regions, but in sulcal regions the cerebrospinal fluid gap between apposed pial surfaces may be very narrow. SureFit aims for a segmentation whose boundary runs approximately midway through the cortical thickness (i.e., cortical layer 4), because this represents the associated cortical volume more accurately than do segmentations running along either the inner or outer boundary.4 To this end, SureFit generates probabilistic representations for both the inner boundary and the outer boundary, using cues related to image intensity, intensity gradients, and the spatial relationships between the inner and outer boundaries. One general strategy is to postpone deterministic (yes/no) decisions that create binary volumes as long as possible, until various probabilistic representations have been generated and appropriately combined.

Figure 4

Schematic representation of key structural features of cerebral cortex that are relevant to the SureFit segmentation algorithm. The cortical sheet is approximately constant in thickness and adjoins white matter on its inner boundary. Its outer (pial) boundary adjoins cerebrospinal fluid in gyral regions and oppositely oriented cortical sheet in sulcal regions.

Key stages in the SureFit segmentation process are illustrated in Figure 5 for a slice through lateral temporal cortex of a structural MRI volume (Figure5 A). This case was chosen because the image data are relatively noisy and low in contrast, making automated segmentation an inherently challenging process. The intensity histogram (Figure 5 B) for this volume shows a characteristic pair of intensity peaks, one associated with white matter and the other with matter.

Figure 5

(Opposite) Stages of cortical segmentation in SureFit. A, Coronal slice through occipital cortex in a human structural MRI volume. B, Intensity histogram for the image volume shown in A. Arrows indicate values for parameters used to generate intensity-based probabilistic maps of cortical regions or boundaries. C, An intensity-based probabilistic map of the inner boundary. This is based on a Gaussian intensity transformation: Embedded Image where I(n) is the intensity value for the th voxel, Ipeak is an estimate of the most likely intensity value for the tissue type or boundary, and the standard deviations σlow and σhigh are related to the noisiness of the image data. D, Map of the magnitude of the intensity gradient. E, The composite inner boundary map. One component of this composite map is the intensity-based map of the inner boundary (C). Another component is derived by determining where the intensity gradient is intermediate in magnitude (made explicitly by a Gaussian intensity transformation analogous to that in C) and pointed opposite to the gradient of a probabilistic map of matter (not shown; also based on a Gaussian intensity transformation). A third component selectively emphasizes regions near the crowns of gyri (where the underlying white matter is notably thin) by testing for regions containing two gradient-based inner-boundary domains that are in close proximity but pointed in opposite directions. F, Intensity-based map of the outer boundary. G, Map of the outer boundary in sulcal regions, based on evidence for two inner boundaries that are pointed in opposite directions and are each displaced about one cortical thickness (3 mm for human cortex) from the voxel being tested. H, The composite outer boundary map, derived from the intensity-based outer boundary map (F), the sulcal outer boundary (G), and gradient-based cues analogous to those used for the inner boundary. I, Binary map of cerebral white matter, based on thresholding the intensity volume and removing various noncerebral structures. J, A map of positions along the radial axis, generated by blurring both the inner and outer boundary maps, normalizing the output by dividing the difference by the sum at each voxel, and assigning a maximal value to contained in the interior of cerebral white matter (i.e., in an eroded version of the image shown in I). K, The initial segmented volume obtained by thresholding the radial position map. L, The initial segmented volume superimposed on the original intensity volume.

Along the inner (–white) boundary, the image intensity should be approximately midway between the peaks for gray matter and white matter. This intensity-based cue is made explicit using a Gaussian intensity transformation, whose peak represents the most likely inner boundary intensity (inner boundary arrow in Figure 5 B) and whose standard deviation reflects the noisiness in the image data (see legend for details). The resultant inner boundary intensity transformation has local maxima running approximately along the inner boundary, but there are many irregularities owing to fluctuations in the image data. Other cues for the inner boundary are related to the intensity gradient, whose magnitude is shown in Figure 5 D. The gradient along the inner boundary is steeper than in neighboring gray matter and white matter, but it is more shallow than in other regions (e.g., along parts of the outer boundary). This information, along with cues related to the direction of the intensity gradient, is combined with the intensity-based map in Figure 5 C to generate a composite inner boundary map (Figure 5 E) that is more reliable than the individual contributing maps (see legend to Figure 5 for details)

For the outer (pial) boundary, one cue is based on image intensity (Figure 5 F), using a Gaussian intensity transformation centered at the outer boundary arrow in Figure 5 B. This intensity-based map has prominent local maxima along gyral boundaries, where matter adjoins cerebrospinal fluid. In sulcal regions the local maxima are less pronounced, because a prominent cerebrospinal fluid gap is generally lacking. Instead, evidence for the outer boundary in sulcal regions is collected using a customized filtering process whose output is high along regions midway between two oppositely oriented inner boundaries (Figure 5 G). The composite outer boundary map (Figure 5 H) reflects a combination of the cues in Figures 5 F and 5 G and gradient-based cues analogous those used for the inner boundary.

In addition to the probabilistic inner and outer boundary maps, a segmented representation of cerebral white matter is generated (Figure 5 I). This entails thresholding the intensity volume at a level that fills most of the white matter, disconnecting noncerebral structures (brainstem, cerebellum, skull, and eye), transecting mid structures, and removing the extraneous regions by a flood-filling process. The brainstem is disconnected using a diagonal transection applied near the pons (based on its location relative to the anterior commissure). The eye and skull are disconnected by automatically identifying and removing the high-intensity (fatty) region behind the eyeball.

In Figure 5 J, the maps of inner and outer boundaries are combined to generate a smooth map of position along the radial axis (i.e., the axis deated by arrows in Figure 4). This involves taking the difference between blurred versions of the inner and outer boundary maps (and also assigning maximal intensity to regions near the core of cerebral white matter). The resultant radial position map is white in the interior, black on the exterior, and smoothly graded in intensity across the cortical thickness. Thresholding the radial position map at an intermediate intensity level generates a segmented volume (Figure 5 K) whose boundary tends to run midway through the cortical thickness, as can best be appreciated by viewing the segmentation superimposed over the structural MRI intensity volume (Figure 5 L). The initial segmentation is used as the substrate for generating an explicit surface reconstruction.24

Error Detection and Correction

Topological errors (“handles”) in the initial segmentation are typically attributable to noise, large blood vessels, or regional inhomogeneities in the structural MRI volume, or a combination of these. Errors are localized by inflating the initial surface reconstruction to a highly smoothed ellipsoidal shape and using the orientation of surface normals to identify “crossover” regions where the surface is folded over on itself. Clusters of surface nodes associated with cross-overs are mapped from the fiducial surface into corresponding voxel clusters in the volume. If the error is a bridge across opposite banks of a sulcus (an “exohandle”), correction entails removing voxels from the bridge. If, instead, the error is a hole in the white matter between two sulci, an “endohandle,” correction entails adding voxels to the hole. The automated error correction process tests for both exohandles and endohandles in the vicinity of each location determined to contain an error.

The localized patches used for these tests conform to the shape of temporary segmentations that are based on different threshold levels for the radial position map. If the trial patch reduces the number of topological handles in the segmentation, as determined from an Euler count applied to the volume,25 it is accepted as a permanent correction and the process moves on to the next error patch. The automated error correction process sometimes fails, especially for handles that are notably large or irregular. Residual errors can be removed by interactive editing, which allows voxels to be added or removed one at a time or in small clusters using dilation or erosion steps within small masked regions.

The processing time needed for automated segmentation in SureFit is less than an hour for the initial segmentation plus several hours for automated error correction when run on an SGI Octane. Processing times can be significantly faster on PC systems running Linux.

Mapping Geographic and Functional Data

Once a satisfactory segmentation is achieved, regions of buried cortex are automatically identified. This entails a combination of dilation, erosion, and volume subtraction operations to generate a volume that includes sulcal but not gyral cortex. Nodes of the reconstructed surface that intersect this sulcus-only volume are identified as buried cortex and are shaded to provide a representation of sulcal versus gyral geography (Figure 2 E to J).

The process of mapping fMRI data onto surface reconstructions takes advantage of information about local surface orientation. For each surface node, the fMRI image volume is convolved with an ellipsoidal filter centered on that node and oriented parallel to the local surface tangent. The output values are converted to RGB color values according to a predesignated color look-up table and are stored in an “RGB_paint” file format for viewing in SureFit or Caret. In addition, the outputs are stored as scalar values in a “metric file” format for subsequent visualization and analysis in Caret.

Volume and Surface Specification Files

SureFit generates volume and surface specification files at the end of the automated segmentation process. The volume specification file lists the structural MRI volume, the segmentation files, and several key intermediate files. The surface specification file includes both VTK file format (Visualization Tool Kit26) and the Caret geometry file format (in which node coordinates and node topology are stored in separate files) as well as paint files that represent cortical geography. This facilitates data entry into SuMS as well as visualization and surface-based analyses in Caret.


Surface Visualization

Caret includes numerous options for surface visualization, manipulation, and other surface-based analyses. We illustrate some of these visualization capabilities using a surface-based atlas for the macaque monkey as an exemplar data set.

The main Caret screen (Figure 6 A) includes a surface visualization window as well as menu options along the top and a status display row along the bottom. The initial step in loading data is to select an appropriate surface specification file (using the “Open Specification File” option in the File submenu shown in Figure 6 A). This brings up a dialog box (Figure 6 B) for choosing the subset of data files to be loaded. Surface geometry is determined by selecting one topology file and two coordinate files—one as the reference (REF) configuration and another as an auxiliary (AUX) configuration. Other available file types are grouped according to the type of experimental data.

Figure 6

A, The main Caret screen, showing a cortical flat map from the macaque surface-based atlas. The tear-off menu on the upper left shows options available in the File menu. B, The File Selection dialog after a specification file for the macaque surface-based atlas has been selected. Files listed in the specification file are displayed in appropriate categories. Default file sections in each category are indicated by depressed buttons on the left. The currently loaded border file or border projection file can be toggled on and off using the “B” (Borders) toggle button. Selection of the geography (“G”) toggle button colors nodes according to the node identities and the color assignments contained in the currently loaded “area color file.”

The specification file in Figure 6 B lists four classes of ancillary data—latitude-longitude files, paint files, border files, and atlas files—that are described more fully below. To facilitate the selection process, topology files are identified as CLOSED, OPEN, or CUT, and coordinate files are identified by their configuration (FIDUCIAL, INFLATED, SPHERICAL, or FLAT). Additional information can be obtained by pressing the query (“?”) button, which brings up information in the file header, and by pressing the “List” button for paint and border files, which shows the different labels or categories contained in the file.

Once the selected data files are loaded, numerous menu options are available for changing the surface configuration, the viewing perspective, and the information displayed on the surface or in relation to it. In Figure 6 A, the flat map configuration has been selected (using the “AUX” toggle button), surface nodes were painted to show cortical geography using the “G” (geography) toggle button, surface tiles were filled using the “Tile” toggle button, and latitude and longitude isocontours were displayed using the “B” (borders) toggle button.

The ancillary data shown in Figure 6 A were generated using several of the data analysis options available in Caret. The latitude and longitude isocontours were generated by a process that requires a spherical map to be loaded in the AUX configuration. The contour points were then converted from the initial configuration-specific three-dimensional coordinates to a “border projection” file format. This allows contours to be viewed in relation to any surface configuration (e.g., the flat map in Figure 6 A) by expressing contour point locations in relation to the nearest tile (i.e., in “barycentric” coordinates27).

To generate the map of cortical geography shown in Figure 6 A, the sulcal pattern was initially represented by a map of folding (mean curvature) computed for the fiducial surface but displayed on the flat map. Border contours were drawn around the perimeter of each sulcus as visualized on a flat map. Nodes enclosed by one or another closed sulcal boundary were automatically identified, and the assignments were stored in a paint file that contains up to five separate categorical types of information for each node.

Figure 7 illustrates several additional Caret visualization options that facilitate comparisons between different surface configurations and different partitioning schemes. On the left (Figure 7 A and B) is the Felleman and Van Essen14 areal partitioning scheme for visual areas; on the right (Figure 7 C and D) is a probabilistic atlas based on the Lewis and Van Essen areal partitioning scheme,18,28 with a few specific areas identified (V2, V4, TE1–3). A blending option allows the visual areas and the geography map to be viewed concurrently. The probabilistic atlas was generated from architectonic maps of visual areas in five macaque hemispheres by deforming them to the macaque atlas (discussed below). In this scale figure, brightness reflects the fraction of cases that have the same areal identity at a given location; on the Caret screen, different hues are used to discriminate the individual areas.

Figure 7

Visualization and identification of multiple data types simultaneously loaded into Caret. A, The fiducial configuration of the macaque surface-based atlas, showing visual areas in the Felleman and Van Essen14 partitioning scheme. B, Flat map showing the same set of visual areas. C, Probabilistic map of visual areas in the Lewis and Van Essen28 partitioning scheme, shown on the fiducial configuration. Brighter shading indicates a higher fraction of cases associated with the same visual area (which appear in color on the Caret screen). D, Flat map showing the Lewis and Van Essen areas. In A through D, a node in one of the temporal lobe areas was highlighted (white square, which appears green on the Caret screen). E, The pop-up Caret status message window, displaying information about the highlighted node. The three-dimensional position represents the coordinates of the nodes in the REF configuration (here, the fiducial surface). The two-dimensional position represents the coordinates in the AUX configuration (here, the flat map in Cartesian standard coordinates with the origin at the ventral tip of the central sulcus). The latitude (−20.5 ) and longitude (−101.9 ) represent values in the currently loaded latitude-longitude file, which was previously generated using a spherical map with areal distortions minimized and oriented to the spherical standard coordinate frame (ventral tip of the central sulcus at the lateral pole). The fourth row displays information from the currently loaded paint file (LOBE.TEMP signifying the temporal lobe; “???” signifying no entry for that column of the five-column paint file). The last row displays information from the currently loaded atlas file. This contains five entries, only three of which are shown here, indicating that the selected node is an architectonic area TE1-3 in some but not all cases from the atlas file data set.28 The data sets illustrated here can be downloaded for viewing in Caret from http://stp.wustl.edu/sums/sums.cgi?specfile=2001-03-02.79O.R.LEWIS_VE_ON_ATLAS.spec.

An option for identifying nodes using the mouse cursor (enabled using the “IDN” toggle button) provides a useful way to assess relationships between different configurations and to extract textual information about each node. Selecting a node in one configuration (e.g., the flat map) highlights that node when the view is switched to other configurations or to a different node coloring scheme (white squares and arrows in Figure 7 A to D). In addition, the pop-up Status Message window (Figure 7 E) shows many types of information about the selected node. This includes the node identity plus its spatial location in three different coordinate systems—three-dimensional stereotaxic coordinates (if the fiducial configuration is loaded in the REF configuration), Cartesian map coordinates (if a standard flat map is loaded in the AUX configuration), and latitude and longitude (if a latitude-longitude file is loaded). Additional information includes node attributes contained in the currently loaded paint file (in this example, assignments of cortical geography, lobe, functional area, and modality) and atlas file (in this case, the areal identity for each hemisphere contributing to the probabilistic atlas).

Caret includes several data formats for visualizing and analyzing fMRI and other spatially complex patterns of neuroimaging or neuroanatomic data (cf. Figure 2). Data types that can be assigned to nodes and used to color the surface include RGB_paint and metric files, which are generated by SureFit when it maps fMRI data from an initial volume representation onto a surface. A file type that is particularly useful for surface-based deformations is the “activation file,” which is generated by applying an adjustable threshold to the data in a metric file. Cell files represent data related to the spatial distribution of labeled neurons or other discrete data. Cell files can be converted to representations of cell density to facilitate quantitative analyses of connectivity patterns. Foci files represent an alternative type of data related to the centers of PET or other neuroimaging data. Data points in foci files can be represented with associated uncertainty limits reflecting individual variability and spatial uncertainty in mapping data based on stereotaxic coordinates.29

Caret also contains options for generating, visualizing, and saving various types of information about surface geometry. These include representations of mean curvature (folding), intrinsic (Gaussian) curvature, distortion (AUX surface compared with REF surface), and geometric cross-overs (creases or topological handles in the surface).

Modifying Surface Geometry

Caret includes numerous options for modifying surface configuration (shape) or topology, or both, many of which were used in generating the data shown in , Figures 2 5 6 and 7. These are grouped into six general classes—smoothing, geometric projections or transformations, contour-based reconstruction, automated flattening and distortion reduction, interactive surface editing, and surface-based deformation.

  • Smoothing. Smoothing operations include conventional smoothing (iteratively moving each node to the average of its neighbors), targeted smoothing (preferentially smoothing nodes that are highly distorted relative to the reference surface), and surface inflation (smoothing coupled with preferential expansion of nodes close to the center of gravity, to accelerate the smoothing of highly convoluted surfaces).

  • Geometric transformations and projections. Options for geometric projection or transformation of surfaces include translation, scaling, rotation, and mirror-reflection; projection of a surface radially outward to a sphere or ellipsoid; and projection of a sphere to a plane (by a topology-preserving transformation if the sphere has a hole oriented along the positive z direction).

  • Surface reconstruction. Contour-based surface reconstruction is used when the input data are a series of section contours rather than pre-existing surfaces generated by SureFit or other automated volume segmentation and surface reconstruction methods. Caret includes options for aligning individual section contours and generating surface reconstructions from the stack of aligned contours, using the Nuages algorithm.30

  • Flattening. Automated flattening and distortion reduction involves two main stages of processing. In the first stage, the fiducial surface is reconfigured by smoothing or geometric projection steps to attain a spherical or flattened shape, depending on the desired geometry. Multi-resolution morphing is then applied to generate a minimally distorted flat map or spherical map. In both cases, the underlying algorithm10,11 involves resampling, downsampling, and morphing (iterative application of forces to each node to reduce distortions relative to the fiducial configuration). Surface resampling represents the initial reference and auxiliary configurations with a hexagonal array of nodes that are uniformly spaced in the auxiliary configuration. Once resampled, a surface can be downsampled (by deleting alternate nodes) to generate progressively coarser representations of the original shape. Multi-resolution morphing starts at the coarsest level of resampling, and is then reapplied after successive stages of upsampling to a finer resolution. The cycle is repeated until the overall distortions have reached an acceptably low level. When Caret is run on an SGI Octane, this takes several hours for a complete human hemisphere.

  • Interactive editing. Interactive editing of surface geometry includes a number of options—drawing and applying cuts to the surface, deleting regions that have been disconnected by cuts, repositioning individual nodes, connecting or disconnecting selected node pairs, and processing the modified topology file to identify edge nodes and establish a consistent orientation of surface normals.

  • Surface-based deformation. Options for surface-based deformation in Caret allow the map (the source) to be deformed to another (the target) while constrained by explicitly designated landmarks. The algorithm for deforming flat maps was developed by Joshi and Miller.15,21 Landmarks are drawn along what the user judges to be corresponding locations on the source and target maps. The landmark contours are resampled to establish corresponding numbers of landmark points on each source and target landmark contour. The landmarks are then used as constraints for a two-dimensional differeomorphic deformation algorithm. This algorithm computes solutions to a transport equation in which the landmark matching is constructed to minimize a running smoothness energy on the velocity field. Additional data (as a coordinate file, border file, or activation file) are carried passively with the deformation.

Spherical maps can be deformed using an algorithm developed by Bakircioglu et al.17 The basic strategy is similar to that for flat deformations, but it entails using Laplacian differential operators constrained to the tangent space of the sphere and basis functions that are expressed as spherical harmonics. We have recently developed an improved method that is based on landmark-constrained smoothing and morphing of coordinates. As a practical matter, landmarks can be drawn on flat maps of the source and target hemisphere, where visualization is easiest, then projected to the corresponding spherical maps. Conversely, once deformations have been applied to the spherical maps, it is generally preferable to view the results on flat maps (cf. Figure 2 I and J).


The architecture of SuMS (Surface Management System) comprises four main components:

  • A Java-based database structure designed specifically for cortical surface representations

  • A high-capacity, secure data cache capable of storing large amounts of data

  • A database manager (Sybase SQL Server) for coordinating the file cache and database access

  • Search and retrieval capabilities using either a web browser (WebSuMS) or the downloadable SuMS Client software (a downloadable graphical user interface that runs on Java-compatible platforms).

For data entry it is currently necessary to use the SuMS Client. For most purposes WebSuMS is the easier to use, but it does not yet include all the capabilities of the SuMS Client.

Data Entry

Data entry in SuMS is a simple process in which the names of the volume or surface specification files intended for insertion are entered into the appropriate dialog box in the SuMS Client. For successful data entry, all files must be present in the directory locations listed in the specification file and the requisite metadata must be included in the appropriate file headers. Files generated using SureFit and Caret (v4.3 or higher) automatically include this information in the individual files or in the specification files. At the outset of the data entry process, the metadata for all files are checked for completeness, and prompts are given if required information is missing. A cross-checking process avoids duplicate insertion of files and ensures that only new data sets are added to SuMS.

Data Search and Retrieval

WebSuMS provides several convenient ways to identify and download files of interest, starting from the SuMS search page (http://stp.wustl.edu/sums/sums.cgi), shown in Figure 8 A). One option is to enter the name of a particular volume specification file or surface specification file into the appropriate dialog box, if the name (or portion of the name) is known. Another is to search for particular files using one or more search criteria shown in the bottom half of the window.

Figure 8

Screen displays using WebSuMS to access the SuMS database. A, The WebSuMS search page (http://stp.wustl.edu/sums/sums.cgi) can be used to identify specification files in the SuMS database that meet whatever search criteria are entered by the user. This can be a partial or complete name of a particular volume or surface specification file (top rows) or a particular case from the currently selected species (macaque in this example) plus the option of adding criteria such as key words in the comment section of any data file or in the data fields of paint files, border files, and cell files contained in SuMS. B, The current listing of surface-based atlas specification files obtained by selecting the “View Atlases” menu button. C, The specification file obtained by selecting one of the macaque atlases (LEWIS_VE_ON_ATLAS.spec). The entire set of files can be downloaded as a group, then viewed in Caret after being uncompressed. For WebSuMS, the user interfaces with the SuMS Web Server via HTTP (HyperText Transfer Protocol) using standard HTML forms and scripts based on CGI (Common Gateway Interface). For the SuMS Client, Remote Method Invocation (developed by Sun Microsystems) is used as the interface with a JDBC (Java Database Connectivity) driver. One Java application or applet (the SuMS client in this context) can call the methods of another Java application (the SuMS server) running on a different host machine.

A third option is to select View Atlases near the top of the window. This brings up a list of four standard specification files for the macaque and human atlases (Figure 8 B). Selection of the MACAQUE_ATLAS. spec brings up the list of files shown in Figure 8 C. These can be downloaded as a group, then viewed in Caret after the files are uncompressed.

An even simpler and faster way to retrieve files is to make a direct hyperlink connection to a particular specification file in the database, starting from a separate application such as an on journal article, PDF file, or e-mail message. For example, the data for the individual hemisphere illustrated in Figure 2 can be immediately accessed by a hyperlink connection to the appropriate specification file (http://stp.wustl.edu/sums/sums.cgi?specfile=Demo.L.full.jamia.Fig2.spec). This directly links to a specification file containing the relevant data files. Selecting the “Download All” option downloads these files to the desired directory in a single step.

The search process in the SuMS Client is similar, but is more flexible because it includes an intermediate repository. The first stage is to choose a combination of search criteria for identifying files of potential interest and to initiate the search process. The second stage is to view the results of the initial search and to select any or all of these files for placement in a search repository. The search repository is a listing of files provisionally targeted for downloading. The contents of the repository (akin to a “shopping cart”) can be expanded by repeating the search process using a different set of criteria. The third stage is to select the final set of files from the search repository and then to initiate the download process.

Access Control

SuMS uses a security model based on the designation of owners, readers, and writers for each file in the database. This ensures that private data are protected, public data are freely available, and any data can be made available to some investigators (e.g., collaborators) but not others. This security scheme is flexible enough for any owner to allow access to restricted data sets on an individual-user or group basis.


Surface-based analyses have tremendous potential for enhancing progress in understanding cortical structure, function, and development in health and disease. If widely adopted, the software tools described here for computational cortical cartography, along with related tools developed in other laboratories, will aid in capitalizing on this potential. The ultimate measures of progress will, of course, be based on the actual scientific and medical discoveries that benefit from surface-based analyses.

In addition, several more proximate measures will be of interest to monitor over the next several years. These include the pace with which the neuroscience community adopts cortical surface reconstructions as a standard way to analyze and display experimental results from individual subjects; adopts surface-based atlases and probabilistic representations as general strategies for bringing experimental data into register on a common substrate; adopts surface-based coordinates as a concise and objective primary metric for describing locations on the cortical surface, to complement the use of Talairach stereotaxic coordinates; and uses surface visualization software and databases of surface representations as a supplementary option for assessing and visualizing experimental data associated with published studies.

SureFit, Caret, and SuMS have been under development in our laboratory for a number of years. However, they have only recently reached a state of maturity that allows semi-automated execution of a complete processing sequence, proceeding from primary structural and functional data to data sets that are represented on cortical flat maps, transformed to surface-based atlases, and stored in a database. Some of the functionality of the SureFit/Caret/SuMS suite is currently unique, but most of its capabilities are shared by one or more other brain-mapping software packages. These include FreeSurfer,4,13 Brain Voyager,5,31 and mrGray.6 It is useful to discuss briefly the key issues involved in evaluating any of these packages, although detailed analysis and comparison are outside the scope of this discussion.

Segmentation and Surface Reconstruction

Two primary sets of issues are involved in evaluating different surface reconstruction methods:

  • How robust, easy to use, and accessible is the software for generating surfaces and mapping data onto surfaces?

  • How accurate are the surface representations and how accurately are experimental data mapped onto the surfaces?

Structural image data obtained in neuroimaging and neuroanatomic studies vary widely in overall quality. This is attributable to differences in image contrast, noise, and regional inhomogeneities arising from the particular methods or devices used for image acquisition and from the individual subjects under study. Accordingly, each method of segmentation should be tested for its robustness and accuracy, using data sets that span a wide range in quality. This has yet to be done systematically, because of the newness of the methods, the inherent complexity of the data, and the lack of generally accepted standards for evaluation.

One important metric is whether the segmentation/surface is topologically correct. Topological errors (handles) are particularly deleterious for flattening and subsequent processing stages, unless they are small and outside the main region of experimental interest. Another important metric is spatial accuracy, i.e., how close the surface runs to the desired target boundary. This is especially important if the objectives of the experimental analysis are sensitive to modest but systematic biases or to less frequent but larger deviations from the desired trajectory. However, assessment of spatial accuracy is inherently problematic, because a “ground truth” or “gold standard” that precisely represents the target layer is generally not available for structural MRI or cryosectional image data.

A reasonable alternative strategy is for one or more expert anatomists to draw contours along the target layer in a number of selected regions in which the trajectory can be confidently estimated. The distance from each point on the target contour to the nearest point on a fiducial surface reconstruction can then be determined and displayed as a histogram, and the distribution of errors can be expressed by measures such as the error and standard deviation. If those who draw the target contours are unaware of the results of any particular segmentation, this approach can serve as an objective basis for evaluating and comparing the performance of different segmentation methods.

Surface Manipulation: Flattening and Deformations

A number of methods are currently available for generating cortical flat maps and spherical maps from fiducial surface representations. Given that the fiducial surface contains a complex pattern of intrinsic (Gaussian) curvature,29 significant areal distortions are inevitable on any flat map or spherical map representation. Nonetheless, it is desirable to minimize these distortions and to quantify the magnitude and distribution of residual distortions.

Residual distortions can affect surface-based analyses in two major ways. First, they affect visual impressions about the relative sizes of different regions and intracortical distances between regions. This is analogous to how distortions on earth maps affect impressions about the relative sizes of different continents (e.g., Greenland vs. South America). Fortunately, compensation for such perceptual biases can be largely made by analyzing and reporting surface areas and geodesic distances determined on the fiducial reconstruction.

Another problem is that surface-based warping from one hemisphere to another can be affected by distortions, especially if the pattern of distortions differs on the source and target maps. This puts a premium on minimizing distortions on whichever surface configuration (sphere or flat map) is used as the substrate for surface-based deformations. Histograms of areal distortion for different surface nodes provide an objective basis for comparing different flattening algorithms.

An inherent advantage of surface-based deformations is that the process respects the topology of the cortical sheet in compensating for individual variability. For this reason, surface-based deformations should in principle achieve substantially better registration than volume-based deformations, even if the volumetric approach uses a very high-dimensional parameter space to constrain the deformation. This prediction is supported by initial comparisons of surface-based vs. volume-based approaches to the registration problem.16,20 However, the issue is of sufficient general importance that more extensive quantitative comparisons are strongly warranted.

As progressively more data are mapped to surface-based atlases, it will be feasible to generate an increasingly diverse and rich set of probabilistic representations. One type of representation, relatively close to the primary data, involves probabilistic maps of fMRI activation patterns from multiple subjects carrying out similar or identical behavioral tasks16 (see Figure 2 above). Given the enormous diversity of behavioral tasks that have been or will be used on multiple subjects in fMRI studies, the number of such probabilistic maps is effectively open-ended.

Another type of representation, involving greater abstraction from the data, involves probabilistic maps of cortical areas whose boundaries are estimated from functional, architectonic, or other types of experimental data (cf. Van Essen et al.18). To the extent that most or all cerebral cortex is in fact divisible into genuine, well-defined subdivisions, it might be hoped that probabilistic maps of cortical areas would converge toward a single, consensual representation. However, a more likely scenario is that numerous alternative partitioning schemes will remain in widespread use. This will lead to a multiplicity of probabilistic maps based on different schemes, different criteria used to deform individual data sets to a surface-based atlas, or different substrates used as the target surface-based atlas.


These considerations emphasize the pressing need for continued progress in several aspects of computerized cortical cartography. One need is to develop improved methods for evaluating the quality of registration of individual data sets to an atlas, on the basis of objective measures that can be applied to a diversity of data types. Another need is to improve the methods for surface-based deformation. For example, it is desirable to have hybrid methods that combine aspects of the landmark-based approach described here with approaches based on a continuous valued representation as used by Fischl et al.16 A third need is to enhance interoperability, so that surface-based analyses for any given data set can easily migrate from one software suite to another. This will facilitate comparisons among data sets obtained in different laboratories and will allow a more extensive sets of analyses to be carried out on data sets of particularly broad interest.

A fourth need is for ongoing enhancements in the database infrastructure, for coping with the flood of surface-related data. We believe that the SuMS database described here has considerable promise as an initial effort in this direction. If the concept is indeed successful, however, it will need to be scaled up by several orders of magnitude to cope with the estimated 100,000 cortical surface reconstructions per year that may emerge in this decade from brain-mapping efforts in basic and clinical laboratories around the world.22 The advantages of bringing these data under the umbrella of one or more databases will become increasingly evident, just as it has in such other scientific arenas as genomics and protein structure.3234 This will impose substantial pressures for increased data capacity, network speeds, and a richer array of search capabilities.


The authors thank the many beta testers at Washington University and elsewhere whose bug reports and constructive suggestions have greatly helped to improve Caret, SureFit, and SuMS.


View Abstract