A key issue (at least for some paleo folks) is whether the term "introgression" gives aid and comfort to the idea that Neandertals were a distinct species from us. To the extent that we rely on hybrid zones to account for the interaction, it sure looks like we are talking about the interaction of different species. If we are really talking about subspecific interactions, then we shouldn't really be using the term "hybrid".
Even Wikipedia describes introgression as the movement of a gene "from one species into the gene pool of another" by backcrossing.
Now, what do we know about whether Neandertals and modern humans were different species?
- Speciation in primates, from commencement of prezygotic isolation to full postzygotic isolation, has taken between 1 and 4 million years to occur, considering pairs of living primate sister taxa (Curnoe et al. 2006).
- Mitochondrial DNA suggests that modern humans and Neandertals derived from a single ancestral population at most 250,000 - 500,000 years ago (the population divergence time consistent with a 350,000 - 700,000 year genetic divergence).
- Craniometrics suggest that Neandertals and modern humans were more different than many primate subspecies pairs (Harvati et al. 2004).
- Nonmetrics suggest that archaic Homo populations were no more genetically differentiated than human races (Hawks and Wolpoff 2001).
- Early Upper Paleolithic Europeans had a relatively high proportion of traits otherwise common in Neandertals.
I could go on with a few more, but you get the point: Despite their morphological idiosyncracy, genes and comparisons with other primates reject the hypothesis that modern humans and Neandertals were reproductively isolated. In that context, the morphological differences among archaic humans are (presumably) largely adaptive, and the reason that modern humans don't look like archaic humans is a matter of their different adaptations.
But if we aren't talking about different species of Homo, at least not in the sense of complete reproductive isolation, then why are we talking about introgression?
The thing is, introgression and species boundaries have emerged as different problems in the literature on genetics and biogeography.
For example, here's a passage from Dowling and Secor's (1997) review of introgression in animals:
Hybridization is defined as "the interbreeding of individuals from two populations, or groups of populations, which are distinguishable on the basis of one or more heritable characters" (Harrison et al. 1993, p. 5), and introgression is "the permanent incorporation of genes from one set of differentiated populations into another, i.e., the incorporation of alien genes into a new, reproductively integrated population system" (Rieseberg and Wendel 1993, p. 71) (Dowling and Secor 1997:595).
It is worth noting that this definition involves populations that could be defined as phylogenetic species -- populations differentiated by at least one morphological character. Of course, phylogenetic species are not evolutionary or biological species, but concerning the definition of fossil taxa like Neandertals, this is precisely the point at issue!
Another passage from Rhymer and Simberloff (1996:84) approaches the question from the standpoint of conservation genetics:
We define "hybridization" as interbreeding of individuals from what are believed to be genetically distinct populations, regardless of the taxonomic status of such populations. "Hybridization" most commonly refers to mating by heterospecific individuals but has been applied to mating by individuals of different subspecies and even of populations that, though not taxonomically distinguished, differ genetically. Arnold et al. (1991) suggest restricting "hybrid" to matings between species and using "intergrade" for matings between subspecies and "cross" or "interbreed" for matings between individuals of geographically distinct populations. Although such distinctions might clarify future discussions, all these terms seem so widely used in the literature for matings at every taxonomic level that they are unlikely to be restricted. Instead one must depend on accurate taxonomic description of the entities between which mating occurs.
Introgression is gene flow between populations whose individuals hybridize, achieved when hybrids backcross to one or both parental populations. Beyond F1 hybrids, the point at which an individual is no longer viewed as a hybrid but rather as a member of one of the parental populations that has undergone introgression is arbitrary. A hybrid swarm is a population of individuals in which introgression has occurred to various degrees by varying numbers of generations of backcrossing to one or both parental taxa, in addition to mating among the hybrid individuals themselves. Hybridization need not be accompanied by introgression; for example, offspring of hybrid matings might all be sterile. Introgression can be unidirectional, with backcrossing to one parental population only (Rhymer and Simberloff 1996:84, citations omitted).
From these passages, it becomes clear why "introgression" is used so broadly: Biologists still don't agree on what constitutes a species! This should be no surprise -- the species problem is one of the fundamental issues in biology. But it is useful to remember that fossil species are not an exceptional case.
The problem is not with defining "hybrid" or "introgression." The problem is with defining species.
The different definitions of the term "hybrid" evident in those passages also carry a lot of baggage. For the conservation geneticist, "hybridization" may mean something more or less undesirable -- something that ought to be avoided. From the point of view of defining species, "hybridization" ought to be unusual -- out of the ordinary. From the point of view of evolutionary genetics, "hybridization" may just mean reticulation -- a process making it possible for genes to move between populations that are more or less isolated. It is not just very common to talk about trans-subspecies matings as "hybridization" -- it is ubiquitous.
And for that matter, the classical genetics definition of "hybrid" has nothing whatever to do with species. Remember hybrid corn? Mendel's peas? Hybridization is about crossing lines maintained by selection. And lest we forget the etymology of "hybrid", the original Latin hybrida was the offspring of a tame sow and a wild boar. In other words, all this disagreement about the relevant taxonomic level for "hybridization" is highly subject-specific, and emerges from the conservation literature rather than from genetic principles.
I would make two observations. First, the threshold for "introgression" is arbitrary. For example, Ellstrand et al. (1999) define "introgression" as the gene flow between taxa (implying species), but discuss it mainly in connection with introgression from domesticated to wild plants, where the "species" distinction is based on the history of domestication. In the conservation literature, "introgression" concerns the detection of "alien genes", largely from invasive or cosmopolitan species (e.g., mallard genes entering American black duck populations). In the last several years of journals like Molecular Ecology there have been one or two papers per issue dealing with introgression between natural populations of animals -- mainly documenting the apparent movement of alleles between classical subspecies and morphospecies.
References to introgression are accelerating in part because of the prominent role of mitochondrial systematics in the 1990's -- people are discovering that mtDNA phylogenies don't tell the whole story of gene flow between wild populations. This is no surprise at all from an evolutionary perspective, but it has pretty clear application to the systematics of Homo, where much (so far) has ridden on the proposition that mtDNA is an accurate guide to population histories.
My second observation is that the movement of adaptive alleles from one population to another is especially likely to take the form of introgression. Genes under selection doesn't respond to population boundaries in the same way as neutral genes. The way that most people have framed the issue of the archaic-modern transition is in terms of neutral genes and population movements. But this is a poor model for the behavior of adaptive genes. This means that most people's notion of ancient population dynamics is different from the expectations of population genetics. Like the problem defining "hybrids", the mismatch of models and theory is deeply rooted in the species problem: If you think Neandertals were a different "species" from moderns, then you probably think it must follow that there was no "important" genetic interaction between the two populations.
Genetics over the past couple of decades has shown that species "boundaries" are permeable, that postzygotic isolation in mammals takes millions of years, that the flow of adaptive alleles across species boundaries in mammals is ubiquitous, and that reticulate evolution between mammalian genera is far from rare.
We could just conclude (as some of my readers have) that biology just got the species problem "wrong", and that we should be talking about subspecies instead of species. Maybe we should limit species to "really, really" isolated populations, or populations that "diverged at least 4.5 million years ago", or some other metric. There may be a lot of truth in that, but if wolves and coyotes are subspecies, cattle and bison are subspecies, and all baboons are subspecies, then I think we have to abandon the idea that species are a meaningful unit of adaptation! More to the point, most biologists use subspecies to mean "allopatric", or at least "peripatric" populations, yet hybridization and introgression commonly occur among sympatric (yet partially isolated) populations.
(UPDATE: A reader let me know that it sounds like I am actually proposing that wolves and coyotes are subspecies here. Quite the opposite -- wolves and coyotes are good species for reasons of their clear adaptive differences in sympatry. My -- possibly botched -- point is that the problem is not that the species concept is wrongly applied here; the problem is that the correct application of the species concept still gives us species that interbreed a lot! If you try to fix the problem by applying a different species concept, then we end up with a lot of very strange looking "subspecies".)
I take a different tack. There will never be any tidy solution to the species problem, because all species have unique evolutionary histories and constraints. Given these difficulties, the species status of archaic Homo populations is basically an intractable problem. That is, I am happy to suggest that archaic Homo populations correspond to classical subspecies, and as far as I know, no evidence strongly contradicts that position. But I can recognize that some people will never agree with this assignment. And from the perspective of their evolution, it just doesn't matter. Evolutionarily important gene flow occurs between mammal species, subspecies, and populations.
As you can probably tell, I have become greatly disgusted by the species problem. My reasons for this extend beyond the present discussion, but in any event I think it is a hopeless task to build any kind of consensus about the nature of fossil species.
So we have to begin by identifying patterns of interaction and gene flow. Introgressive gene flow is then a category of gene flow between differentiated populations. In particular, introgression is extensive (as opposed to merely local) and permanent (as opposed to ephemeral). Because of this, the pattern of introgression is fairly likely to involve adaptive alleles, but it need not do so. However, a widespread signature of interbreeding in neutral (or even deleterious) alleles is very likely to reflect a higher level of gene flow than would usually be indicated by "introgression". Is this a distinction without a difference? I think it's a pattern, and one that has now been replicated by several genes. It remains to be seen if it is the dominant pattern, or whether a broader pattern of genetic similarities will emerge -- but keep in mind that I think another pattern is also at play that will help to explain much.
Finding evidence for introgression in genes like MCPH1 is basically the operational procedure by which people are now looking for introgression in natural populations -- with one exception: for extant populations, we can test the genes of both populations directly. For extinct archaic populations, we can have evidence of introgression only by inference, which means that we will likely miss many true instances of gene flow from archaic humans. This does raise the risk of valuing "introgression" more substantially than it may "deserve" -- in particular, that adaptive alleles like MCPH1 will get a lot more attention than other genes that may have more ambiguity.
But I think that evidence of introgression reinforces the hypothesis that modern humans emerged in an adaptive context, making use of adaptive variation from a widespread (possibly pan-Old-World) population of archaic Homo. It's one of the two main patterns in the evolution of modern humans.
References:
Harrison RG. 1993. Hybrids and hybrid zones: historical perspective. In: Hybrid zones and the evolutionary process, ed. Harrison RG. pp. 3-12. Oxford University Press, Oxford UK.
Rieseberg LH, Wendel JF. 1993. Introgression and its consequences in plants. In: Hybrid zones and the evolutionary process, ed. Harrison RG. pp. 70-109. Oxford University Press, Oxford UK.
Dowling TE, Secor CL. 1997. The role of hybridization and introgression in the diversification of animals. Ann Rev Ecol Systemat 28:593-619.
Ellstrand NC, Prentice HC, Hancock JF. 1999. Gene flow and introgression from domesticated plants into their wild relatives. Ann Rev Ecol Systemat 30:539-563.
Rhymer JM, Simberloff D. 1996. Extinction by hybridization and introgression. Ann Rev Ecol Systemat 27:83-109.
