The evolution of enclosed nesting in passerines is shaped by competition, energetic costs, and predation threat

Many avian species breed in enclosed nests that may provide better protection against predation and climatic conditions compared to open nests and are generally associated with larger clutch sizes and slower offspring growth. Here we show that different enclosed nesting strategies are each linked to behaviors with very different costs and benefits on a macroevolutionary scale. Using a detailed dataset of nest structure and location from the order Passeriformes, we employed phylogenetic comparative methods to evaluate (1) how predation, competition, design complexity, and energetic costs have shaped evolutionary transitions between different nesting strategies, and (2) whether these strategies also have distinct relationships with life-history traits. We find that flexible strategies (i.e., nesting in both open and enclosed sites) as well as energetically demanding strategies are evolutionarily unstable, indicating the presence of underlying ecological tradeoffs between antipredator protections, construction costs, and competition. We confirm that species with enclosed nests have larger clutch sizes and longer development and nestling periods compared to open nesters, but only species that construct enclosed nests rather than compete for preexisting cavities spend more time incubating and are concentrated in the tropics. Flexible strategies prevail in seasonal environments and are linked to larger clutches—but not longer development—compared to nesting in the open. Overall, our results suggest that predation, competition, and energetic costs affect the evolution of nesting strategies, but via distinct pathways, and that caution is warranted when generalizing about the functions of enclosed nest designs in birds.