Agility has been a long and hard fought lesson learned over the past few decades. In CS classes we're told about ancient DOD style projects with names like Wooden Man, Stone Man and Iron Man. The pattern is pretty clear: Requirements Definition, Design, Implementation, Maintenance and Disposal (though we don't learn much about the last 2 as they're naturally somewhat difficult to teach given the time constraints of a semester).
When this theory hits practice what I've found is that the value of detailed up-front design is inversely related to how far into the future you're trying to design. Put another way: across many disciplines, not just our own, our ability to see into the future is *verifiably* not very good. Excellent works on the failure of prediction include Daniel Kahneman's "Thinking Fast and Slow" and "The Signal and the Noise: Why So many Predictions Fail - But Some Don't" among many others.
So, it's reasonable to ask, why shouldn't software be developed like other engineering disciplines? If you spend more time up-front in design then aren't you both being conscientious and efficient (problems caught in design are much cheaper to fix than in later stages)? And isn't the alternative (assuming there's only one) a recipe for death marches?
But, to choose another engineering discipline, don't mechanical engineers have the same problem? This is true but software, being soft, operates in an artificial environment. If an architect thought the Eiffel Tower would look better if only the Pyramids of Giza were off in the distance, possibly just ahead of the snow-capped peaks of the Swiss Alps no one would take him seriously. The alps aren't movable (yet), nor are the pyramids (yet).
Yet an analogous request in software, precisely because it's soft, can't be dismissed outright because software lacks (usually) such hard (and obvious) constraints. Built a program that knows how to read log files? Excellent! We've got tons of database logs that we need to parse. It wasn't designed for reading database log files? No worry, it's software, we can change it.
Essentially the softness of software, its malleability, guarantees that requirements defined will be incomplete (=incorrect) the second after the ink dries (assuming you still print them out). The Pyramids of Giza can be moved to France, the mountains can be twisted into a more photogenic orientation and nearly any idea is not only possible but a wonderful morass of design into which us software engineers happily but, usually unprofitably, dance.
A natural response to this is the impulse to "nail down the requirements and don't allow them to change". Unfortunately malleability is possibly the biggest value proposition that software brings to the table. Software is valuable precisely because it can be easily changed without having to reorient an assembly line, stamp out possibly millions of replacement widgets, ship them, track them, etc...
So we have this intrinsic tension. Software is valuable because it's easy to change. Accommodating change requires design. Design is essentially trying to predict the future. Prediction is hard because of change and change, as a fundamental value proposition, is and will always be the norm in software. Ergo, we should expect our ability to predict the future to be even worse for software than other disciplines (for which malleability is a less significant part of their value proposition).
So what do you do when you must see into a future to satisfy requirements well but that future is almost certainly guaranteed to have changed the next time you look? Look more frequently and don't spend too much time looking too far into the future.
This, to me, is the reason Agile gained prominence in the mid-90s. It's also, IMHO, the driving force behind shorter release cycles, earlier feedback and many of the other practices associated with Agile. We can't predict the future since, in our business, that future does not really exist yet. It's created in the iteration cycles during which stakeholders mold and shape the product; the molding and the shaping itself molds and shapes the future as stakeholders discover what they like, what they don't like, what they thought they'd like but don't, what they don't like but works so well they'll take, etc...
The further into the future we engage in detailed design the more time and energy we spend on a future that will be very different by the time we get there. The lack of obvious artificial constraints means that few blind alleys can be discarded outright. The future is often so different when we get there that instead of helping ourselves by designing for our expected future, we've designed ourselves into a corner with loads of technical debt created to get out of that corner.