Sunday, August 1, 2010

The Success of Failure, The Failure of Success

It is widely believed that students with high grades move on to eventually make great discoveries, innovations and other major contributions to their chosen field. However, sometimes, indeed many times it seems, the reverse is true. The world of scientific research provides us with a window into the true origins of many major discoveries.

Strangely, it is sometimes harder for well-trained and highly successful scientists to make truly original discoveries. They are kept busy with many obligations -- preparing lectures, advising students, consulting with coauthors: journal deadlines, peer review, grant applications, student recommendations, conference presentations, expert advisory groups, government commissions, faculty parties and retirement dinners. Plenty to keep one busy and fully focused on the pressing schedule at hand.

It is little wonder that Einstein recommended that a scientist should remain apart as much as possible from the politics of science ("the battle of the brains") and instead take work as a lighthouse keeper or shoemaker or some other less demanding employment -- to keep his mind relatively free of interruptions and worrisome obligations. His suggestions seem romantic to us now, but his basic approach has begun to appear increasingly sensible to experienced researchers who find their lives are too full of career obligations to focus on really original work.

Strangely, in a similar way, lack of career success or even a long illness may be of benefit to the creative person. These may help to give the creative person the uninterrupted time to get the work done -- not unlike the paradoxical benefit of imprisonment, in the old days, in getting books written. (It may be recalled that it was while he was in prison that Marco Polo had time to set down the account of his extensive travels in China.) At least one book has been written on the beneficial role of long illnesses in providing uninterrupted time for single-minded creative effort. A not too successful career path may also be a help. Maxwell wrote his two-volume Treatise during a period of semi-retirement in the middle of a none too successful career.

(The cause and effect relationships are never perfectly clear, however. Would the book never have been completed if he had not the time, or would he have found the time in any case? How many would be willing to interrupt or divert a career that is in high gear to devote full attention to a fringe project with a risky and uncertain outcome? Or, indeed, how many ambitious, career-conscious professionals would allow themselves to focus on questionable topics such as cloud formations when the boss is really interested in laser fusion?)

Accordingly, if one wishes to be really creative, it seems that sometimes it is essential to have one's time less than fully committed -- to be able to follow where one's thoughts lead rather than having to succeed in a series of tasks largely defined by one's career, the limits of one’s specialist area or the ambitions of one's competitors.

One wonders what might have happened to Einstein's early work without his period of independent study (following his own fancy) as a school dropout, his lecture cutting and continued self-directed study during his university years, his two years of intermittent unemployment (with contrasting growth in intellectual excitement) after graduation and then his relatively undemanding patent office job once he did enter the world of conventional work.

We might wonder what would have happened if he had become immediately enmeshed in the teaching, administrative and social demands of a conventionally successful career. As he himself observed, in such a path there is a strong tendency to do research that is comparatively superficial and predictable -- little steps that do not risk serious failure or threaten existing beliefs, modest research programs that can be relied upon to produce publishable results and supportive, unthreatened mentors.

We might wonder how many have been diverted from greater accomplishments because of early success and recognition. Once again, a kind of natural selection may be at work, having curious and unexpected consequences. Sometimes, strangely, we can argue that truly great accomplishments may be severely hindered by even modest levels of early conventional success.

In addition, there may be other, more subtle barriers to really original creative work. There are other reasons that the expert and the professional may find it hard to change their ways of thinking, especially when it involves some of their most basic concepts and beliefs.

Many years of hard work and one's self-image as a competent professional may be closely bound up with a belief in what one has been taught -- which, in time, often one has come to be engaged in teaching oneself. Of course, little changes and corrections are needed here and there to keep up the momentum of gradual progress and to make one's reputation. But there is often little incentive to question or overturn fundamental elements of the discipline -- making obsolete some of one's own career accomplishments as well as those of many others. Even worse is to blur the boundaries of the discipline, inviting territorial battles and the threat of lost professional status and credibility.

As James Gleick observes, this conservatism continues to be a powerful force in the still emerging science of chaos: “ . . . The language of mathematics remained a serious barrier to communication. If only the academic world had room for hybrid mathematician/physicists -- but it did not. . . . Mathematicians continued to speak one language, physicists another. As the physicist Murray Gell-Mann once remarked: ‘Faculty members are familiar with a certain kind of person who looks to mathematicians like a good physicist and looks to physicists like a good mathematician. Very properly, they do not want that kind of person around.’ The standard of the two professions were different. Physicists had theorems, mathematicians had conjectures. The objects that made up their worlds were different. Their examples were different.”

If one wants to have a successful career, it is much wider and safer to clearly identify with one group or another. The risk of confusion or loss of credibility is often too great to be seriously considered by any prudent professional, especially those who have learned early and well the rewards of staying within the acceptable and desirable boundaries. Balance a doubt against a certainty and stay within the conventions of the norm.

In the end, there is even a greater problem: the important thing is that truly original discoveries sometimes require unlearning and relearning not only what one has been taught, but also fundamental and basic elements in the way one thinks about things in general -- some of the most fundamental concepts at the core of one's own thought processes.

We must not go too far down this road, of course, without pointing out that there are all kinds and levels of accomplishment and creative discovery. Each set of weekly and monthly science periodicals is a feast of enticing new developments. We are all too aware that the conventional system -- of university courses and grant programs and research laboratories -- is turning out vast quantities of wonderful and exciting and frightening things.

Clearly, the conventional system does produce. It would be wrong to suggest otherwise. (Although it seems to often produce pieces that make the whole less clear. The most general and comprehensive and integrating explanations are the most exciting, but are also the most rare.) However, there are other ways. Sometimes the biggest ideas can come from the least likely sources. And perhaps the most important ideas come rarely from those who would appear (by conventional standards) to be the best and the brightest.

All of this is, of course, an administrative problem as well. Such an unconventional process is hard to manage and control. Yet there it is. Perhaps all that can be done is to learn that such things do sometimes happen; to hope to be able to recognize them as they begin to develop; and to try to keep from destroying real creativity as it grows and begins to mature.

In the end, probably the most effective deterrent to really original work is not so much the career distractions and disincentives as the fact that the professionals often have learned their lessons too well. If you have had great success in learning to think the way your teachers and associates think, then it is all the more difficult to think in some really original and unexpected way.

In fairness, it should be noted that this problem clearly flows from the necessary double bind of any accumulated knowledge. Most of the gains are through the further application of what is already known. However -- occasionally -- this same factor can be a major cause of error and lack of progress.

This essential problem may explain why it is sometimes best to work in a field other than the one that one originally studied -- if one wants to make truly original contributions. In one's own field, one has already developed a strong internal editor that may serve to criticize and to demolish the “silly” thoughts before they can take hold internally -- or before they can get out to embarrass you. It is hard to unlearn what you have learned so well. Perhaps, also, this is the reason that it has been observed that the most original ideas often come not from those at the top of their field, but from those who are at the fringes of the field.

The double problem of knowing too much or knowing too little is nicely summarized in a passage from one of the classic studies of scientific creativity. When a field is developing normally, expert knowledge is a great advantage; but when there is an abrupt change in the course of development (or some major new factor becomes evident), expert knowledge may be a considerable disadvantage: “Thus in subjects in which knowledge is still growing, or where the particular problem is a new one, or a new version of one already solved, all the advantage is with the expert, but where knowledge is no longer growing and the field has been worked out, a revolutionary new approach is required and this is more likely to come from the outsider. The skepticism with which the experts nearly always greet these revolutionary ideas confirms that the available knowledge has been a handicap.”

It is, of course, a double-bind in another sense; one has to be close enough to conventional thought in order to obtain the needed information from the conventional sources, to check one's findings and to be able to explain one's new ideas in a way that is understandable and acceptable to others conventional modes of thought.

But this is not a new problem. Long ago, an instance of coming close to a discovery but not being able to make the conceptual changes needed to achieve the desired result was observed in a colleague by Michael Faraday. In a letter to a friend who had described to Faraday this colleague's researches on the magnetic condition of matter, Faraday wrote: “Many thanks . . . for your note. I have in consequence seen Bequerel's paper, and added a note at the first opening of my paper. It is astonishing to think how he could have been so near the discovery of the great principle and fact, and yet so entirely miss them both, and fall back into old and preconceived notions.”

Here we see that the power of “old and preconceived notions” may serve as a barrier to imminent discovery in any time or age. Then, as now, for some, one of the greatest deterrents to original thinking and discovery may be nothing more than long-established habits of thought.

As an old man, reviewing his life's work, Einstein observed that one of the most difficult things he had to do in his own work was to unlearn old patterns of thought. He observed that, in retrospect, a certain line of development could be seen as almost inevitable -- yet barriers of inflexible basic concepts could deter progress to that inevitable solution for many years: “That the special theory of relativity is only the first step of a necessary development became completely clear to me only in my efforts to represent gravitation in the framework of this theory . . .. This happened in 1908. Why were another seven years required for the construction of the general theory of relativity? The main reason lies in the fact that it is not so easy to free oneself from the idea that coordinates must have a direct metric significance.”

In other words, Einstein saw that he could not move from the special theory to the general theory without first changing -- in his own mind -- some extremely basic ideas and conceptions. This, it took seven years to do.

Einstein commented elsewhere on this problem -- remarking that, in time, some of his theories could be easily understood by young students, but that made them no easier to find in the beginning, when he was wandering alone in the dark.

The hard part, apparently, is seeing for the first time things in a way that is different from the way they have ever been seen before: “In the light of knowledge attained, the happy achievement seems almost a matter of course, and any intelligent student can grasp it without too much trouble. But the years of anxious searching in the dark, with their intense longing, their alternations of confidence and exhaustion, and the final emergence into the light -- only those who have experienced it can understand it.”

Presumably, many scientists (and others) are never able to do this deep exploration. They have neither the special ability nor the essential inclination. Basic concepts, once learned, are fixed for life.

Thus we see that, sometimes, great quantities of knowledge are not enough. Sometimes one has to be willing to change basic assumptions and thought processes in order to see clearly the unexpected truths that could be clearly evident -- if only we could see them in the right way. Sometimes knowing a lot is not nearly so important as being able to view it from a different and truly original perspective.

Adapted from chapter 8, In the Mind’s Eye, where full references are provided.