The Success of Failure, The Failure of Success

As I look at the history of science and current investigations of all kinds, I continue to be more and more impressed with the way conventional ideas of discovery appear to be far off the mark. One can be quite successful in one’s training and career but never produce new knowledge of any consequence. It is striking that in many cases, the opposite seems to be true. The observations below are based on an excerpt from In the Mind’s Eye. Although it was written long ago, the pattern outlined here comes often to my mind. (Full references are, of course, available in the original text.)

It may be that it is sometimes harder for well-trained and successful scientists to make truly original discoveries. They are kept busy with many obligations--preparing lectures, advising students, consulting with coauthors: journal deadlines, peer review, grant applications, student recommendations, conference presentations, expert advisory groups, government commissions, faculty parties and retirement dinners. Plenty to keep one busy and fully focused on the pressing schedule at hand.

It is little wonder that Albert Einstein recommended that a scientist should remain apart as much as possible from the politics of science (“the battle of the brains”) and instead take work as a lighthouse keeper or shoemaker or some other less demanding employment--to keep his mind relatively free of interruptions and worrisome obligations. His suggestions seem romantic to us now, but his basic approach has begun to appear increasingly sensible to experienced researchers who find their lives are too full of career obligations to focus on really original work.

Strangely, in a similar way, lack of career success or even a long illness may be of benefit to the creative person. These may help to give the creative person the uninterrupted time to get the work done--not unlike the paradoxical benefit of imprisonment--in the old days--in getting books written. (It may be recalled that it was while he was in prison that Marco Polo had time to set down the account of his extensive travels in China.)

A not too successful career path may also be a help. James Clerk Maxwell wrote his two-volume Treatise during a period of semi-retirement in the middle of a none too successful career. (The cause and effect relationships are never perfectly clear, however. Would the book never have been completed if he had not the time, or would he have found the time in any case? How many would be willing to interrupt or divert a career that is in high gear to devote full attention to a fringe project with a risky and uncertain outcome? Or, indeed, how many ambitious, career-conscious professionals would allow themselves to focus on deep but questionable topics.)

If one wishes to be really creative, however, it seems that sometimes it is essential to have one's time less than fully committed, to be able to follow where one's thoughts lead rather than having to succeed in a series of tasks largely defined by one's career, one's competitors or by other outside forces.

One wonders what might have happened to Einstein's early work without his period of independent study (following his own fancy) as a school dropout, his lecture cutting and continued self-directed study during his university years, his two years of intermittent unemployment (with contrasting growth in intellectual excitement) after graduation and then his relatively undemanding patent office job once he did enter the world of conventional work.

In contrast, we might wonder what would have happened if he had become immediately enmeshed in the teaching, administrative and social demands of a conventionally successful career. As he himself observed, in such a path there is a strong tendency to do research that is comparatively superficial and predictable--little steps that do not risk serious failure or threaten existing beliefs, modest research programs that can be relied upon to produce publishable results and supportive, unthreatened mentors.

We might wonder how many have been diverted from greater accomplishments because of early success and recognition. Once again, a kind of natural selection may be at work, having curious consequences. Sometimes, truly great accomplishments may be severely hindered by even modest amounts of early conventional success.

But there may be other, more subtle barriers to really original creative work. There are other reasons that the expert and the professional may find it hard to change their ways of thinking, especially when it involves some of their most basic concepts.

Many years of hard work and one's self-image as a competent professional may be closely bound up with a belief in what one has been taught--which, in time, often one has come to be engaged in teaching oneself. Of course, little changes and corrections are needed here and there to keep up the momentum of gradual progress and to make one's reputation. But there is little incentive to question or overturn fundamental elements of the discipline--making obsolete some of one's own career accomplishments as well as those of many others. Even worse is to blur the boundaries of the discipline, inviting territorial battles and the threat of lost professional status and credibility. As James Gleick observed in his book Chaos, this conservatism continues to be a powerful force in the emerging science of chaos:

“. . . The language of mathematics remained a serious barrier to communication. If only the academic world had room for hybrid mathematician/physicists--but it did not. . . . Mathematicians continued to speak one language, physicists another. As the physicist Murray Gell-Mann once remarked: ‘Faculty members are familiar with a certain kind of person who looks to mathematicians like a good physicist and looks to physicists like a good mathematician. Very properly, they do not want that kind of person around.’ The standard of the two professions were different. Physicists had theorems, mathematicians had conjectures. The objects that made up their worlds were different. Their examples were different.”

So we are told that it is much safer to clearly identify with one group or another. The risk of confusion or loss of credibility is often too great to be seriously considered by any prudent professional, especially those who have learned early and well the rewards of staying within the acceptable and desirable categories. Balance a doubt against a certainty and stay within the conventions of normal science.

But in the end, there is a greater problem: the important thing is that truly original discoveries sometimes require unlearning and relearning not only what one has been taught, but also fundamental and basic elements in the way one thinks about things in general--some of the most fundamental concepts at the core of one's thought processes.

We must not go too far down this road, of course, without pointing out that there are all kinds and levels of accomplishment and creative discovery. Each set of weekly and monthly science periodicals is a feast of enticing new developments. We are all too aware that the conventional system--of university courses and grant programs and research laboratories--is turning out vast quantities of wonderful and exciting and frightening things.

Clearly, the conventional system does produce. It would be wrong to suggest otherwise. (Although it seems to often produce pieces that make the whole less clear. The most general and comprehensive and integrating explanations are the most exciting, but are the most rare.) However, there are other ways. And sometimes the biggest ideas come from the least likely sources. And perhaps the most important ideas come rarely from those who would appear, by conventional standards, to be the best and the brightest.

This essential problem may explain why it is sometimes best to work in a field other than the one that one originally studied--if one wants to make truly original contributions. In one's own field, one has already developed a strong internal editor that may serve to criticize and to demolish the “silly” thoughts before they can take hold internally or before they can get out to embarrass you. It is hard to unlearn what you have learned so well. Perhaps, also, this is the reason that it has been observed that the most original ideas often come not from those at the top of their field, but from those who are at the fringes of the field.

The double problem of knowing too much or knowing too little is nicely summarized in a passage from one of the classic studies of scientific creativity. When a field is developing normally, expert knowledge is a great advantage; but when there is an abrupt change in the course of development (or some major new factor becomes evident), expert knowledge may be a considerable disadvantage:

“Thus in subjects in which knowledge is still growing, or where the particular problem is a new one, or a new version of one already solved, all the advantage is with the expert, but where knowledge is no longer growing and the field has been worked out, a revolutionary new approach is required and this is more likely to come from the outsider. The skepticism with which the experts nearly always greet these revolutionary ideas confirms that the available knowledge has been a handicap.”

It is, of course, a double-bind in another sense; one has to be close enough to the conventional in order to obtain the needed information from the conventional sources, to check one's findings and to be able to explain one's new ideas in a way that is understandable and acceptable to conventional modes of thought.

But this is not a new problem. Long ago, an instance of coming close to a discovery but not being able to make the conceptual changes needed to achieve the desired result was observed in a colleague by Michael Faraday. In a letter to a friend who had described to Faraday this colleague's researches on the magnetic condition of matter, Faraday wrote:

“Many thanks, my dear Wheatstone, for your note. I have in consequence seen Bequerel's paper, and added a note at the first opening of my paper. It is astonishing to think how he could have been so near the discovery of the great principle and fact, and yet so entirely miss them both, and fall back into old and preconceived notions.”

Here we see that the power of “old and preconceived notions” may serve as a barrier to imminent discovery in any time or age. Then, as now, for some, one of the greatest deterrents to original discovery may be nothing more than long-established habits of thought.

As an old man, reviewing his life's work, Einstein observed that one of the most difficult things he had to do in his own work was to unlearn old patterns of thought. He observed that, in retrospect, a certain line of development could be seen as almost inevitable--yet barriers of inflexible basic concepts could deter progress to that inevitable solution for many years:

“That the special theory of relativity is only the first step of a necessary development became completely clear to me only in my efforts to represent gravitation in the framework of this theory. . . . This happened in 1908. Why were another seven years required for the construction of the general theory of relativity? The main reason lies in the fact that it is not so easy to free oneself from the idea that coordinates must have a direct metric significance.”

In other words, Einstein saw that he could not move from the special theory to the general theory of relativity without first changing, in his own mind, some extremely basic ideas and conceptions. This, it took seven years to do.

Einstein commented elsewhere on this problem--remarking that, in time, some of his theories could be easily understood by young students, but that made them no easier to find in the beginning, when he was wandering alone “in the dark.” The hard part, apparently, is seeing for the first time things in a way that is different from the way they have ever been seen before:

“In the light of knowledge attained, the happy achievement seems almost a matter of course, and any intelligent student can grasp it without too much trouble. But the years of anxious searching in the dark, with their intense longing, their alternations of confidence and exhaustion, and the final emergence into the light--only those who have experienced it can understand it.”

Apparently, many scientists are never able to do this deep exploration. They have neither the special ability nor the essential inclination. Basic concepts, once learned, are fixed for life.

Thus we see that, sometimes, great quantities of knowledge are not enough. Sometimes one has to be willing to change basic assumptions and thought processes in order to see clearly the unexpected truths that could be clearly evident--if only we could see them in the right way. Sometimes knowing all the data is not nearly so important as being able to view it from a different and truly original perspective.

Sometimes the illusion of difficulty alone is enough to deter progress--sometimes to a shocking extent. Some physicists and astronomers have recognized that the presumed difficulty of some of Einstein's ideas has long been greatly over stated and that this same attitude has significantly retarded advances in a number of related scientific areas--in this case, according to some, for a period of over forty years. Thus one prominent astronomer, who was closely acquainted with the scientists involved in the experimental proof of Einstein's general relativity theory, observed in a talk given in 1982:

“I may parenthetically remark that this supposed difficulty in understanding the general theory of relativity was greatly exaggerated: it contributed to the stagnation of the subject for several decades. Many of the developments of the sixties and the seventies could easily have taken place during the twenties and the thirties.”

Another curious aspect of creative activity that we can learn from this same source is the way some people can immediately recognize the validity of another's work. They can see without hesitation or reservation that the work of another is correct and revolutionary. No testing is required for their certainty--although objective verification is, of course, required for presentation to others.

In 1918, the British astronomer, Sir Arthur Eddington, mounted expeditions to Brazil and West Africa in order to make astronomical observations to verify Einstein's theories. One colleague, S. Chandrasekhar, had complimented Eddington on his courage in planning the expedition when the outcome seemed so uncertain. In Chandrasekhar's words:

“To my surprise, Eddington disclaimed any credit on that account and told me that, had he been left to himself, he would not have planned the expeditions [at all] since he was fully convinced of the truth of the general theory of relativity!”

Eddington never really had any doubt about the correctness of Einstein's theories. As soon as he had heard of Einstein's work, he knew Einstein had to be right. We are told that this was known to be so because the great influence of Einstein's work on Eddington. When Einstein formulated his fundamental field equations, he concluded his paper with the observation “scarcely anyone who has fully understood this theory can escape from its magic.” Chandrasekhar comments:

“Eddington must surely have been caught in its magic; for, within two years, he had written his Report on the Relativity of Gravitation for the Physical Society of London, a report that must have been written in white heat.”

Among like minds, sometimes there can be an immediate recognition of the truth of a new discovery--however different it is from everything that is taught by one's teachers and believed by one's peers.

Based on an excerpt from In the Mind's Eye, chapter 8, “Patterns in Creativity.”