Massy-Goldman report alleging 50% CSE Ph.D. overproduction to be re-issued due to flawed data

July 28, 1995

During the first two weeks of July, a study of Ph.D. production by William F. Massy of Stanford and Charles A. Goldman of RAND received widespread attention, stimulated by a New York Times article.

Both the study and the article turn out to have been badly flawed.

The study

On behalf of the Computing Research Association, Jeffrey D. Ullman from Stanford and Robert W. Ritchie from Hewlett-Packard met with Bill Massy, the senior author of the study, during the last week of July. Here is Ullman's report, edited slightly for clarity:
It turns out that Bill Massy has just discovered that a critical input parameter to his model -- the total number of doctorates employed in the field -- was incorrect for CS.

Table 1.7 (p. 1-32 of the report) has 5,376 for this number.

Historical data from NSF, though, shows a steady increase to 19,800 for the two years prior to the year used as input data, and 5,376 for that year. The anomaly was due to a change in definition by those in NSF responsible for collecting the data.

Massy has concluded that the figure of 5,376 is inappropriate for use in his model. He has decided to extrapolate the historical data to a figure of 21,000 instead.

As a result, CS now shows an employment gap of only 3.6% instead of the published 50.3%. This number is one of the lowest of all fields, and suggests that the projections of 15 years ago calling for a buildup to 1000 CS Ph.D.s/year are in remarkable agreement with the revised Massy-Goldman estimate of current demand.

Massy says he is going to release the corrections to everyone who received the report. We also expect Massy and Goldman to issue a full, revised report shortly.

It's worth noting that the Massy-Goldman report was just that -- a technical report, totally unrefereed and replete with numerous obvious typographical errors in addition to the methodological glitch in the case of computer science and engineering.

The New York Times article

The New York Times article leads one to believe that the Massy-Goldman study was a "survey," when in fact the study involves a first-order Markov model. The study is perfectly clear on this point, so the misleading statements are solely due to the New York Times reporter.

A statement from the article such as:

"The surplus of doctoral computer science degrees currently awarded over the number of those who get desirable jobs in their field is 50.3 percent ..."
is thus doubly flawed: (Needless to say, a properly constructed, validated, and parameterized model could be useful as a policy planning vehicle.)

What is a "desirable job?"

The question of what constitutes "fit employment" for a Ph.D. is an interesting one. Here is a comment from Forest Baskett, Vice President and Chief Technology Officer for Silicon Graphics:
"My company hires quite a number of Ph.D.s. While it is probably true that they don't have "jobs that really require a Ph.D." [language from the Massy-Goldman study], they tend to be better employees (by a lot) in those jobs than less well educated and less well (academically) trained people we have also hired. These people are now moving up through our organization and becoming the technical and business leaders of the company. I believe that this is a perfectly legitimate role and goal for Ph.D. production even if it is not uppermost in the minds of professors in charge of that production."
Baskett goes on to say:
"A trend I do find disturbing and counterproductive is the trend toward postdoctoral fellows in computer science. In my mind, opportunities for postdoctoral fellowships allow students to further postpone the making of hard career choices. This potential for procrastination also breaks the feedback loop from students to professors about the "relevance" of their education, both in courses and research projects. When students know that they have to actually get a career-defining job at the end of their degree program and they know that academic jobs are no longer easy to find, those students will, I believe, more effectively vote with their feet about courses and research areas in a way that better couples what's happening in our industry to what's happening in our university departments. We have seen this loss of coupling happen in other areas. Physics and life sciences come to mind.

"We've had a fabulous 30 year history of great relations between our computer science departments and a vibrant industry without postdoctoral fellows gumming up the works. One of the reasons has been that the Ph.D.s who have gone into industry have been young and energetic and fearless. They have often quickly realized that they were still doing their research. In our fast moving industry, it also often just happened to be product development. This is a good model. Let's not break it."

Baskett raises important questions concerning the nature of Ph.D. education which deserve serious thought.

lazowska at