In the UK and most European countries a PhD in a subject is the way to gain entry to serious academic discussions of that discipline. The structure of the PhD examination is designed to achieve this. The two criteria are:
(a) does the thesis make a contribution to knowledge?
(b) is the candidate a reliable member of the academic discipline?
As you can see, both the thesis and the person are being examined. Both aspects are crucial.
The understanding of what is a contribution to knowledge, across almost all disciplines, is that the candidate has conducted a systematic investigation into a question of interest to the academic discipline and has established a new (surprising) result. The words "systematic" and "established" imply that the thesis contains a rational defence of the result, which a sceptical member of the academic discipline will find convincing.
For the second criterion, about the person, examiners will start with the literature review that the thesis contains. The examiners will want to see that it includes references to all previous material that is directly relevant to establishing the new contribution, including papers that prompted the research and provided the framework or starting points, and papers that have reported results that appear relevant to the investigation. The way that these papers are reviewed is important: the PhD examiners will carefully consider the fairness and accuracy of the candidate's judgements on these previous contributions to the subject. Are any important contributions missing from the review? Are the candidate's judgements of the work of others well-founded?
In both cases, in the viva voce examination the examiners will probe any areas of doubt.
For the quality of the investigation and the reliability of the conclusions the examiners will want to check the research methodology and the coherence of the argument. A thesis is well-argued if all statements made in the thesis either are previously established knowledge or have been established by the candidate's investigation. There should not be repetition or non-sequiturs. The methodology and the breadth of the investigation will be satisfactory if they support the claims that are made for the generality of the conclusions. That is, even if the evidence that has been gathered has been correctly evaluated, what confidence can there be that further evidence would not yield different results?
For the adequacy of the candidate's knowledge and judgement, the examiners will carefully consider the list of references supplied and the way they are dealt with in the literature review. All bona fide members of the academic discipline will immediately notice any important omissions, especially of recent work relevant to the thesis. For the purposes of the thesis while it is important to acknowledge the first source of an idea, it is even more important to cite more recent work. And irrelevant material should not be included, as this would indicate that the candidate does not really understand the academic discipline.
So, in a typical case a PhD examiner will start with the research overview (typically in Chapter 1), will check whether a contribution to knowledge is claimed at all (first and last chapters) and if so, whether it really is a worthwhile contribution to knowledge. If the research turns out not to be in an area of interest to the examiner, or there is no contribution to knowledge, the process may well stop at this stage.
Next the examiner will consider the list of references, and then work in detail through the thesis, making careful notes. These notes will generally be of the following kinds:
(1) typographical or grammatical errors, or other inappropriate use of language
(2) factual errors about the academic discipline or its history, gaps in the reasoning or unsubstantiated assertions
(3) missing or irrelevant references or inappropriate comments on literature
(4) poor methodology or evaluation, or over-enthusiastic generalisation
Finally the examiner will check that the abstract is an accurate summary of the thesis, and return to the two criteria mentioned at the start of this article.
Wednesday, December 28, 2011
Sunday, August 7, 2011
What size is a PhD?
Research, and contributions to knowledge, do not come in standard sizes. The unit of currency varies by discipline, and some research projects take a lifetime, while some discoveries are made quite suddenly.
PhDs, however, have a constraint: they are designed to finish within 3-4 years of full time work by a new researcher. It is not just a matter of a contribution to knowledge (albeit small) but whether the contribution to knowledge is enough to merit a PhD.
So in addition to the qualitative judgements about orginality, rigour, novelty there are some important quantitative issues:
Is the scope to wide or too narrow? Do we need to triangulate with some more fieldwork? Do we need a further caser study? etc etc.
These quantitive judgements by the supervisors and examiners are justified by appealing to the qualitative issue about whether the conclusions has been "established" on the basis of the evidence used, or require further work.
Each discipline has its traditions about whether negative results represent contributions to knowledge, what sample sizes are required, whether the conclusions based on one study can really apply to other scenarios etc.
And of course, all research requires further work.
But while PhD students may be entitled as a last resort to submit in defiance of advice from supervisors, it is not to be recommended. Although this is yet another area where there are no clear rules, there tends to be a strong consensus among academic examiners from any particular discipline. And they are likely to ask about the progress of the research: what were the initial objectiives, what advice was received, what were the opportunities and setbacks? Why, in the end, was this particular result set deemed worthy of submission?
And of course, has any external body, conference, journal been impressed?
PhDs, however, have a constraint: they are designed to finish within 3-4 years of full time work by a new researcher. It is not just a matter of a contribution to knowledge (albeit small) but whether the contribution to knowledge is enough to merit a PhD.
So in addition to the qualitative judgements about orginality, rigour, novelty there are some important quantitative issues:
Is the scope to wide or too narrow? Do we need to triangulate with some more fieldwork? Do we need a further caser study? etc etc.
These quantitive judgements by the supervisors and examiners are justified by appealing to the qualitative issue about whether the conclusions has been "established" on the basis of the evidence used, or require further work.
Each discipline has its traditions about whether negative results represent contributions to knowledge, what sample sizes are required, whether the conclusions based on one study can really apply to other scenarios etc.
And of course, all research requires further work.
But while PhD students may be entitled as a last resort to submit in defiance of advice from supervisors, it is not to be recommended. Although this is yet another area where there are no clear rules, there tends to be a strong consensus among academic examiners from any particular discipline. And they are likely to ask about the progress of the research: what were the initial objectiives, what advice was received, what were the opportunities and setbacks? Why, in the end, was this particular result set deemed worthy of submission?
And of course, has any external body, conference, journal been impressed?
Wednesday, September 9, 2009
Agents and Bee Foraging
Some recent work at UWS was inspired by the real or imagined activities of a colony of foraging bees. Things got awkward when the model that was being used for experiments turned out to be different from the way bees actually forage. The supervisors said “all models are wrong, but some are useful” and of course wanted to explore the proposed approach, but the student largely lost interest. To my surprise, some readers felt that the study of unnatural systems was intrinsically repugnant, and that the story illustrated the need for science and religion to work hand in hand.
Suppose that a multi-agent system, with the task of looking for a particular sort of cluster in a large data set, observes a potential sub-cluster. We can imagine an automated step of spawning a new agent trained to look for further evidence of such a cluster. However, it is a bit fanciful to think of this new agent as a specially trained infant bee, as bees may learn the habits of the nest, but do not seem to receive the sort of individual instruction found in species with nuclear families. Other work at UWS examined the development of language in interacting groups of automata, and the introduction of a new word in that experiment is not unlike the introduction of a new agent in this one, since the introduction of the word implies a new subset of individuals that use it.
Leaving aside the biological inspiration, could a commercial system be imagined with similar properties? We could imagine such a system working in the data centre of a large supermarket or bank. If new agents can be spawned in this way, there would undoubtedly be issues of monitoring or control. A novel data cluster might result in a massive generation of new agents which might appear as unexpected additional activity in the system. In a commercial data centre it is possible that such an event would lead to suspicions of an intrusion or system fault.
If the autonomous agents are required to do a lot of status reporting to explain what they are up to, the additional monitoring traffic might create so many external messages as to call into question the wisdom of using agents. On the other hand, if the reporting traffic was cleverly aggregated within the swarm, a coherent report could be made to a monitor that a particular observation led to the deployment of 123,000 agents to investigate the possible existence of a new cluster, and this activity had now ended. Some computing systems build this sort of observable surface over chaotic, Brownian, internal motion; just as the apparently random behaviour of autonomous bees creates a regular-shaped nest. For example, network management systems aggregate event reports that have a shared cause, and (doubtless) Microsoft’s performance reporting systems do something similar.
In this way, the investigation of circumstances for and rules for the creation of a new agent leads to a new and interesting control problem, where the new problem is that of explaining the new situation that has arisen, in terms that make sense to those who have not been tracking all the details…
Suppose that a multi-agent system, with the task of looking for a particular sort of cluster in a large data set, observes a potential sub-cluster. We can imagine an automated step of spawning a new agent trained to look for further evidence of such a cluster. However, it is a bit fanciful to think of this new agent as a specially trained infant bee, as bees may learn the habits of the nest, but do not seem to receive the sort of individual instruction found in species with nuclear families. Other work at UWS examined the development of language in interacting groups of automata, and the introduction of a new word in that experiment is not unlike the introduction of a new agent in this one, since the introduction of the word implies a new subset of individuals that use it.
Leaving aside the biological inspiration, could a commercial system be imagined with similar properties? We could imagine such a system working in the data centre of a large supermarket or bank. If new agents can be spawned in this way, there would undoubtedly be issues of monitoring or control. A novel data cluster might result in a massive generation of new agents which might appear as unexpected additional activity in the system. In a commercial data centre it is possible that such an event would lead to suspicions of an intrusion or system fault.
If the autonomous agents are required to do a lot of status reporting to explain what they are up to, the additional monitoring traffic might create so many external messages as to call into question the wisdom of using agents. On the other hand, if the reporting traffic was cleverly aggregated within the swarm, a coherent report could be made to a monitor that a particular observation led to the deployment of 123,000 agents to investigate the possible existence of a new cluster, and this activity had now ended. Some computing systems build this sort of observable surface over chaotic, Brownian, internal motion; just as the apparently random behaviour of autonomous bees creates a regular-shaped nest. For example, network management systems aggregate event reports that have a shared cause, and (doubtless) Microsoft’s performance reporting systems do something similar.
In this way, the investigation of circumstances for and rules for the creation of a new agent leads to a new and interesting control problem, where the new problem is that of explaining the new situation that has arisen, in terms that make sense to those who have not been tracking all the details…
Monday, September 7, 2009
On Scholarship 2.0
During August Reinventing Academic Publishing Online appeared on Scholarship 2.0. It is a polemic against what its authors see as an exclusive establishment consisting of the "top academic journals" that only the richest universities can afford, and a self-serving institutional system that distorts the academic process in order to make the job of funding bodies and appointing committees easier.
Now there are many misguided people who think that there are such things as "top academic journals" where the best computing research is to be found, and regrettably some of these people do appear to hold positions of power and influence. But the fault is theirs alone.
I believe strongly in the value of computing research conferences: but the large ones have pursued profit at the expense of discrimination. It is easy to find dreadful papers presented at even the best conferences, with half-baked ideas and without results or any pretence at evaluation. But it is precisely the Web and Web 2.0 that allows us to find quality independently of the vehicle used for publication.
I have been following with some interest the response in UK academia to the recent Research Assessment Exercise. The Computing panel noted that excellent articles were to be found in journals with low impact factors, and conversely. They were astonished at the huge number (1247) of refereed journals that submitted articles had appeared in, and were amazed that relatively few university departments had submitted conference publications. They restated their policy that conference papers could be just as good as those found in the "top-rated journals".
Interested readers can follow this debate in the Conference of Heads and Professors of Computing and the consultation about the Research Excellence Framework.
Not only is their analysis of the last RAE excellent: so are the proposals for the next RAE, which will recognise that originality, rigour and impact are not usually found in a single publication. So let's just get on with the research, and leave the task of re-inventing academic publication to those who have time for it.
(Update 9 Nov: and in the meantime, join www.mendeley.com, which looks like a good Web 2.0 scholarship repository!)
Now there are many misguided people who think that there are such things as "top academic journals" where the best computing research is to be found, and regrettably some of these people do appear to hold positions of power and influence. But the fault is theirs alone.
I believe strongly in the value of computing research conferences: but the large ones have pursued profit at the expense of discrimination. It is easy to find dreadful papers presented at even the best conferences, with half-baked ideas and without results or any pretence at evaluation. But it is precisely the Web and Web 2.0 that allows us to find quality independently of the vehicle used for publication.
I have been following with some interest the response in UK academia to the recent Research Assessment Exercise. The Computing panel noted that excellent articles were to be found in journals with low impact factors, and conversely. They were astonished at the huge number (1247) of refereed journals that submitted articles had appeared in, and were amazed that relatively few university departments had submitted conference publications. They restated their policy that conference papers could be just as good as those found in the "top-rated journals".
Interested readers can follow this debate in the Conference of Heads and Professors of Computing and the consultation about the Research Excellence Framework.
Not only is their analysis of the last RAE excellent: so are the proposals for the next RAE, which will recognise that originality, rigour and impact are not usually found in a single publication. So let's just get on with the research, and leave the task of re-inventing academic publication to those who have time for it.
(Update 9 Nov: and in the meantime, join www.mendeley.com, which looks like a good Web 2.0 scholarship repository!)
Saturday, July 4, 2009
On Truth and Information
In recent years, several philosophers of computing, such as Fred Dretske(1932-) and Luciano Floridi (1964-), have established to a great many people’s satisfaction that for something to be called information, it should allow us to learn something that is true. They argue that false information is not information, just as a false policeman is not a policeman.
In my June 20 contribution to this blog I rather incautiously mentioned the pursuit of the truth about (the laws of) the physical world, and I hinted that I felt this was not the business of science, or research for that matter. Truth is a great concept and a noble ideal, but as I mentioned in the April 18 contribution, the more truthful we make any statement, the less clear it is, and the narrower the scope of its application. In a way, the only absolutely truthful statements are the formal tautologies of pure mathematics, the necessary truths that depend on nothing, and so add nothing to our knowledge. Conversely, any statement that is not necessarily true is merely (by definition) contingently true. It might be true, that is, if (um..) everything were really as it says. There are few philosophers, and even fewer scientists, who feel that this sort of circular discussion is worth while.
And yet, precisely this kind of argument has fascinated people for millennia. Descartes’ necessary truth “cogito, ergo sum” (1641) was the start of his argument to prove the existence of God and the immortality of the soul. A very similar exercise by Bernard Lonergan (1904-1984) achieved a wide currency in 1956. Now the reader should always smell a rat if someone claims to prove a contingent truth from a necessary one. The Australian philosopher David Stove (1927-1994) catalogued a large number of such arguments from Plato to Habermas: explaining that since they want to make everyone accept their opinion, it is a good trick to make it appear merely a logical deduction from a necessary truth. The trick can be made to work, as he explains, with the support of some impressive but contradictory concept, in the same way that division by zero can be used to create a convincing-looking proof. Usually though, such philosophers are vainly trying to use logic to establish some belief which predates their attempt and will outlast their failure.
To return to computing: every piece of information, according to all of us who follow Floridi and Dretske, contains its very own claim of contingent truth. By labelling it as information, we claim that it is not just some sample data: it will allow a suitably placed observer to learn something about the real world (Dretske 1981), at the time the information was constructed. This field of thinking has its own thought-experiments, for example, a bear-track in the woods contains the information that a bear passed that way whether or not anyone ever notices the spoor, and anyone suitably placed to notice it can learn this content.
The raw data from a survey, or a sheaf of newspaper cuttings, may well contain a lot of information that can be drawn to our attention by a suitably placed researcher. Unlike the bear-track in the woods, however, the survey was collected, and the newspaper articles written, by humans who have well-known tendencies to misperception, mistake, misinformation… So while the Internet doubtless contains a lot of useful information, I know it also contains much that is erroneous, ill-informed, and misleading. In research we don’t accept any of it uncritically. We try to stick to good sources of information, we try for honesty in our evidence gathering, and we try to take care in our conclusions, in our mission to increase the stock of knowledge in our academic discipline.
In my June 20 contribution to this blog I rather incautiously mentioned the pursuit of the truth about (the laws of) the physical world, and I hinted that I felt this was not the business of science, or research for that matter. Truth is a great concept and a noble ideal, but as I mentioned in the April 18 contribution, the more truthful we make any statement, the less clear it is, and the narrower the scope of its application. In a way, the only absolutely truthful statements are the formal tautologies of pure mathematics, the necessary truths that depend on nothing, and so add nothing to our knowledge. Conversely, any statement that is not necessarily true is merely (by definition) contingently true. It might be true, that is, if (um..) everything were really as it says. There are few philosophers, and even fewer scientists, who feel that this sort of circular discussion is worth while.
And yet, precisely this kind of argument has fascinated people for millennia. Descartes’ necessary truth “cogito, ergo sum” (1641) was the start of his argument to prove the existence of God and the immortality of the soul. A very similar exercise by Bernard Lonergan (1904-1984) achieved a wide currency in 1956. Now the reader should always smell a rat if someone claims to prove a contingent truth from a necessary one. The Australian philosopher David Stove (1927-1994) catalogued a large number of such arguments from Plato to Habermas: explaining that since they want to make everyone accept their opinion, it is a good trick to make it appear merely a logical deduction from a necessary truth. The trick can be made to work, as he explains, with the support of some impressive but contradictory concept, in the same way that division by zero can be used to create a convincing-looking proof. Usually though, such philosophers are vainly trying to use logic to establish some belief which predates their attempt and will outlast their failure.
To return to computing: every piece of information, according to all of us who follow Floridi and Dretske, contains its very own claim of contingent truth. By labelling it as information, we claim that it is not just some sample data: it will allow a suitably placed observer to learn something about the real world (Dretske 1981), at the time the information was constructed. This field of thinking has its own thought-experiments, for example, a bear-track in the woods contains the information that a bear passed that way whether or not anyone ever notices the spoor, and anyone suitably placed to notice it can learn this content.
The raw data from a survey, or a sheaf of newspaper cuttings, may well contain a lot of information that can be drawn to our attention by a suitably placed researcher. Unlike the bear-track in the woods, however, the survey was collected, and the newspaper articles written, by humans who have well-known tendencies to misperception, mistake, misinformation… So while the Internet doubtless contains a lot of useful information, I know it also contains much that is erroneous, ill-informed, and misleading. In research we don’t accept any of it uncritically. We try to stick to good sources of information, we try for honesty in our evidence gathering, and we try to take care in our conclusions, in our mission to increase the stock of knowledge in our academic discipline.
Saturday, June 27, 2009
Supervision
Whose research is it? Yours. The basic idea will be something that the supervisor is an expert in (or you need another supervisor), but since you are making a new contribution to knowledge, at the end of the process you will be the world expert in your subject.
Your supervisor plays hugely important roles throughout the process though. At the start, they will help you with the relevant literature and established approaches. As the research progresses, they will help with methodology, with planning the research, and helping you phrase your research questions. Once you have parts of your thesis in draft, they will provide an invaluable critique of the flow of argument, and the construction of your thesis as a piece of rationally-argued writing. Your supervisor will also play a crucial role in selecting your external examiners, and being your supporter and eyes and ears during the viva.
Above all, throughout the process, they are following your journey, engaging in the discussions, playing the part of reader of your thesis and papers, reacting in the ways that your audience and your examiners might to the parts of your work that are new and surprising, so that you can fine-tune your arguments and make sure there are no loose ends.
The relationship between student and supervisor can sometimes be stormy - it is always a two way process, and a second supervisor can sometimes play a useful role in getting things back on track. It can and should be inspiring.
Your supervisor plays hugely important roles throughout the process though. At the start, they will help you with the relevant literature and established approaches. As the research progresses, they will help with methodology, with planning the research, and helping you phrase your research questions. Once you have parts of your thesis in draft, they will provide an invaluable critique of the flow of argument, and the construction of your thesis as a piece of rationally-argued writing. Your supervisor will also play a crucial role in selecting your external examiners, and being your supporter and eyes and ears during the viva.
Above all, throughout the process, they are following your journey, engaging in the discussions, playing the part of reader of your thesis and papers, reacting in the ways that your audience and your examiners might to the parts of your work that are new and surprising, so that you can fine-tune your arguments and make sure there are no loose ends.
The relationship between student and supervisor can sometimes be stormy - it is always a two way process, and a second supervisor can sometimes play a useful role in getting things back on track. It can and should be inspiring.
Saturday, June 20, 2009
All models are wrong
.. but some models are useful (George Box et al, 2009, p.61). What makes a model useful? Some theories of science have made grand descriptions in terms of prediction, explanation etc, but it really comes down to a consensus. Today (June 20th) it is reported that the British Government has decided that the spelling rule “I before e except after c” should no longer be taught in schools because the large number of exceptions made it useless. Such a rule is of course one of observation rather than a law of nature, but on close inspection it is easy to find hidden qualifications to any law you care to mention.
Until very recently, many in the scientific community used to imagine that they were discovering the truth about how the physical world works. Whewell (1833, p.256) quotes Lagrange’s opinion “that Newton was fortunate in having the system of the world for his problem, since its theory could be discovered once only”. Now a lake can be discovered only once, but systems are merely constructed, and many refinements and re-interpretations will be possible. Twentieth century physics revealed unimaginable strangeness, needing many alternative and conflicting models to apply to quantum mechanics, diffraction, cosmology, etc., and there was some useful criticism of old notions such as “final causes” (basically, boundary conditions at infinity).
Many researchers in the late nineteenth and early twentieth century searched only for natural laws expressible in terms of differential equations. Since this search followed so closely after the development of the calculus it appears with hindsight that these men with a new hammer suddenly saw nails everywhere.
The same hindsight opens our eyes to the serious untruths in their “natural laws”: on close inspection a natural law does not actually apply everywhere, but only (um..) where it applies (e.g. in the absence of discontinuities, in a neighbourhood of the origin). To be fair, talk of truth or laws was mostly a habit of speech: the models described in these laws are useful in telling us what to expect in the sort of situation for which the model was designed. In other situations or on close inspection we might need a different or more refined model.
As with final causes, or the ether, models can be useful even when they conflict with other models (seem counter-intuitive) or don’t fit with current ideas of causality. For example, classical field theory remains useful, even though we know that action at a distance is impossible, and that there is a better model based on radiation. Non-existent lakes will eventually be removed from atlases, but models will continue as long as somebody finds them useful.
References:
Box, G. E. P; Luceo, A.; Paniagua-quinones, M. d. C. (2009): Statistical Control by Monitoring and Feedback Adjustment, 2nd ed. (Wiley) ISBN 0470148322
Whewell, W. (1833) Astronomy and General Physics: Considered with Reference to Natural Theology
Until very recently, many in the scientific community used to imagine that they were discovering the truth about how the physical world works. Whewell (1833, p.256) quotes Lagrange’s opinion “that Newton was fortunate in having the system of the world for his problem, since its theory could be discovered once only”. Now a lake can be discovered only once, but systems are merely constructed, and many refinements and re-interpretations will be possible. Twentieth century physics revealed unimaginable strangeness, needing many alternative and conflicting models to apply to quantum mechanics, diffraction, cosmology, etc., and there was some useful criticism of old notions such as “final causes” (basically, boundary conditions at infinity).
Many researchers in the late nineteenth and early twentieth century searched only for natural laws expressible in terms of differential equations. Since this search followed so closely after the development of the calculus it appears with hindsight that these men with a new hammer suddenly saw nails everywhere.
The same hindsight opens our eyes to the serious untruths in their “natural laws”: on close inspection a natural law does not actually apply everywhere, but only (um..) where it applies (e.g. in the absence of discontinuities, in a neighbourhood of the origin). To be fair, talk of truth or laws was mostly a habit of speech: the models described in these laws are useful in telling us what to expect in the sort of situation for which the model was designed. In other situations or on close inspection we might need a different or more refined model.
As with final causes, or the ether, models can be useful even when they conflict with other models (seem counter-intuitive) or don’t fit with current ideas of causality. For example, classical field theory remains useful, even though we know that action at a distance is impossible, and that there is a better model based on radiation. Non-existent lakes will eventually be removed from atlases, but models will continue as long as somebody finds them useful.
References:
Box, G. E. P; Luceo, A.; Paniagua-quinones, M. d. C. (2009): Statistical Control by Monitoring and Feedback Adjustment, 2nd ed. (Wiley) ISBN 0470148322
Whewell, W. (1833) Astronomy and General Physics: Considered with Reference to Natural Theology
Subscribe to:
Posts (Atom)
