Saturday, March 28, 2009
Here comes the big question. What will you do? You will conduct an investigation, and your account of it and its conclusions will get published. As we have seen, although the computing discipline is all about software, the actual software is not considered published research.
If you are developing new algorithms or software frameworks, think carefully about the journals you want your work to get into. Do they publish that sort of thing? They will probably want them to be evaluated, so think how you can show your work is better that what is currently available.
If you are developing new theory, mathematical approaches, or decision models, then the journals of interest to you are likely not to be interested in the software itself. How will you validate the new mathematical approach?
If you are applying existing ideas in a challenging area, or considering snags in them, then presumably you are coming up with some fresh approaches. Will your suggestions be useful to others? Will your modified success criteria command general acceptance? Can you write up your case study in a way that allows your approaches to work in other situations?
How much can your experiments be generalised? Can you show that the evaluation data you used were in some way typical? The time to consider all of these issues is now, before you start work. By thinking things through at this stage, you will be considering your own work from the viewpoint of a potential reader. Why this test? Why that interviewee? Why that data?
For students we call this the research methodology section - a title that is really too grand for simply ensuring that what we plan to do will be of publishable quality. Being clear on what we can contribute is the first step towards framing the research question, which we will consider next.
Saturday, March 21, 2009
Critically analyse each of the papers in your collection: if one of them is really interesting from your point of view, look to see what papers have cited it. Carefully consider what is the contribution of the paper. Has it given you a theoretical framework for investigation? How has it been validated? What primary evidence does it review? What tests does it apply? How does it demonstrate its value?
Partly these questions allow you to assess the research itself. More importantly for your future research, it allows you to learn about the research culture that is current in your discipline. What sort of experimental analysis is regarded as conclusive, what sorts of aspects are the ones to take into consideration, and what sort of contributions are seen as valuable. You will almost certainly find some aspects of this a little strange. Rather than dismiss them in favour of your own prejudices, though, look in nearby articles to see if conforming to the approach and culture is a price that you will need to pay for getting your ideas published.
There probably aren't enough details in the paper, especially if it is computing related. Retrace the steps it describes and check that you understand all of them. If there is source code, look at it, install it, compile it, and try to trace parts of the execution. If you are going to use this as a starting point of your research, then you really need to be able to repeat the tests it applies. If you get stuck, or really need to contact the authors for their data or code, check that there aren't other avenues to explore before you pester the authors. Bear in mind that not everyone is willing to share their code, for reasons that might be justifiable.
All of this comes under the heading of secondary research, because no matter how much time this takes, none of this is your original work. Even if you improve their description of their framework, implement it on another platform, or run more interesting tests, it is still secondary work.
Just occasionally there might be some primary research that directly follows from such a paper. Looking at its evidence base, or combining it with evidence from another source, you may be able to make, and validate, some significant improvement to their approach, or re-interpret their approach from a very different point of view.
It will be more usual, though, for your primary research to address a research question that is not directly considered in the papers you have found. That will be the next topic in this blog.
Saturday, March 14, 2009
It may be that what we would like to research is simply not in the current research literature. This can happen with software technologies or products. We would like to know more about them, apply them to new situations or take them further. But it won't count as research unless we can find a group of researchers who think it does. This seems a bit like chicken-and-egg - how does a new topic get started?
Maybe we need to find an angle that is in the current research. A good place to start for this is Google Scholar (under more> in the main Google search pages). Search for your idea there and look at the sort of documents that Google Scholar provides.
Now be careful: Google Scholar is a great start, but it contains many things that are not research papers. Watch out for things that seem to be in the academic community and look for how academic researchers describe topics that might be relevant to your line of enquiry.
Try limiting your search to recent documents (since 2006, say).
The next step is to access The Web of Knowledge (http://www.isiknowledge.com/).
All researchers use this. To use it, you need to access it from your university, or from home with your Athens or Shibboleth login. You can use it to review your own publications, to see which publications are sensible to publish in, find the very latest research and see who is working on it.
From there you can go to researcherid.com, which can help you to create a research profile online, identifying all your own publications. Best of all, if you have a research track record you can see how many times your papers are being cited (and by whom): nowadays this is the final measure of your standing in the research community.
You can search for articles by title, content, authors etc. This will all take you a while but don't give up. Everything you need for research is guaranteed to be there somewhere. If you can't find it at first, go back and read the notes above: if it counts it will be there, but you may need to use different words to describe the area of interest.
Saturday, March 7, 2009
To count as research, work must (a) be based on an existing body of knowledge in an academic discipline (b) investigate some question along a path others can follow (c) form conclusions based on public evidence (d) result in a contribution to knowledge according to the standards of the academic community. As I argued in my chapter in "Interdisciplinary Research" (Atkinson et al 2006) this has been the plan for research set out since the days of the founding of the Royal Scociety and other national academies.
Naturally, the identities and expectations of academic communities change over time. The verifiability requirements imposed by different communities are very diverse: the rules for experimental design in medical research are very different from the rules for experimental design in marine biology or the criteria for evidence used in social economics. In the computing community we no longer see articles with titles such as "XYZZY: An Interactive Whizzywig" in our respected journals (and maybe we never did, despite examples such as Stallman 1981).
As a result, research applicants who approach us with ideas such as "I want to use JavaFX on the Semantic Web" pose us a particular problem. It can be easy to see the kind of innovative software they plan to produce, but it fails the research criteria given above. For example, we fail immediately to find examples of research literature that they can use as starting points for their research: the manifest starting point of JavaFX is not really found anywhere in the research literature. At the time of writing (7 March 2009) it can be found in the title of just one MSc thesis and no journal articles. The Semantic Web is of course a different story, and so patiently we try to persuade the student to use start from one of the lines of research in the literature, and maybe find a way to bring JavaFX into the work at some stage.
If software itself is not seen as a contribution to knowledge, it is more obviously useful for other purposes within the research paradigm. It can demonstrate the viability of a mathematical or analytical approach. Programs can be used to conduct experiments on some other research idea. Most such experiments, however, are carried out using throw-away software. Some new way of visualizing something, or a new framework or pattern might be of interest, but the research would be about the impact or usefulness of the approach rather than the actual code.So, what are the options for computing research? We can
- Develop some new theoretical approach or algorithm for problems of some type
- Investigate the best way of doing something (reasonably general)
- (Re)evaluate some suggestions by other researchers in some slightly different context
- Examine the demand for some kind of computing approach
We should try to avoid
- Discussing or comparing solutions from different vendors
- Improving the efficiency of existing solutions
Comments on these ideas are welcome.References:
Atkinson, J., Crowe, M. (eds)(2006) Interdisciplinary Research: Diverse Approaches in Science, Technology, Health and Society, Wiley, Chichester, 978-1-86156-47-2: also Chapter 1: Research Today
Stallman, R.: EMACS: The Extensible, Customizable Display Editor. ACM Conference on
Text Processing, 1981
Update 11 April 2009: Interestingly the recent Research Assessment Exercise RAE2007 in the UK did accept some published software as high-quality research outputs. But I believe the comments above contain good advice for PhD students.
I also enjoy writing software too much, and teaching diverse new topics, and neither of these help to maintain a research focus.
So now I need to choose a new topic, and in case it proves useful to other academics in the same position I plan to document the process. I'm 61 but that's not too old - I still learn new things, and just this week passed one of Sun's professional exams. I have some ideas for different angles to write about in this blog, and hope to add an article about once a week.
Comments and contributions to this blog are welcome - please email me. But put the blog title in the subject line please, as I have a strong email filter.