Another way of approaching this problem (although not identical) is to be able to view and edit several files as if they were one big file (sort of like the “scrivenings mode” in scrivener).

Now if only scrivener had proper PIM functions…

Some readers might think a full text search through one’s documents will be enough when you go back to assemble the notes one will deploy in a dissertation. That may be the case depending on your academic field and the kinds of materials and sources one is working on.

However, during the course of my historical research I have learned the hard way that for the several hundred note files I have (not to even include the digital copies of some of the sources I have) full text search yields less than a quarter of the potential. I have also heard many horror stories from my fellow PhD students on how hard it is to track down information when going back to hunt through one’s year or two of notes taken for a dissertation.

The problems with this “full text search” approach of one’s own notes for a dissertation such as mine are legion. Let us say I want to write a section of my dissertation on women spies or traitors (I do). But the dozens of fragments I have on this through my many files may refer to words like “woman”,”female”,”女”,”破鞋“，or merely the names of the females in questions (川島芳子). Searches for some of these terms, like woman or female will result in hundreds of hits that are completely irrelevant whereas the approach I described will only turn up items I have tagged with something of this nature. While many irrelevant items may turn up, and I may have multiple related tags the software might offer the ability to look through a list of all tags (and merge some of them as delicious.com does) or at any rate, construct certain logical connections, like things tagged “female” and “spy”

Adding a habit of tagging can promote the thinking of one’s fragments as belonging to certain categories of information.

Full text search is powerful and one cannot live without it, but by itself I have time and time again found that I have failed to find (again) many great fragments of historical material, or, spent far too much time playing a search guessing game. Tagging, or the whole approach I suggested above does not completely eliminate the possibility of information getting lost due to poor tagging (or not being able to predict how certain information will be useful) but I think the success of tagging in many applications all over the net shows the advantages of this extra layer of meta data.

Again this may not be the case if one is dealing with different sets of information, smaller sets of notes, or maintain disciplined consistency in terms of what words one uses to describe certain things when taking notes. However, this isn’t the case for me and I suspect many historians out there doing research.

Can some of their solutions work for a historian or other academic researcher in the way I described? I took a look at some of his proposed apps but will continue to explore them.

The short answer – from what I can tell so far, none of these solutions provide anywhere near the simplicity and straight forward middle layer of organization I am looking for. The biggest reason why is that they all seem to share the fact that none of them work well at the primary task for the historian to take notes on documents and assume you are working with digital copies of your primary materials. They are also all prohibitively expensive.

Kerim has pointed to a category of software which could potentially overlap closely with what I’m talking about and I’ll continue to investigate but so far I am very disappointed at the clunky and hugely expensive packages that are anything but intuitive to work with. I think what I have described above is both simpler and can serve as well or better than these solutions. The “coding” approach of social science is intriguing but doesn’t seem to fit well the workflow I’m familiar with as a historian. Perhaps I need a better appreciation for this alternative approach but I would need considerable persuasion.

Hyperresearch $370 – I gave this a spin through their limited edition and I’m afraid I find the interface terrible. I would never want to spend more than a few minutes in this application. Also, I can’t imagine how one would adopt this as a note-taking environment, interface aside. The features I described above should be built upon a strong foundation of an outliner. If the application is not a good solid and comfortable outliner, above all things, then no one will want to actually use the software.

MaxQDA (Windows Only) 900 Euros or 430 Euros educational or 115 Euros for students. This is primarily a text analysis application. It is designed to work with the primary texts themselves. You can replicate some of what I have described by treating your own notes as primary documents but it twists the purpose of the application well beyond what it is designed for. Still investigating once I get trial version up but I don’t see this serving what I described.

Atlas TI (Windows Only) – $140 for students, $595 – still looking more into this. Will report back here. Looks very powerful but again, treating your own notes as the kind of primary documents of analysis here seems to stretch beyond what these are designed for.

nVivo (formerly also NUD*IST) $240 ($595) – looking into it but from tutorials looks like has some of same problems of other “coding” apps mentioned above.

AskSam (Windows Only) – better described as a database application and while it seems to overlap with above solutions seems to have no basic outlining features and trying to be “everyone’s” tool so that it doesn’t fit nicely the needs of a historian like me. Also, tags (keywords) appear to be only file based, not fragment based.

My friend Kerim made a number of comments via twitter which I think I ought to respond to. First is the suggestion of making use of Voodoo Pad.

I love VoodooPad, which I bought in its earliest days and used for some time until eventually abandoning it. It is a great app for the Mac which allows you to create your own personal offline and online Wiki. It has an esasy way of creating wikilinks within each document that can connect to other pages on your own personal wiki. It also supports some hierarchical outline features like most traditional outline software applications. It also has the ability to check “backlinks” to see what links to a given page.

Using this solution it is possible to emulate some of the features I have described here. You can use wikilinks as a kind of tagging feature, typing a wikilink corresponding to a tag or idea after lines of text but this does not allow you to do anything I described in the last half of this posting regarding the dynamic display of fragments in the form of a ‘smart outline’

Also, even in the most recent version, the hierarchical outlining features of VoodooPad are still very basic – it is not really what it is designed for and programs like OmniOutliner are much more robust and rich in features as an outliner in this regard. Tagging is supported in the latest version of the application, but again at the level of files, not fragments from individual sources unless one creates a new file for every fragment from a source.

Thus, while I heartily recommend this wonderful little application for many basic personal organizational needs, I don’t think it can serve as this kind of powerful solution as I described above.