I am excited to have the opportunity to join THATCamp New England this November and look forward to learning from everyone I meet there. I was asked to post an entry here about the issues that I hope will be discussed at the event. I have no doubt many of the main themes I’m interested in will receive plentiful attention but I would like to bring up two issues that I find of particular importance: 1) the continued need for the appreciation of and promotion of what I’ll call the “little dh” of the digital humanities and 2) an action oriented discussion about the need to plant seeds within each and every department that promotes the cultivation of both real skills and the requisite appreciation for a spirit of experimentation with technology in the humanities not merely among the faculty but even more importantly as a part of the graduate curriculum.

Little dh

I have been unable to keep up with the ever-growing body of scholarship on the digital humanities but what I have read suggests that much of the work that has been done focuses upon the development of new techniques and new tools that assist us in conducting research and teaching in the humanities in roughly four areas: the organization of sources and data (for example Zotero, metadata practices), the analysis of data (e.g. using GIS, statistical text analysis), the delivery and representation of sources and research results (e.g. Omeka) and effective means for promoting student learning (e.g. teaching with clickers, promoting diverse online interactions).

I’m confident that these areas should and will remain the core of Digital Humanities for the foreseeable future. I do hope, however, that there continues to be an appreciation for digital humanities with a small “d” or little dh, if you will, that has a much longer history and I believe will continue to remain important as we go forward. So what do I mean by little dh? I mean the creation of limited, often unscalable, and usually quickly assembled ad hoc solutions tailored to the problems of individual academics or specific projects. In other words, hacks. These solutions might consist of helping a professor, student, or specific research project effectively use a particular combination of software applications, the writing of short scripts to process data or assist in creating workflows to move information smoothly from one application to another, the creation of customized web sites for highly specialized tasks, and so on. These tasks might be very simple such as helping a classics professor develop a particular keyboard layout for a group of students or particular project. It might be more complex, for example, involve helping a Chinese literature professor create a workflow to extract passages from an old and outdated database, perform certain repetitive tasks on the resulting text using regular expressions, and then transform that text into a clean website with automatic annotations in particular places.

The skill set needed to perform “little dh” tasks is such that it is impossible to train all graduate students or academics for them, especially if they have little interest or time to tinker with technology. “Little dh” is usually performed by an inside amateur, for example, the departmental geek, or with the assistance of technology services at an educational institution that are willing to go beyond the normal bounds of “technical support” defined as “fixing things that go wrong.” Unfortunately, my own experience suggests that sometimes the creation of specialized institutes that focus on innovation and technology in education has actually reduced accessibility for scholars to resources that can provide little dh instead of increased it because it is far more sexy to produce larger tools that can be widely distributed than it is to provide simple customized solutions for the problems of individual scholars or projects. One such center to promote innovative uses of technology in education I have seen in action, for example, started out providing very open-ended help to scholars but very quickly shifted to creating and customizing a very small set of tools that may or may not have been useful for the specific needs of the diverse kinds of scholarship being carried out in humanities. There is a genuine need for both, even though one is far less glamorous.

I hope that we can discuss how it is possible to continue to provide and expand the availability of technical competence that can provide help with little dh solutions within our departments and recognize the wide diversity of needs within the academic community, even as we celebrate and increasingly adopt more generalized tools and techniques for our research and teaching.

Planting Seeds

I have been impressed with progress in the digital humanities amongst more stubborn professors that I’ve come across in three areas: 1) an increasing awareness of open access and its benefits to the academic community, 2) an appreciation for the importance of utilizing online resources and online sites of interaction, and 3) the spread of use of bibliographic software amongst the older generation of scholars. This is, to be honest, the only areas of digital humanities that I have really seen begin to widely penetrate the departments I’ve interacted with both as a graduate student and earlier as a technology consultant within a university. I’m now convinced the biggest challenge we face is not in teaching the skills needed to use the software and techniques themselves to the professors and scholars of our academic community, but the pressing need for us to, as it were, “poison the young,” and infect them with a curiosity for the opportunities that the digital humanities offer to change our field in the three key areas of research, teaching, and most threateningly for the status quo, publishing.

There are a growing number of centers dedicated to the digital humanities but I wonder if we might discuss the opening of an additional front, (and perhaps such a front has already been opened and I would love to learn more of it) that attempts to plant a seed of digital humanities within every university humanities department, by asking graduate students to take, or at least offering them the opportunity to take, courses or extended workshops on the digital humanities that focus on: some basic training in self-chosen areas of digital humanities techniques and tools, the cultivation of a spirit of experimentation among students, and finally a more theoretical discussion on the implications of the use of digital humanities for the humanities in general (particularly on professional practices such as publishing, peer review, and the interaction of academics with the broader community of the the intellectually curious public). Promoting the incorporation of such an element into the graduate curriculum will, of course, be a department by department battle, but there are surely preparations that can be made by us as a community, that can help arm sympathetic scholars with the arguments and pedagogical tools needed to bring that struggle into committee meetings at the university and department level.

I recently shifted my note taking to a knowledge database called DEVONthink and as I recently discussed here, I created a special template script to add individual note files (like note cards in the old days) for each fragment or note I take on a source and keep them in a folder with, and linked to a main overview note file for the source. Each fragment created duplicates the tags of the main note file.

Automating a Zotero to DEVONthink Workflow with Applescript

I thought it be nice to automate the creation of a folder and a main note file in DEVONthink for each and every source I have in my Zotero database with a script. I also thought it would be nice if the script brought over any and all tags the source had in Zotero, including tags for whatever collections the source was found in. Finally, I wanted the script to only create folders and note files for those sources that I did not already have a folder for in DEVONthink so that I can run the script every once in a while to keep it synced with my Zotero without too much difficulty.

I created such a script this evening and it can be downloaded here:

Zotero to DEVONthink (Version 1.6 2010.10.16)

To use it:

1. Download and unzip the script

2. Open it in the Applescript Editor and edit the two configuration variables (the name of the group you want to put all the source folders and note files in, and the location of your Zotero database)

3. Put a copy of the saved script into your DEVONthink scripts folder and run it every time you want to import Zotero all your entries or, thereafter, check if there are new sources to be added.

4. Please read my notes in the script for more details on how it works and some things to keep in mind.

Note to developers who can do better:

My script accomplishes its task by using sqlite3 shell commands to directly query the Zotero database. While this is read only and seems to work fine, this is not the most graceful way of going about this.

Zotero has an open API and a developer who is less of an amateur than I could probably think of some effective way of using applescript and perhaps a combination of something else like a Firefox plugin to talk to this API directly in Firefox and get information out of the database that way, which is recommended by the Zotero team. Read more here, here, and here.

Other possible future improvements that I think could be made to this script or some combination of it and a Firefox plugin if someone has the time to work on it. I unfortunately don’t have any time to work on any of these. Please let me know if you add such features and I’ll post updates to the script:

1. If Zotero has any attachments, such as PDFs of the sources or snapshots for the webpages, the script could be improved so that these could be imported along with the entry.

2. It would be great if a formatted bibliographic entry for the source was added to the main note file created in DEVONthink. Currently this must be done by hand by dragging and dropping citation into the note file.

3. Any notes already in the Zotero entry for a source should be added to the main note file created in DEVONthink.

4. Ideally, the script would recreate the collections structure found in Zotero within DEVONthink.

5. Ideally, the script would check to see if any tags have been added or deleted from Zotero and not only add tags on the first import of the source.

6. Ideally, the script would keep track of itemID info for each entries and use that to judge whether an entry has already been imported into DEVONthink. That way the user can shorten or edit the folder titles etc. in DEVONthink without the script re-importing that entry because it doesn’t find an exact match by title.

7. Ideally, the script or a script and a plugin would somehow eliminate the need to be run – that somehow every time I added a new source to Zotero, DEVONthink would automatically get updated.

UPDATE 1.1

-I got rid of a “display dialog” command leftover from my last minute debugginge
-I added a check to see if Firefox is running, and gives the user the opportunity to quit Firefox, otherwise the script cannot run since the Zotero database is locked.

UPDATE 1.5

-The script now puts all the sources in a sub group “_All” (option to change the name) and then recreates the collection group hierarchy you had in Zotero. (Note: if you move or change the hierarchy of folders in Zotero and run the script again later, it will not delete the old groups or move them)

-I added a few more configurations, an option to control whether you are warned that Firefox will be quit, an option to add the author name to the name of the group/note file created

-Firefox is automatically restarted at the end of the script

UPDATE 1.6

-The script now supports logging. Every new group and file created will log an entry in a log file. This can be turned off and the location of the log file can be customized in the configuration section of the script.

UPDATE:

There are some problems with the way it handles unusual titles of books and although the script works fine for me, I have seen some comments saying others are having trouble. Please post your comments on the script at the DEVONthink thread for the script where I hope we can get some help from more skilled scripters who might have time to work more on this:

DEVONthink and Zotero – DEVONthink Forum

To summarize what I wish I had again in terms of a knowledge database:

As I make a note on a source, e.g. recording a single fact, fragment of information, observation, or summary of an idea from a work I want that piece of information to be taggable so that it can be easily found in the future when searching for that tag. I want to be able to add and tag many such notes quickly and efficiently, some of which are “under” others in the form of a hierarchical order, and which then inherit the tags of their parent notes so that I am saved a lot of repetitive tagging. Every single fragment or note must also contain some link, tag, or meta-data which indicates the source it came from (a book, article, archival document, interview, etc.) so that when I use that note in my dissertation or book, I can easily find the source it came from.¹

DEVONthink Pro

I am in the process of shifting my note taking to a powerful knowledge database program called DEVONthink Pro. I was impressed at how quickly and easily I could import all of my nearly one thousand OmniOutliner documents, which I can now preview, search, tag, and group within DEVONthink. I don’t just want to reproduce my existing source-based note structure. I want to experiment with using this application to get just a little closer to my dream knowledge database described above. How am I doing this?

In DEVONthink, I create a group (which is what DEVONthink calls folders) for Sources.

Add a Group for a New Source – Each time I take notes on a new source (a book, movie, archive document, etc.), I create a group for it within this Source folder with the title of the source.

Create and Tag an Overview Document for the Source – In this newly created group I create a new text document with the name of the source in which I give some general information about that source (an overall description or summary) and give it some general tags that well represent the whole source.

Because DEVONthink also creates a gray colored pseudo-tag to every member of a group with the name of the group, any notes that go into this source group will contain a pseudo-tag indicating what source it is from.

Add Notes Using Customized Template Script – After creating and tagging the overview document, every time I want to add a note from this source, I select the overview document and invoke a keyboard shortcut connected to a DEVONthink template I have called “Note On Source” (I’m using Ctrl-Cmd-M) This invokes the creation of a hacked version of an existing template that comes with DEVONthink called “Annotation” written by Eric Böhnisch-Volkmann and modified by Christian Grunenberg. In its modified form the new template script does the following:

a. A new note is created in the source’s group
b. The new note gets a link created by the template script which links to it back to the overview document for the source (assuming it was selected when invoking the script).
c. The new note is then automatically tagged with whatever tags the overview document contained. I can then, of course, add further tags or delete any that may not be relevant to this particular fragment or note.

So what does this method accomplish?

Well, using this method, all my fragments, quotes, and notes from a particular source are together in its own folder, a typical default way of organizing one’s notes. However, every single note can also be found by searching for a particular combination of tags using DEVONthink’s various methods for looking up tagged items. Alternatively, one can create “Smart Groups” that include notes using certain tags. Every note contains a link back to its source, however, both through a direct link in the document, and through its pseudo-tag attached to the originating group. In short, one can find all notes related to certain tags without losing their source (or needing to input it manually in the note), and all notes related to a particular source. The default tagging of new notes on a source saves me a lot of typing, and I can just add any more specific tags relevant for that specific note.

Remaining Issues

Although I’m really impressed with the new 2.x version of the application, there are still a few things that I find less than ideal with DEVONthink to work in, some of which are no fault of the designers, but merely are a result of its developers not having the same specific goals that I have when they created the application.

1. Unlike Yojimbo or Evernote, DEVONthink supports hierarchical groups/folders. This is wonderful, and makes a lot of things possible. However, when selected, parent groups do not list contents of its child groups. Thus if I have a group called “Sources” and a sub-group called “Movies” inside of which I have files or groups related to individual movies, clicking on Sources reveals only an empty folder/group in the standard three pane view (or in icon view, a list of the folders that are in it) instead of all files under it in the hierarchy. Of course, the Finder and other applications often work the same way but it would be fantastic if there was an option to be able to “Go Deep” as one can when viewing folder contents in an application like Leap

2. Although I think someone could further modify the script I hacked to make this work, currently the system as I have it now does not permit no cascading notes: all notes are children of the original source, there aren’t any children notes of notes on a source. Thus the benefits of the kind of hierarchy of bullet points one is used to seeing in a note file is lost.

3. Because almost everything that was originally (in a note taking app like OmniOutliner) fragments that take form as hierarchical bullet points in single document are now fragment files in a hierarchy of folders, much of the power of viewing all of the content of these various fragments together at once is lost. DEVONthink lists all notes as files with single-line names. Ideally my dream note-taking software wouldn’t even need names for the fragments (my hacked version of the script just names them the date plus the name of the source) and would merely directly display the contents of notes so they can be seen juxtaposed with whatever other notes are in the list.

Downloading and Using the “Note on Source” Template

Again, I didn’t write this from scratch, but modified an existing template that comes with DEVONthink Pro. To use it, follow the instructions above. To install it:

1. Download the Script: Note on Source
2. Unzip the script and double click on the _Note on Source___Cmd-Ctrl-M.templatescriptd file inside. DEVONthink Pro will ask you if you want to import or install the template. Choose “Install” and it should now be active with the Cmd-Ctrl-M shortcut or directly in the menu at Data->New from Template->Note on Source

1. My more ambitious and detailed description of this (including the idea of a “smart outline” which would then become possible) can be found in summary form in this posting.

On other occasions I carried another smaller camera in my backpack for emergencies when I wanted to copy some pages out of books but the pictures were often blurry. I recently discovered, however, that the camera on my iPhone 3GS contains a good enough camera to take decent pictures of books and documents if you have moderate indoor lighting.

The Pics Need Processing

To get optimal results however, pictures of books and documents taken from an iPhone 3GS need to be processed: the contrast and brightness need to be turned way up, the size of the image can be significantly reduced in size (from about 1.1MB to 0.25MB each), and if you are making copies of an article or part of a book, ideally you want the result to be a PDF, not a folder full of pictures. Indeed, it is for this purpose I have logged dozens of hours standing in front of the various PDF scanners in the libraries here at Harvard that I wrote about here.

Processing these pictures is time consuming, and begs for a hack. iPhone applications like JotNot and PocketScan are a nice idea but I find them to be incredibly slow and awkward to use.

So I spent a few hours last night and came up with an inelegant but effective solution that, once set up, makes the whole process of getting iPhone pictures processed and into a readable PDF fast and painless. A real hacker would create a script that does all this for the user in a single step, and I would love to get my hands on such a script but in the meantime, in case there is someone out there who would find this useful, here is my current solution using OS X 10.6 and Adobe Photoshop CS3.

Preparations

You only need to do these steps once to get your computer set up. but they are kind of convoluted. I’m sure someone out there has a more efficient method:

1. Create a folder somewhere easy to get to on your hard drive and call it “Convert”

2. Create a folder (in the same folder as Convert for example) and call it “Converted”

3. Open “Automator” in your Applications folder and create a new Automator workflow that looks like this:

Save this as a workflow that we can attach to the “Convert” folder as a folder action. In the top pop-up menu select “Other…” and choose the “Convert” folder which will contain the iPhone photos you will drop in to have converted into a PDF. The applescript will command Photoshop to do an action I have called “CreatePDF” which will process the images one at a time (see below). The automator workflow then grabs all the files, which Photoshop will save into a folder called “Converted” which you should indicate, and create a PDF from them. The final step cleans up the images in the Convert and Converted folder by deleting them. You can delete this step if you don’t want it to delete the images but I usually drop in copies or exported images so I don’t need them once the PDF has been created. You can if you like, download my Automator application version of this workflow here, modify it for your own use and folder locations and save it as a workflow. Keep in mind you need to change the path on the “rm” commands to point to your Convert and Converted folders.

4. Now we need to open Photoshop in order to create two actions. You can see what my actions look like below and create your own version, or download mine here, import them into Photoshop and modify them for your own needs. In the picture below you can see that I have one action called PrepPDF which actually processes a single image by a) changing from color to grayscale b) increasing the brightness and contrast and c) reducing the size of the image and d) saves the image as a JPEG and compresses it significantly. You may find that you want to process it in some different way. The second action, CreatePDF runs Photoshop’s batch command, performing the PrepPDF action on every image it finds in the Convert folder and saves the resulting processed image in the Converted folder.

5. Finally, in the Finder, right click on the “Convert” folder and choose “Folder Actions Setup…” and attach the workflow you created in Automator.

Now things are set up and you will be able to convert your pictures to PDF whenever you like by the means below. You won’t have to repeat the steps above:

If things don’t go right when setting up, make sure the files are all pointing to the right locations, the correct folders, and the correct names for the actions in Photoshop and the action set they are saved in.

Going from iPhone Pictures to Readable PDF

1. Take pictures of the document or book in decent lighting. Click on the screen to focus if it is not focusing properly. Would be nice to put together a nice copy stand to hold the iPhone up while you take pictures, but I’m not that kind of hacker.

2. Import your pictures of the documents/books from your iPhone into iPhoto or, via Image Capture, into your computer somewhere. I don’t recommend importing the pictures from the iPhone directly into the “Convert” folder as the copying process is slow and the script seems to speed ahead of the copying and end up with incomplete PDFs.

3. Open Photoshop. The script should launch it, but I find it work better when it is already open.

4. With Photoshop open, drag and drop your images (or a copy of them, by holding down the option key) into the “Convert” folder. It will run the Automator workflow, which will run the Photoshop action CreatePDF which will run PrepPDF on each picture found in the Convert folder, dump them into the Converted folder after processing them, and when it is done the Automator script will take those processed images in the Converted folder, create a PDF out of them, and delete all the images in both folder so it is clean and ready for the next job. The PDF will be found on the Desktop (this old Automator action seems to be broken in Snow Leopard and I can’t get it to save the PDF anywhere else).

With this I have been able to, even while standing in the stacks of my library, whip out my iPhone and, holding the book open, snap pictures of an interesting chapter etc. and process them quickly and easily into PDFs once I get home. Here is one short example of a PDF created from some pictures taken in the stacks with my iPhone.

Update: If you are converting a lot of pictures into a single PDF, the Applescript in the first command can time out. I added two lines to my workflow to increase the timeout from the default two minutes to 10 minutes:

tell application "Adobe Photoshop CS3"
  with timeout of 600 seconds
    do action "PrepSave" from "Default Actions"
  end timeout
end tell

After trying (and writing reviews of) many different flashcard programs over the years, and even designing some of my own many years ago, I become a loyal daily user of an open source project called Anki (read my review here). It is, in my opinion, the best program around that uses “spaced repetition” or interval study to prompt you only to review information that you are on the verge of forgetting. It helps me keep up on vocabulary in various languages and even serves as kind of daily “meditation of repetitive action” for me.

I can use Anki on my iPhone/iPad through a browser based script called iAnki but there were some things about the layout of the iAnki plug-in which I didn’t think worked well for the big screen of the iPad, which is now my primary way of studying vocab decks when I’m out of the house.

I made some changes to the HTML in the plug-in that I think work better for me. These include:

1. Increasing the font sizes of several fields. 2. Removing the “Show Answer” button and making most of the screen function as a “Show Answer” button so you you don’t need to reach and hit the button. 3. Moving the 1 and 3 buttons to the left edge where I can easily reach them while holding my iPad. 4. Moving the 2 and 4 buttons to the right edge where I can easily reach them.

For anyone out there also using iAnki on an iPad who want to try my hack here is what you do:

1. Download the hacked template here.

2. Unzip it and use it to replace the existing ianki.html file that is in
the iAnki plugin folder. For example, on my Mac the old ianki file is
found:

~/Library/Application Support/Anki/plugins/ianki_ext/templates/
ianki.html

Replace that file with the new one you downloaded.

3. Open up Anki, launch the iAnki plug and install it on your iPad (you’ll need to
install and bookmark it again if you had it installed already)

If you use Anki, please support Damien’s programming efforts in Japan with a donation and congratulate him on his recent marriage.

Such is life, people say to me, or else quote me some snotty French equivalent. That is the reality of this harsh academic world we live in. Well, perhaps I’m suffering from an early onset of old-age grumpiness, but I just don’t want to play that game. I don’t care that I’m still a graduate student, that job committees will look over everything they can find by me in search of sub-standard material, or that publishing firms will want me to explain why an earlier version of something I have submitted to them is available for download somewhere online. I don’t care if someone else finds some topic I have done some preliminary work on interesting, runs with it, and ends up publishing something on it. I may feel a momentary pang of regret that I didn’t get my own butt in gear and finish the project myself, but if they did a good job, then I really have no cause for complaint.

I’ve decided to just go ahead and start posting everything I produce academically, including short conference presentations and other research works in progress. You can find this material on a new research page here at Muninn.

I argued that one of the key elements of this new note taking software would be that every bullet point in one’s notes carried within itself information about where it came from: what its source was.¹ This would allow a graduate student like me who is writing a dissertation or a scholar who is writing a book to drag and drop individual bullet points of notes made on a given source, such as an archival document into a broader outline of a chapter or dissertation without having to make an additional note as to what source I got that bullet point from.

This is particularly useful if, like me, you have dozens of pages of notes, taken from hundreds of archival documents, books, articles, etc. but you want to extract individual points from these sources and compile them into a larger outline as you plan the writing process.

While the tooth fairy never produced such software for me, I have created a little AppleScript hack for my favorite note taking software OmniOutliner that gets me closer to what I want. Simply put, the applescript assigns some text to the “note” field of a group of selected bullet points in an OmniOutliner document. Thus, if those bullet points are dragged and dropped into another document, they will always carry with them whatever source information you assigned to those bullet points.

I created two AppleScripts: One which takes whatever text is in the clipboard and sets the “note” of each selected row in your notes to that text. The second script ignores the clipboard and asks you directly in a dialog box what text you wish to put into the “note” of each selected row. You may download these two scripts from my Huginn script collection:

Set Note to Source

I don’t get it, why would you want to do this? Watch my screencast where I explain what I’m trying to do. If the youtube video below doesn’t appear, you can visit it directly on youtube through this link.

(Sorry about the poor sound quality)

How do you get this work?

1. Download the script.
2. Place the “Set Note to Source in Clipboard” script in one of your script folders (either that for OmniOutliner or in the [your home folder]/Library/Scripts/
3. Open OmniOutliner
4. Select some text that corresponds to the source of some notes that you took, and Copy it via the edit menu (or Command-C).
5. Select one or more rows that you wish assign this source to.
6. Choose the “Set Note to Source in Clipboard” script from the script menu. If your script menu is not visible in the menubar, turn it on.
7. If you find yourself doing this often, consider adding a keyboard shortcut for the script, using triggers for quicksilver or fastscripts or the like.

1. I also suggested adding the ability to easily add tags to a bullet point and those bullet points under it.

I’d be very interested in hearing about how other graduate students have sought to resolve the problem of connecting the large quantity of notes, outlines, and unprocessed raw sources with the grand outline of a huge writing project like a dissertation. Below I describe briefly how I have essentially integrated this process into my own task management routine.

First, let me describe how I have been organizing the historical materials I have been collecting in the field and while back at university. Read on for the details.The primary historical sources I have been working on can be divided into four kinds:

1) Sources I have read but of which I have no hard or digital copy. These are represented on my computer only in the form of the OmniOutliner notes I have taken on these files, along with the appropriate information necessary for citations.

2) Sources which I have only a photocopy hard copy of. Usually I write any important citation information directly on the first page of the document. Then, to each of these sources (which may be primary materials or photocopied books, articles, etc.) I have given an index number composed of a letter (I have used the language the text is in, which in retrospect was not the best way to do things) and an incrementally increasing number so that I can easily refer to my documents in various other files and relocate them in my boxes of files. I have a single OmniOutliner ‘document index’ which lists the name or description of the document, its index number, and the date I found the document. This index number is also noted under the entry for the appropriate day in a separate chronological dissertation log I keep which describes what work I have been doing on my dissertation and the context in which I came across the material). Two examples from my document index (description | index number | date found):

汪偽政府所屬各機關部隊學校團體重要人員名錄 C1010 2008.7.22

BA0155460 사법경찰지도교양재료송부의건 K1003 2008.5.16

3) Sources which I have taken a photographic image of. Some archives and libraries in China, Korea, Taiwan, and the United States allow me to (or at least don’t stop me from) bringing my own camera and taking pictures of the sources I’m looking at. For each source, I usually create a separate OmniOutliner document. In it I take notes on the content of the source, noting any useful information I find in the source. For each photograph of a page of material I take, I record the photograph’s ID number (there is a different numbering scheme for every camera). Sometimes, the notes are very simple, especially if I don’t know if a piece of material will ever be of any value. However, hard disk space is cheap so I take photos of a lot of material I may never look at in any great detail. When I import my photographs, I keep them separated, by source, in folders within my ‘Images’ folder in my ‘Dissertation’ folder. The file name of imported images is the ID number. That way, if I ever need to look at the original image, I can see the ID number in the notes for a source and then search for that number.

For example. In my notes from my reading of the newspaper 青島公報, which is divided into microfilm reels and years, I have an like this in my note file:

青島公報 Reel 1 From 1946.3 to 1946.10

9.1:3 three 漢奸, including a woman ‘cultural hanjian’ 焦墨筠 who gets 3 years and 6 months with other two geting 10 and 15 years each #0508

In this case, this is a short note summarizing an article on page 3 of the September 1st issue of 青島公報 on the sentencing of three Chinese traitors. I will probably never use this information in my dissertation but if I decide at some point to talk more about female traitors or ‘cultural traitors’ then I know I have this short newspaper article on the sentencing of one such woman at the conclusion of her treason trial. In my dissertation log, I find that I was reading through this reel on the afternoon of 2008.12.2 at Shandong Provincial library microfilm room where they let me take somewhat readable photographs of the screen of the microfilm machine. The numeric portion of “#0508″ can be searched for on my hard drive that search will yield an image in the 青島公報 subfolder of my images folder named DSCF0508.JPG that contains the full text of the original article. I use the prefix ‘#’ because I have already taken over 10,000 pictures during the course of my dissertation research and the camera has begun to recycle the numbers. I used to use ‘$’ but to keep these numbers unique in my notes, I now use ‘#’ for the second 9,999 pictures. When I search for 0508 I now get two pictures, but this is hasn’t been too much of a hassle to sort out.

4) Sources which I have a PDF or other digital document for. Many secondary sources or scanned materials I have are available in PDF format, which I have stored among my dissertation files. When I have read these, I record notes for them in an OmniOutliner file. Sometimes, when I have read several articles on a particular topic, I will have a separate OmniOutliner file summarizing the points and arguments from a collection of articles. For example, I have read a series of articles and book chapters on the history of the Shandong column of the 8th route army during wartime China and have compiled notes on this into a separate file I have for historical background on wartime Shandong province. I also have a note file compiling various notes on my notes related to Korea’s treason trials in 1949.

Making the Connections

So the challenge remains, how do I link all these notes, notes on notes, to the actual dissertation without constantly going on a long hunt through all my files?

The way I have approached this problem is to simply use my existing task management software, or ‘to-do’ software or ‘GTD’ (‘Getting Things Done’) software.

I use a program called OmniFocus both on my MacBook Pro laptop computer and on my iPod Touch to manage my tasks (they synch their content with each other). Like many other ‘GTD’ programs out there like Things or iGTD in the case of Mac software, OmniFocus allows you to organize your tasks by projects and contexts. You might first drop tasks into a general ‘inbox’ of unprocessed tasks and then add a project and context for later review if you don’t have time to perform a given task when reviewing the inbox. The former allows you to easy find tasks related to specific projects you are working on while the latter allows you to find tasks that can be done in certain ‘contexts’ of your life. So, for example, I have a ‘web’ and ’email’ context which are sub contexts under ‘comp’ for when I’m working on my computer but also a ‘harvard-yenching’ and ‘shandong provincial library’ and ‘taiwan national library’ context all three which are sub-contexts under the ‘library’ context.

The key efficiency move in this process of connection creation is, however, the combination of project organization and really simple adding of items to that project in the task management software. Many applications like OmniFocus allow you to very quickly and painlessly add tasks to your inbox or specific projects through a ‘quick entry window’ that is accessible in response to a keyboard shortcut. It is with this feature that I have found a simple and easy way to connect my notes with my final chapter writing: In OmniFocus I have created a folder of projects for my dissertation and created separate projects for each of my planned chapters.

Then, as I am taking notes on my sources, each time I finish looking through a source, I reread my notes and try to estimate what chapter the various information I have found can potentially contribute to. I then copy either the name of the source, its file number, etc. (if the notes are relatively few) or a unique phrase or date+phrase from the relevant bullet point in my notes and activate the ‘quick entry window’ of OmniFocus. I assign it the context of ‘writing’ for when I actually get to writing my dissertation, briefly describe what I think the source contributes, and then assign the item the project corresponding to the chapter I believe I will likely have use for the information. If I don’t really know what chapter some material will useful for, I might add it to some other more general topic projects I have added to the dissertation folder of OmniFocus projects, or, for the time being, drop it into the inbox for later consideration.

After adding many such items, not all which will necessarily make it into the dissertation, I return to OmniFocus and review the various items I have added, eliminating those I don’t think will actually be useful when I begin writing, and grouping the various items into hierarchical categories within the chapter projects corresponding to sections of my chapter as I currently imagine it.

What I hope will result from this is a smoother writing process. As I write the various portions of each chapter I can find exact references to individual note files or parts of files where I can find the relevant source material and notes on that source material that will help me make the arguments I am planning to make in the final project. All I have to do is look at the OmniFocus project for a given chapter and I will see a full list of references, grouped by the various points or arguments I hope to make, ready for incorporation into the writing.

As I said in my first posting, I don’t really think this is really all that original a method, and is probably just a variation of some kind of similar process (though perhaps without using software) that many graduate students might use when preparing for a large writing process. What I think is particularly useful with this approach is that I can very quickly add these little references – or pointers to sources, whenever I finish typing up notes on a given source without ever actually leaving the outlining software. It is very fast and simple and hardly interrupts the note taking process.

In a shorter final posting on this topic, I want to suggest how I think this process could be made even more fast and well integrated, if the more powerful outlining applications, such as OmniOutliner for Mac (or competing outlining applications on Windows or Linux), specifically targeted this kind of workflow.

Workshop Wire (The RSS Feed for it is here)

Today I faced this kind of problem. There are lots of “annotator” websites online that make use of the free CEDICT Chinese dictionary but I have yet to find one which outputs a simple, and nicely formated (with all […], and /…/ stuff removed) tab delimited vocab lists.

I have recently been frustrated by the fact that I often come across Chinese characters that I haven’t learn, or, more often, characters that I only know how to pronounce in Japanese or Korean. I also am frustrated at the fact that I have forgotten the tones for a lot of characters I knew well many years ago when I studied Chinese formally.

Over the summer I want to review or learn the 3500 most frequently used Chinese characters, particularly their pronunciation, so that I can improve my tones and more quickly lookup compounds I don’t know.¹

I found a few frequency lists online (see here and here for example) and I stripped out the data I didn’t need to create a list with nothing but one character on each line.² Although it is an older list based on a huge set of Usenet postings from ’93-’94 you can download an already converted list of 3500 characters here.³

Since I’m not in the mood to look up 3500 characters one by one, I spent a few hours this evening using this problem as an excuse to write my second script in the Ruby programming language.

In the remote possibility that others find it useful who are using Mac OS X, you can download the result of my tinkering here:

Cedict Vocabulary List Generator 1.1

This download includes the 2007.8 version of CEDICT, the latest I could find here.⁴

How this script works:

1. After unzipping the download, boot up the “Convert.app” applescript application. It will ask you to identify the file you want to annotate. It is looking for a text file (not a word or rich text file) in Unicode (UTF-8) format with either simplified or traditional Chinese characters or word compounds, one on each line.

2. This application will then send this information to the convert.rb ruby script which will search for the words in the CEDICT dictionary in the same folder, format the information it finds (the hanzi, pinyin, and English definition), including the putting of multiple hits for the same character/word within the same entry with the definitions numbered. It does not currently add the alternate form of the hanzi (it won’t add simplified version to traditional or vice versa).

3. It will then produce a new file with the word “converted” added to its name. It will create tab-delimited files by default but you can change this by changing this option at the top of the convert.rb file in a text editor.

4. Though this version of the script doesn’t do this yet, you may want to run the resulting text through the Pinyin Tone dashboard widget or a similar online tool such as the one here or here. That will get rid of the syllable final tone numbers and add the appropriate tone marks. I am having a bit of trouble converting the JavaScript that my widget and this site uses into Ruby so if anyone is interested in working on this let me know!

If the script doesn’t work: make sure you are saving your text file as UTF-8 before you convert. I am also having trouble when my script is placed somewhere on a hard disk where the path has lots of spaces. Try putting the script folder on your Desktop.

Note: If you don’t have Mac OS X but can run Ruby scripts on your operating system, you may be able to run my script convert.rb from the command line. It takes this format:

convert.rb /path/to/file.txt /path/to/cedict.u8

UPDATE 1.1: The script now replaces “u:” with “ü” (CEDICT uses u:).

1. The top 3000 make up some 98-99% when their cumulative frequency is considered.
2. A few of the frequency lists I have seen have Cedict dictionary data included but not in a very clean format
3. I notice that there is a high frequency of phonetic hanzi for expression emotion in the postings and some other characters one doesn’t come across as often in more formal texts, I actually don’t mind
4. If you find a newer version (in UTF-8) put it in the same directory as my script and name it cedict.u8

I am, however, going through one of my 3-month programming cravings, so I have decided to play with AppleScript Studio and see if I can make an upgrade for one of my favorite old applications (more on that if I ever make any progress). I made a little weblog to chronicle my efforts and leave some tips behind for other beginners who might happen upon it later:

Fool’s Applescript Workshop

Many years ago, before Unicode became dominant, I used a Microsoft Word macro written by a Chinese language scholar, James Dew, as the basis for making an old Mac OS 9 application that translated texts between various pinyin fonts that were floating around online. Later, I made an online script that could convert tone numbers into unicode tone marks. I was surprised to hear from various Chinese language instructors at a conference I presented at a few years later (2003) that many of them used the script regularly when preparing texts for their Chinese language classes.

The online script still works but there is a much more elegantly written online script which does the same thing written by a more skilled programmer in Taiwan named Mark Wilbur hosted on his site Doubting to Shuō. You can find his tool here: Pinyin Tone Tool.

My old PHP script is ugly by comparison to Mark’s compact javascript so I have essentially installed his script to work in an OS X dashboard widget. You can download the widget here:

Pinyin Tone Widget v. 1.02

To use it, just enter the pinyin into the large text field and press the “Add Tone Marks” button. It will add the tones and select the resulting text so you can copy it for use elsewhere with Command-C.

If you like the widget and find it useful, send your thanks to Mark at doubtingtoshuo at gmail.com.

This was my second Dashboard widget. The first was the 檀紀 Conversion Widget that students of Korean history may find useful when reading newspapers and documents from the days when Korea used that calendar more frequently. I really love the free Apple development environment for creating these widgets called Dashcode, which comes with the developer tools for OS X. You just need to know a little javascript and you can be up and running with your own widget.

Next time I need a distraction I hope to create another widget which changes romanized Korean text using “eo” or “eu” into ŏ and ŭ. I won’t be writing a full tool for conversion between McCune-Reischauer and the new government romanization system, which is a pain, but some of us when writing romanized Korean words prefer to keep using the McCune-Reischauer system but have trouble creating the ŏ and ŭ easily directly from the keyboard.

Another calendar widget is in the works too, one for Chinese dates that will convert Qing (I am too lazy to go back further) and republican period calendar years into the Western years, and another for Japanese years that will convert Meiji, Taisho, Showa, and Heisei years via a widget. Anyone who wants to add older imperial reigns can modify the resulting widget when it comes out.

UPDATE: I have updated the widget and changed two things: 1) I saved it to be backward compatible with 10.4.0 instead of 10.4.3 and 2) I changed it so that if you press enter, it will trigger the button for conversion.

UPDATE: None of the widgets I make seem to be compatible with OS X 10.4.11. I’m not sure why but there were some things changed in 10.4.3 regarding dashboard widgets. If you want to upgrade your OS to 10.4.3 then you first need to upgrade to 10.4.2 and then 10.4.3, both of which are free upgrades.

UPDATE: 10.4.11 is a newer version than 10.4.3 – the newest Tiger has to offer (it was 11 as in after 10, not as in after 1 but before 2, my bad). So I guess my widgets are doomed to only work on Leopard. That is lame! Still have no idea why this is…also happened with the calendar widget I just made with far less and different code.

Fixing these tags can be a pain and some of the older tools such as once awesome “MP3 Rage” and “ID3 Editor” often make things worse due to their inconsistent handling of 2-byte non-Roman languages.

An Apple Support page, however, recently pointed me to a great shareware application ($12) called ID3Mod2 which looks like it is made by the same people that made the incredible Chinese input method QIM that I talked about in an earlier posting (I don’t know this developer personally so it is not as if I’m trying to find good things to say about their work). You can freely use the software for a number of days, during which I was able to go through and fix all of the garbled tags in music files I have collected in China, Korea, and Japan over the last decade. Amazing – I might now actually learn the names of some of the songs I have been listening to for so long and someday even gather the courage to request them on a future karaoke adventure.

To save me a few keystrokes I wrote the following simple applescript to take the current page open in Safari and open it in a new tab in Firefox:

tell application “Safari”
   activate
   set my_URL to the URL in document 1
end tell

–Convert Unicode Text of my_URL to Plain text
set my_URL to «class ktxt» of ( (my_URL as string) as record)

tell application “Firefox”
   activate
   Get URL my_URL
end tell

Download this as an applescript or as an applescript compiled application that you can invoke easily from Quicksilver.

See the script below.

Remember to change the “[PATH TO FILE]” to the location in your home where you want this info to be saved (for example /Users/ephialtes/Documents/statistics”

You can easily modify this to ask your own questions. This script is not very efficient either and doesn’t check for bad input on your part.

–Choose day to record this will start each line of the log files for this day
set my_date to ( (current date)’s year as string) & “.” & ( (current date)’s month as integer) as string) & “.” & ( (current date)’s day as string)
set chosen_date to display dialog “Which day are you recording stats for?” default answer my_date
set chosen_date to text returned of chosen_date

–Ask the questions you want to record data for and save them in variables
set bedtime to text returned of (display dialog “What time did you go to bed yesterday?” default answer “”)
set hours_slept to text returned of (display dialog “How many hours did you sleep?” default answer “”)

set health_notes to text returned of (display dialog “How do you feel today?” default answer “”)

set bfast to text returned of (display dialog “What did you eat for breakfast?” default answer “”)
set lunch to text returned of (display dialog “What did you have for lunch?” default answer “”)
set dinner to text returned of (display dialog “What did you have for dinner?” default answer “”)
set snack to text returned of (display dialog “Did you have any snacks or other food?” default answer “”)

set exercise to text returned of (display dialog “Did you exercise? If so, what did you do?” default answer “”)
set my_weight to text returned of (display dialog “Did you weight yourself? If so, how do you weigh in kg?” default answer “”)

–Write the files. I have chosen to save the answers in different files according to my own needs
set sleep_loc to “/Users/[PATH TO FILE]/sleep.txt”
set sleep_file to POSIX file sleep_loc
write_to_file(((chosen_date & tab & bedtime) & tab & hours_slept & tab & sleep_notes & return), sleep_file, true)

if health_notes is not “” then
   set health_loc to “/Users/[PATH TO FILE]/health.txt”
   set health_file to POSIX file health_loc
   write_to_file( (chosen_date & tab & health_notes & return), health_file, true)
end if

set food_loc to “/Users/[PATH TO FILE]/food.txt”
set food_file to POSIX file food_loc
write_to_file( (chosen_date & tab & bfast & tab & lunch & tab & dinner & tab & snack & return), food_file, true)

if exercise is not “” then
   set exercise_loc to “/Users/[PATH TO FILE]/exercise.txt”
   set exercise_file to POSIX file exercise_loc
   write_to_file( (chosen_date & tab & exercise & return), exercise_file, true)
end if

if my_weight is not “” then
   set weight_loc to “/Users/[PATH TO FILE]/weight.txt”
   set weight_file to POSIX file weight_loc
   write_to_file( (chosen_date & tab & my_weight & return), weight_file, true)
end if

–The following useful file writing routine found on Apple’s website.
on write_to_file(this_data, target_file, append_data)
try
set the target_file to the target_file as text
set the open_target_file to ¬
open for access file target_file with write permission
if append_data is false then ¬
set eof of the open_target_file to 0
write this_data to the open_target_file starting at eof
close access the open_target_file
return true
on error
try
close access file target_file
end try
return false
end try
end write_to_file