News from Nullspace

Tim Cowlishaw's blog

About me | Twitter | Github

Weeknotes: 22nd July - 29th July 2015

This was my first full week back from holiday, but it didn’t take too long for me to get back into the swing of things! First order fo business was to catch up with Jiyin - she’d begun to work on a server application for our computer use study while I was away, so we looked at that together and discussed what other work needed to be done with it. The main outstanding piece of work is on the user interface - Participants will need to authenticate themselves, define a ‘Todo list’ for the day, and subsequently annotate their daily computing activity with tasks from this list. We spent some time discussing and sketching out how the User Interface could work in each of these cases. We also had a chat about the timing of our study and ways of recruiting participants - we’re in a slight bind as many of the submission deadlines for conferences we might want to publish at are in October, but as the undergraduate students only return at the end of September, we might have trouble recruiting a suitable number of participants before then! We had a chat with John, my second supervisor about this, as he works in the Human-Computer Interaction department, who perform this sort of study quite frequently and he gave us some useful advice - pointing us towards the mailing list for masters students (who are here all summer working on their dissertations), and the psychology department subject pool. In addition, we finalised the design of the pre-questionnaire we’ll use as part of the study, and created a Google form to administer it.

Subsequently, I put together a quick interactive prototype of one of our ideas for the log annotation UI, in order to test it out in something approximating real use. This was a really useful process - for me, it’s almost as quick to knock together an interactive prototype in HTML/CSS/JS using Materialize CSS to provide a library of standard UI elements, as it would be to do the same in something like Balsamiq or UXPin, and allows me to quickly prototype a broader set of interactions than could be done with a standard wireframing tool. This was an invaluable process - trying out our own design and performing ad-hoc usability testing on other colleagues highlighted several shortcomings immediately, and we were able to refine our design to compensate for them.

Most of the rest of the week was spent working on my activity logger software - I gave it a simple system tray icon user interface, and made it output the event log to a CSV file, rather than simply spewing information to the console - the WPF NotifyIcon library, and this blog post on creating a WPF application in F# were both really useful for this. I then let it run while I completed a few simple tasks using my Windows VM as I’d use my regular computer.

Analysing the results was very interesting, it appears the accessibility API events are very low level - referring to the creation, deletion and ordering of individual UI elements. Am now investigating how best to identify higher-level user actions (such as opening an application, saving a file, or switching between windows) from this stream. There’s further metadata about the object which fires the event (in the form of an IAccessible object) which I’m not yet logging, so will look into that that this week, to see if there’s any additional data that will help us identify higher-level actions. I’m also going to investigate methods of automatically identifying repeating patterns in these low level logs that correspond with some higher level action - I’m going to look at some of the literature on learning with categorical time-series here (Which sadly seems a little thin on the ground) to see if offers any clues. I’ve also created logs of single UI ‘gestures’ (minimise a window, open an application, paste some text, etc.) to use as a source of ground truth for this analysis.

I also read through Andreas Vlachos' papers on semi-supervised and active DPMM learning. His algorithm is based on a modified Gibbs sampler, so I also did some revision on Gibbs sampling in general - both BRML and Resnik and Hardisty had loads of useful background here. Vlachos' model definitely seems appropriate to the problem at hand, as I mentioned last week - if you view each underlying task being carried out by a user as a probabilistic process which generates actions and resources, and the entire activity stream as a mixture over these tasks, the DPMM seems like a natural method of modelling it. I’ll definitely look into using Vlachos’s algorithm in the analysis of my field study data, however, I can anticipate some problems in using it (in its current form) in application - In particular, in order to interactively cluster activities into tasks on real usage, we’d need some way of deriving an online algorithm for Vlachos' constrained DPMM model. This is probably a good opportunity for further research. In the coming weeks, I’m planning to have a go at implementing Vlachos’s work myself, in order to better understand the details of how it works.

At the complete other end of the spectrum of my research interests, I’ve also been reading Corbin and Strauss’s Basics of Qualitative Research, in order to learn more about Grounded Theory, and to prepare myself for the user activity study we’re organizing. Jiyin, Emine and I have talked at some length about how we can classify computing activities at a higher level, across different users, and I’m quite keen to use a GT analysis in order to discover a high level classification that reflects users own understanding of their tasks and how they link together, and which reflects the available evidence. I’m planning to write a separate, more detailed post with my notes and thoughts from reading Corbin and Strauss soon, but in the meantime I’m particularly taken with the way that Grounded Theory appears to offer a principled generative process for developing hypotheses and theory from data, in contrast to the traditional quantitative scientific method’s discriminative process for falsifying existing hypotheses and theory. In fact, I can see how the two work very naturally together - using Grounded Theory to develop and discover hypotheses for testing, then quantitative methods to formulate tests for their veracity.

Otherwise, I’ve read Ahmed Awadallah et al’s paper on Supporting Complex Search Tasks, attended a seminar by Maria-Florina Balcan on Learning submodular functions (which was super interesting, and I can see a very obvious application of it to the sort of query recommendation work in Awadallah’s paper that I will write about in more detail soon), and (in a very small way) helped Jeremy Corbyn get nominated for Labour Leader by the Walthamstow CLP on Saturday (more thoughts on which are here).

Interesting links:

Music: Algiers, One track off the new Darkstar album, Wu-Tang Clan, Loads of great stuff on my Spotify discover playlist, In Aeternam Vale, Various stuff on Blue Tapes.