News from Nullspace

Tim Cowlishaw's blog

About me | Twitter | Github

Weeknotes: 29th July - 5th August 2015

I started this week by catching up with Filip on my progress with the logger. This was very useful - he was able to help with a couple of boring technical practicalities which I’d been struggling with related to calling the Accessibility APIs from F#. It appears that a lot of this is very sparsely documented, so it’s really useful to be able to talk through what I’m doing with someone who’s done similar things before!

One particular issue I’d dealt with the previous week is the fact that the accessibility events we’re listening for are very low-level - relating to individual keypresses, mouse gestures and interactions with UI elements. This is a very rich source of information, however, I’m specifically interested in higher level information - when focus is switched between applications, when URLS and files are opened and closed, that sort of thing. Now, this information is clearly captured in the low-level stream of events, however, identifying the ‘signature’ of a set of low level events which make up a higher-level interaction is tricky. For the case of identifying focus switches, however, we came up with the idea of polling for the currently focused application periodically, alongside capturing the event data. That way, we can use the polled ‘current application’ data as training data in order to learn when context switches occur from the event data.

We also briefly discussed one of the more interesting (to my mind) research applications of the technology - providing data on the personal provenance of a piece of information (the path a user took to discover it) for the purposes of information refinding - Filip was very interested in this, which was most encouraging.

Much of the rest of the week was spent doing further work on the tasklogger, In particular, attempting to get more information about the accessible object which was the subject of an accessibility event. So far, I’ve had limited success with this - I can get the accessibility object for the window for which an event occurred, and then traverse its children, aggregating all the text properties I can find. However, it appears that many applications implementation of this is very inconsistent, so I have to guard against all sorts of interesting conditions where an application doesn’t fulfil the contracts of the IAccessible interface. I now need to extend this work to do the same for the specific IAccessible which fired the event, which might give us some more useful contextual information about the underlying activity. I am also going to implement the ‘polling’ idea discussed with Filip above.

This week I also arranged and installed the server side of our logger application on a VPS hosted within the computer science department (it’s important to use an internal machine for ethics and data protection reasons). This was a pretty simple process, however, the UCL information services department have a few additional security requirements due to the sensitive nature of the data we’re gathering. In particular, I need to ensure that our log data (in a MongoDB database) is encrypted at rest - I’ll be looking into how best to do this this week!); I also submitted the (hopefully) final set of revisions to our research ethics application and await a decision!

I’ve also continued my reading on grounded theory, as well as Dumas et al.’s Stuff I’ve seen paper which provides some excellent background on IR systems for refinding, as well as experimental results into effective cues for information refinding. I also read This recent paper by Guha et al. on the user modelling approach taken by the Google Now system, which is useful both for the insights into the user modelling process itself, as well as the practicalities of implementing it in production.

In other news, I started using Spacemacs on a whim, and it appears to be replacing vim as my main text editor. Taking the best ideas from vim (Modal editing, a leader key with memorable and ergonomic shortcuts), and implementing them within Emacs, while providing an excellent set of configuration defaults (and a well-thought-out means of extending them) is a complete winner to me.

Outside work, My friend Tom came to stay for the weekend, and we had a great bike ride out through Epping Forest, along with Ben and Aanand. I also spent Saturday afternoon out leafleting for the Jeremy Corbyn campaign in Walthamstow town centre, which was a thoroughly enjoyable experience - I met so many of our neighbours who were friendly and enthusiastic about the campaign. Finally, on Monday evening, Tom, Vicky and myself all went to Cafe OTO for a gig put on by my friends at The Quietus featuring Hirvikolari, Chrononautz and Sly and the Family Drone which was absolutely incredible.

Interesting links:

Music: Empress of, A load of Blue Tapes stuff, Wolf Alice, Leonard Cohen, Martin Dubka, The Membranes, Jam City, and a lot of disco records.

Weeknotes: 22nd July - 29th July 2015

This was my first full week back from holiday, but it didn’t take too long for me to get back into the swing of things! First order fo business was to catch up with Jiyin - she’d begun to work on a server application for our computer use study while I was away, so we looked at that together and discussed what other work needed to be done with it. The main outstanding piece of work is on the user interface - Participants will need to authenticate themselves, define a ‘Todo list’ for the day, and subsequently annotate their daily computing activity with tasks from this list. We spent some time discussing and sketching out how the User Interface could work in each of these cases. We also had a chat about the timing of our study and ways of recruiting participants - we’re in a slight bind as many of the submission deadlines for conferences we might want to publish at are in October, but as the undergraduate students only return at the end of September, we might have trouble recruiting a suitable number of participants before then! We had a chat with John, my second supervisor about this, as he works in the Human-Computer Interaction department, who perform this sort of study quite frequently and he gave us some useful advice - pointing us towards the mailing list for masters students (who are here all summer working on their dissertations), and the psychology department subject pool. In addition, we finalised the design of the pre-questionnaire we’ll use as part of the study, and created a Google form to administer it.

Subsequently, I put together a quick interactive prototype of one of our ideas for the log annotation UI, in order to test it out in something approximating real use. This was a really useful process - for me, it’s almost as quick to knock together an interactive prototype in HTML/CSS/JS using Materialize CSS to provide a library of standard UI elements, as it would be to do the same in something like Balsamiq or UXPin, and allows me to quickly prototype a broader set of interactions than could be done with a standard wireframing tool. This was an invaluable process - trying out our own design and performing ad-hoc usability testing on other colleagues highlighted several shortcomings immediately, and we were able to refine our design to compensate for them.

Most of the rest of the week was spent working on my activity logger software - I gave it a simple system tray icon user interface, and made it output the event log to a CSV file, rather than simply spewing information to the console - the WPF NotifyIcon library, and this blog post on creating a WPF application in F# were both really useful for this. I then let it run while I completed a few simple tasks using my Windows VM as I’d use my regular computer.

Analysing the results was very interesting, it appears the accessibility API events are very low level - referring to the creation, deletion and ordering of individual UI elements. Am now investigating how best to identify higher-level user actions (such as opening an application, saving a file, or switching between windows) from this stream. There’s further metadata about the object which fires the event (in the form of an IAccessible object) which I’m not yet logging, so will look into that that this week, to see if there’s any additional data that will help us identify higher-level actions. I’m also going to investigate methods of automatically identifying repeating patterns in these low level logs that correspond with some higher level action - I’m going to look at some of the literature on learning with categorical time-series here (Which sadly seems a little thin on the ground) to see if offers any clues. I’ve also created logs of single UI ‘gestures’ (minimise a window, open an application, paste some text, etc.) to use as a source of ground truth for this analysis.

I also read through Andreas Vlachos' papers on semi-supervised and active DPMM learning. His algorithm is based on a modified Gibbs sampler, so I also did some revision on Gibbs sampling in general - both BRML and Resnik and Hardisty had loads of useful background here. Vlachos' model definitely seems appropriate to the problem at hand, as I mentioned last week - if you view each underlying task being carried out by a user as a probabilistic process which generates actions and resources, and the entire activity stream as a mixture over these tasks, the DPMM seems like a natural method of modelling it. I’ll definitely look into using Vlachos’s algorithm in the analysis of my field study data, however, I can anticipate some problems in using it (in its current form) in application - In particular, in order to interactively cluster activities into tasks on real usage, we’d need some way of deriving an online algorithm for Vlachos' constrained DPMM model. This is probably a good opportunity for further research. In the coming weeks, I’m planning to have a go at implementing Vlachos’s work myself, in order to better understand the details of how it works.

At the complete other end of the spectrum of my research interests, I’ve also been reading Corbin and Strauss’s Basics of Qualitative Research, in order to learn more about Grounded Theory, and to prepare myself for the user activity study we’re organizing. Jiyin, Emine and I have talked at some length about how we can classify computing activities at a higher level, across different users, and I’m quite keen to use a GT analysis in order to discover a high level classification that reflects users own understanding of their tasks and how they link together, and which reflects the available evidence. I’m planning to write a separate, more detailed post with my notes and thoughts from reading Corbin and Strauss soon, but in the meantime I’m particularly taken with the way that Grounded Theory appears to offer a principled generative process for developing hypotheses and theory from data, in contrast to the traditional quantitative scientific method’s discriminative process for falsifying existing hypotheses and theory. In fact, I can see how the two work very naturally together - using Grounded Theory to develop and discover hypotheses for testing, then quantitative methods to formulate tests for their veracity.

Otherwise, I’ve read Ahmed Awadallah et al’s paper on Supporting Complex Search Tasks, attended a seminar by Maria-Florina Balcan on Learning submodular functions (which was super interesting, and I can see a very obvious application of it to the sort of query recommendation work in Awadallah’s paper that I will write about in more detail soon), and (in a very small way) helped Jeremy Corbyn get nominated for Labour Leader by the Walthamstow CLP on Saturday (more thoughts on which are here).

Interesting links:

Music: Algiers, One track off the new Darkstar album, Wu-Tang Clan, Loads of great stuff on my Spotify discover playlist, In Aeternam Vale, Various stuff on Blue Tapes.

Why I am supporting Jeremy Corbyn for Labour Leader

I’m absolutely thrilled to bits that Walthamstow CLP voted to give our supporting nomination for the Labour leadership to Jeremy Corbyn on Saturday. Due to some procedural irregularities, not everyone who wanted to speak in support of a candidate had a chance to do so. If I had done, here is what I would have said:

Good morning everyone, my name’s Tim Cowlishaw, and I’ve been a member of the Labour party for a little under three months. While I don’t have a history of membership and campaigning that’s as long or venerable as those of of you who have already spoken, I suspect that the perspective of a new member might be a valuable contribution to this discussion.

As will come as no surprise to those of you I spoke to or thrust a leaflet at outside, I’m supporting Jeremy Corbyn as the next leader of the Labour party, and I am doing so in a spirit of both principle, and pragmatism.

Jeremy is the right candidate for the leadership from a point of view of principle, as so far, he is the only one of the four who has articulated a vision of what the Labour party is actually for. Rather than simply making a case for how he will win a general election, he’s powerfully and plainly outlined why he would want to do so, and what he would do in power. He is the only candidate who is making the very straightforward point that the Labour party exists to represent the interests of the working people of this country, and who also has a concrete plan of action to represent those interests.

Now, many will argue that this sort of idealism is all very well at a CLP meeting, but that it doesn’t win general elections. I don’t agree, and further, I’d argue that this is not necessarily even a useful concern at this point in the history of the party. Therefore, I think that as well as being a principled choice of leader, Jeremy is also a pragmatic one. With five years until the next general election, what the party needs in a leader is someone with the courage of their convictions, and the ability to make a case for these convictions, providing a visible and viable opposition to the Conservatives' economic illiteracy and social barbarism. I’d argue that doing so will bring dividends at the ballot box come 2020, but more importantly, I believe that our role as the party of working people means that this is even more important for its own sake. Further, to those who do believe our electability is an imminent and overriding concern - I would simply point to the massive outpouring of support for Jeremy from all sections of society - from Labour members of many years who stayed silent as we tacked towards the centre ground, and from people like me who are new to organized political activism, especially those younger than me, who, for perhaps the first time, have seen a politician at Westminster who cares about their interests. I’d argue that the popularity of Jeremy’s campaign is a better indicator of his popularity in the country at large than the reckons of those party functionaries at headquarters who for some reason believe that they’ve a miraculous gift to predict the future.

We live in a country with a GDP per head of twenty-five thousand pounds a year, but a quarter of us live in poverty. We’ve a proud tradition of representative democracy, but thirty-five percent of the electorate felt so disenfranchised by Westminster politics that they didn’t even turn up to vote last May. I believe we should see both of these facts as nothing less than a dereliction of our duty as a party (and as Labour activists, councillors and MPs) to stand up for the rights of the working people of this country. I believe that of all the candidates on the ballot, Jeremy Corbyn is the only one with the vision, and the track record as a campaigner, to put that right, and put it right we must, both because it is right in and of itself, and because once we start representing the working people of this country again, they’ll start voting for us again. Thank you.

Weeknotes: 4th July - 22nd July 2015

For the last two weeks, I was on holiday in France!

I did find the time to read Don Norman’s “The Design of Everyday Things” while I was away though, which was a really useful introduction to the psychology of interaction, and refresher of some basics of interaction design, and the human-centred design process. At times I felt that he places a little too much faith in the power of the design process to anticipate future needs and problems (Complex systems produce emergent, unpredictable properties and effects - something that isn’t mentioned), but in general it’s an excellent work.

There’s only been two days since I got back, so these notes will be brief. I’ve continued to work on my desktop activity logger - designing and building a user interface around the logging software I’ve already developed, which has been a bit of a crash course in Windows GUI programming. Not really understanding the trade-offs, I fairly arbitrarily decided to use WPF.aspx) instead of Windows Forms, then subsequently realised this made it more difficult to create a system tray icon, since this is a Windows Forms component. happily, the WPF NotifyIcon library provides a solution, and after finally getting it to load (F# does not automatically load libraries referenced in XAML files), and working out how to add an icon image to a Visual Studio project to, I’m now getting along nicely and hope to have the logger software finished this week.

Jiyin has started writing a server application to store the data for our study, and I’ve been working on deploying it to a server for testing. We’ve also been prototyping the user interface for our study participants to annotate their activity logs, and will begin implementing that next week. We’ve also designed the pre-questionnaire for the study and submitted it for ethical approval.

Interesting links: (A bumper crop since I’ve had plenty of time to read!)

Music: Intergalactic FM - a non stop mix of incredible disco and italo, Richard Skelton (and his ‘The Inward Circles’ alter ego) and Chrononautz.

Weeknotes: 21st June - 3rd July 2015

(A bumper lot this week, as I’m off on holiday this morning and have been rather busy getting things finished beforehand! From now on, I’m going to write my weeknotes from Wednesday to Wednesday, in order to have them coincide with my weekly supervisory meetings, since this is when I typically need to take stock of my progress and plan the next week.)

This week I’ve spent a load of time working on my task tracker. I’m using the Windows accessibility API in order to listen for and log UI interactions - however, I was having real trouble working out how to get information about the process that fired the event. With some help from Filip (and with reference to the code that Emine’s Masters student from last year wrote for her own study), I’ve finally managed to crack it, learning a hell of a lot about PInvoke, and calling C APIs from F# in the process. I’m now working on building a simple user interface for the logger, and am going to then leave it running in the background for a few days (while using my windows VM for everyday tasks) and look at the distribution of events captured in order to work out where I should focus my efforts next. I’m also going to look into writing a version for OSX too - since that’s what I use day to day, it’ll make it a lot easier to evaluate on my own usage.

I’ve also reviewed the literature on exploratory search - this was prompted by reading the findings in one of Manisha and Emine’s publications which suggested that the technique they were proposing performed better on exploratory queries as opposed to focused ones. I’d been thinking about this distinction for a while (especially with reference to Si├ón Lindley et al’s classification of daily internet browsing habits), so was very glad to find a way into the existing literature on the topic. White and Roth’s book on the subject has proved to be really useful, and has tied together several disparate strands of my current research. In particular, there’s two arguments that they make that are especially relevant to my research: Firstly, that systems which support exploratory search behaviours should treat search behaviours as a long-term, iterative and synergistic process (which chimes very nicely with my main research focus - the use of holistic, long-term activity logs for information retrieval). Secondly, White and Roth make reference to the trade-off between Recall and Precision, and identify the fact that exploratory search requires a higher level of recall compared to typical focused search tasks. I’ve been thinking for a while about how active learning techniques could be incorporated into search and retrieval systems, and one of the main challenges I’ve come across is reconciling the user’s need to be shown the most-likely relevant results, while the active learning system requires the user to judge the most-uncertain results - this is roughly analogous to the explore-exploit trade-off of reinforcement learning. White and Roth’s assertion that recall matters more than precision to users engaged in exploratory search activities is particularly interesting in light of this, as it suggests a situation in which the user’s interests and the learning systems interest would be more closely aligned - since the user privileges recall at the expense of precision, the system can elicit judgements on uncertain examples without loss of user satisfaction. This process could even be seen as a form of computer-aided serendipity in which the user and system jointly explore the frontier of relevance.

In other Active-Learning news, I also reviewed a couple of papers on active semi-supervised clustering which elicit pairwise must-link and cannot-link constraints from users in order to arrive at an optimal clustering. I can see a couple of challenges for my use case - in particular, my particular application would really benefit from an online learning algorithm - since a users activity log is longitudinal, with new activities (and new judgements) arriving sequentially, it makes sense to use an online algorithm, rather than re-training from scratch periodically. In addition, we can reasonably expect the number of clusters to vary dynamically over time - while both approaches I read about allow for the number of clusters to be inferred, the time-varying nature of our particular problem is a little more subtle. Intuitively, the problem of assigning interactions or resources to ongoing activities sounds a lot like a Chinese Restaurant Process, so I’m planning on looking into the literature on Dirichlet process mixture models, in order to see if there’s anything useful there. In particular, Andreas Vlachos (a member of Sebastian’s lab who are also based in our office) has done some work on active learning for DPMMs, which might well be relevant here - I will look into this in the coming weeks.

Jiyin and I continued preparing for our field study - making amendments to our ethics committee application, and working out how we will store the activity log data we collect for further analysis. We’ve devised a simple schema for log messages which incorporates all the data produced by both our loggers, and are now working on a system to persist it. Given the log data is just a stream of JSON objects, delivered over HTTP, I suspect Logstash (with the HTTP input) might well do what we need without us actually having to do any development work, which would be nice. I’ll be looking into this when I’m back from holiday. As part of the process of completing our ethics application, we also had to go back over our proposed methodology, which highlighted a couple of shortcomings. In particular, in soliciting task labels from users, we’ve opted to allow users to describe their tasks with free text labels, in order to reflect our lack of prior knowledge about the participants' activities. However, this has one important drawback, in that it does not allow us to make comparisons between participants, given that there’s no common set of descriptors for tasks. In order to address this, we could come up with a higher-level set of ‘task categories’ shared between users, to which they can assign each task - however, the issue then is deciding upon a principled set of categories which provides adequate coverage. Another approach (which I prefer), would be to elicit commonalities between participants as part of the analysis process, or in interviews after the study. Grounded Theory offers a useful methodology for this, and the Affinity diagram would be a useful technique to elicit task categorisations in the post-study interview. I’m going to read up more on this and formulate a more concrete methodology next week.

We had the first meeting of our office Machine Learning interest group on the 2nd, including staff and students from UCL, and BBC R&D. It was great to find out more about others interests and activities, and to explore useful ways of supporting each other. We decided to get started by agreeing on a paper to read and discuss in a ‘journal club’ format, and to reconvene in a month. We’re still deciding on our first paper, but the general consensus was that we’d like to go back to some of the original publications on deep learning, and develop an understanding of it from the ground up.

Interesting links:

Music: Petrels, Tim Hecker, Vessels, Holly Herndon, Lorenzon Senni, Alessandro Cortini, Fennesz, Some incredible South African disco, Laurel Halo, Julius Eastman, Arthur Russell, Sparks, Pet Shop Boys, East India Youth and Shit and Shine.