05 Aug 2015
I started this week by catching up with Filip on my progress with the logger. This was very useful - he was able to help with a couple of boring technical practicalities which I’d been struggling with related to calling the Accessibility APIs from F#. It appears that a lot of this is very sparsely documented, so it’s really useful to be able to talk through what I’m doing with someone who’s done similar things before!
One particular issue I’d dealt with the previous week is the fact that the accessibility events we’re listening for are very low-level - relating to individual keypresses, mouse gestures and interactions with UI elements. This is a very rich source of information, however, I’m specifically interested in higher level information - when focus is switched between applications, when URLS and files are opened and closed, that sort of thing. Now, this information is clearly captured in the low-level stream of events, however, identifying the ‘signature’ of a set of low level events which make up a higher-level interaction is tricky. For the case of identifying focus switches, however, we came up with the idea of polling for the currently focused application periodically, alongside capturing the event data. That way, we can use the polled ‘current application’ data as training data in order to learn when context switches occur from the event data.
We also briefly discussed one of the more interesting (to my mind) research applications of the technology - providing data on the personal provenance of a piece of information (the path a user took to discover it) for the purposes of information refinding - Filip was very interested in this, which was most encouraging.
Much of the rest of the week was spent doing further work on the tasklogger, In particular, attempting to get more information about the accessible object which was the subject of an accessibility event. So far, I’ve had limited success with this - I can get the accessibility object for the window for which an event occurred, and then traverse its children, aggregating all the text properties I can find. However, it appears that many applications implementation of this is very inconsistent, so I have to guard against all sorts of interesting conditions where an application doesn’t fulfil the contracts of the IAccessible interface. I now need to extend this work to do the same for the specific IAccessible which fired the event, which might give us some more useful contextual information about the underlying activity. I am also going to implement the ‘polling’ idea discussed with Filip above.
This week I also arranged and installed the server side of our logger application on a VPS hosted within the computer science department (it’s important to use an internal machine for ethics and data protection reasons). This was a pretty simple process, however, the UCL information services department have a few additional security requirements due to the sensitive nature of the data we’re gathering. In particular, I need to ensure that our log data (in a MongoDB database) is encrypted at rest - I’ll be looking into how best to do this this week!); I also submitted the (hopefully) final set of revisions to our research ethics application and await a decision!
I’ve also continued my reading on grounded theory, as well as Dumas et al.’s Stuff I’ve seen paper which provides some excellent background on IR systems for refinding, as well as experimental results into effective cues for information refinding. I also read This recent paper by Guha et al. on the user modelling approach taken by the Google Now system, which is useful both for the insights into the user modelling process itself, as well as the practicalities of implementing it in production.
In other news, I started using Spacemacs on a whim, and it appears to be replacing vim as my main text editor. Taking the best ideas from vim (Modal editing, a leader key with memorable and ergonomic shortcuts), and implementing them within Emacs, while providing an excellent set of configuration defaults (and a well-thought-out means of extending them) is a complete winner to me.
Outside work, My friend Tom came to stay for the weekend, and we had a great bike ride out through Epping Forest, along with Ben and Aanand. I also spent Saturday afternoon out leafleting for the Jeremy Corbyn campaign in Walthamstow town centre, which was a thoroughly enjoyable experience - I met so many of our neighbours who were friendly and enthusiastic about the campaign. Finally, on Monday evening, Tom, Vicky and myself all went to Cafe OTO for a gig put on by my friends at The Quietus featuring Hirvikolari, Chrononautz and Sly and the Family Drone which was absolutely incredible.
Music: Empress of, A load of Blue Tapes stuff, Wolf Alice, Leonard Cohen, Martin Dubka, The Membranes, Jam City, and a lot of disco records.
29 Jul 2015
This was my first full week back from holiday, but it didn’t take too long for
me to get back into the swing of things! First order fo business was to catch up
with Jiyin - she’d begun to work on a server application for our computer use
study while I was away, so we looked at that together and discussed what other
work needed to be done with it. The main outstanding piece of work is on the
user interface - Participants will need to authenticate themselves, define
a ‘Todo list’ for the day, and subsequently annotate their daily computing
activity with tasks from this list. We spent some time discussing and sketching
out how the User Interface could work in each of these cases. We also had a chat
about the timing of our study and ways of recruiting participants - we’re in
a slight bind as many of the submission deadlines for conferences we might want
to publish at are in October, but as the undergraduate students only return at
the end of September, we might have trouble recruiting a suitable number of
participants before then! We had a chat with John, my second supervisor
about this, as he works in the Human-Computer Interaction department, who perform this sort
of study quite frequently and he gave us some useful advice - pointing us
towards the mailing list for masters students (who are here all summer working
on their dissertations), and the psychology department subject pool. In
addition, we finalised the design of the pre-questionnaire we’ll use as part of
the study, and created a Google form to administer it.
Subsequently, I put together a quick interactive prototype of one of our ideas
for the log annotation UI, in order to test it out in something approximating
real use. This was a really useful process - for me, it’s almost as quick to
knock together an interactive prototype in HTML/CSS/JS using Materialize CSS to provide a library of standard UI elements, as it would be to do the same in something like Balsamiq or UXPin, and allows me to quickly prototype a broader set of interactions than could be done with a standard wireframing tool. This was an invaluable process - trying out our own design and performing ad-hoc usability testing on other colleagues highlighted several shortcomings immediately, and we were able to refine our design to compensate for them.
Most of the rest of the week was spent working on my activity logger software
- I gave it a simple system tray icon user interface, and made it output the
event log to a CSV file, rather than simply spewing information to the
console - the WPF NotifyIcon library, and this blog post on creating a WPF application in F# were both really useful for this. I then let it run while I completed a few simple tasks using my Windows VM as I’d use my regular computer.
Analysing the results was very interesting, it appears the accessibility API
events are very low level - referring to the creation, deletion and ordering of
individual UI elements. Am now investigating how best to identify higher-level
user actions (such as opening an application, saving a file, or switching
between windows) from this stream. There’s further metadata about the object
which fires the event (in the form of an IAccessible object) which I’m not yet logging, so will look into that that this week, to see if there’s any additional data that will help us identify higher-level actions. I’m also going to investigate methods of automatically identifying repeating patterns in these low level logs that correspond with some higher level action - I’m going to look at some of the literature on learning with categorical time-series here (Which sadly seems a little thin on the ground) to see if offers any clues. I’ve also created logs of single UI ‘gestures’ (minimise a window, open an application, paste some text, etc.) to use as a source of ground truth for this analysis.
I also read through Andreas Vlachos' papers
on semi-supervised and
active DPMM learning. His
algorithm is based on a modified Gibbs sampler, so I also did some revision on
Gibbs sampling in general - both BRML and Resnik and Hardisty had loads of useful background here.
Vlachos' model definitely seems appropriate to the problem at hand, as
I mentioned last week - if you view each underlying task being carried out by
a user as a probabilistic process which generates actions and resources, and
the entire activity stream as a mixture over these tasks, the DPMM seems like
a natural method of modelling it. I’ll definitely look into using Vlachos’s
algorithm in the analysis of my field study data, however, I can anticipate some
problems in using it (in its current form) in application - In particular, in
order to interactively cluster activities into tasks on real usage, we’d need
some way of deriving an online algorithm for Vlachos' constrained DPMM model.
This is probably a good opportunity for further research. In the coming weeks,
I’m planning to have a go at implementing Vlachos’s work myself, in order to
better understand the details of how it works.
At the complete other end of the spectrum of my research interests, I’ve also
been reading Corbin and Strauss’s Basics of Qualitative Research, in order to learn more about Grounded Theory, and to prepare myself for the user activity study we’re organizing. Jiyin, Emine and I have talked at some length about how we can classify computing activities at a higher level, across different users, and I’m quite keen to use a GT analysis in order to discover a high level classification that reflects users own understanding of their tasks and how they link together, and which reflects the available evidence. I’m planning to write a separate, more detailed post with my notes and thoughts from reading Corbin and Strauss soon, but in the meantime I’m particularly taken with the way that Grounded Theory appears to offer a principled generative process for developing hypotheses and theory from data, in contrast to the traditional quantitative scientific method’s discriminative process for falsifying existing hypotheses and theory. In fact, I can see how the two work very naturally together - using Grounded Theory to develop and discover hypotheses for testing, then quantitative methods to formulate tests for their veracity.
Otherwise, I’ve read Ahmed Awadallah et al’s paper on Supporting Complex Search
attended a seminar by Maria-Florina Balcan on
(which was super interesting, and I can see a very obvious application of it to
the sort of query recommendation work in Awadallah’s paper that I will write
about in more detail soon), and (in a very small way) helped Jeremy Corbyn get
nominated for Labour Leader by the Walthamstow
on Saturday (more thoughts on which are here).
A fascinating documentary on
Harold Wilson’s two tenures as Prime Minister, and the extraordinary attempts
to remove him from power.
Some vague but tantalising
about the thnking behind Spotify’s ‘Discover Playlist’ feature, which I’ve
been enjoying this week. Matt Ogle’s comments on the use of algorithms that
build upon and complement human behaviour are very reminiscent of Abigail
Sellen’s ideas around ‘synergy’ in intelligent systems.
- Tim Harford on the Alchemist’s Fallacy and the spread of ideas
- An excellent Jacobin essay on the politics of the working day, which formed a nice complement to Sian Lindley’s paper on the relationships between time and technology.
- An example implementation of a Gibbs sampler in functional Scala code
- A short, but very important blog post from Paul Krugman, on Uber’s two
innovations - which reminds us both that technology is not somehow beyond ideology, but that we can also embrace technological innovation in a critical way, unpacking it from the reactionary politics that all too often are tied up with it.
- Pacheco Pereira on the European crisis and the ‘end of history’
- A spellbinding panel discussion hosted by Influx Press featuring Gareth E.Rees, Gary Budden, Tina Richardson and David Southwell talking about storytelling, (psycho-)geography, walking, and more.
- An interesting look at The Guardian’s ‘Creative sprint’ process
- A thought-provoking essay on Markfield park, Tottenham, and the
gentrification of London by Gary Budden
- Fascinating piece on the psychological effects of modern capitalist
and how they led to the rise of identity politics.
- Interesting piece on the millenial generation, and emergent forms of left-wing politics
Music: Algiers, One
track off the new Darkstar album, Wu-Tang Clan, Loads of great stuff on my Spotify discover playlist, In Aeternam Vale, Various stuff on Blue Tapes.
26 Jul 2015
I’m absolutely thrilled to bits that Walthamstow CLP voted to give our
supporting nomination for the Labour leadership to Jeremy
on Saturday. Due to some procedural irregularities, not everyone who wanted to
speak in support of a candidate had a chance to do so. If I had done, here is
what I would have said:
Good morning everyone, my name’s Tim Cowlishaw, and I’ve been a member of the
Labour party for a little under three months. While I don’t have a history of
membership and campaigning that’s as long or venerable as those of of you who
have already spoken, I suspect that the perspective of a new member might be
a valuable contribution to this discussion.
As will come as no surprise to those of you I spoke to or thrust a leaflet at
outside, I’m supporting Jeremy Corbyn as the next leader of the Labour party,
and I am doing so in a spirit of both principle, and pragmatism.
Jeremy is the right candidate for the leadership from a point of view of
principle, as so far, he is the only one of the four who has articulated
a vision of what the Labour party is actually for. Rather than simply making
a case for how he will win a general election, he’s powerfully and plainly
outlined why he would want to do so, and what he would do in power. He is
the only candidate who is making the very straightforward point that the Labour
party exists to represent the interests of the working people of this country,
and who also has a concrete plan of action to represent those interests.
Now, many will argue that this sort of idealism is all very well at a CLP
meeting, but that it doesn’t win general elections. I don’t agree, and further,
I’d argue that this is not necessarily even a useful concern at this point in
the history of the party. Therefore, I think that as well as being
a principled choice of leader, Jeremy is also a pragmatic one. With five
years until the next general election, what the party needs in a leader is
someone with the courage of their convictions, and the ability to make a case
for these convictions, providing a visible and viable opposition to the
Conservatives' economic illiteracy and social barbarism. I’d argue that doing so
will bring dividends at the ballot box come 2020, but more importantly,
I believe that our role as the party of working people means that this is even
more important for its own sake. Further, to those who do believe our
electability is an imminent and overriding concern - I would simply point to the
massive outpouring of support for Jeremy from all sections of society - from
Labour members of many years who stayed silent as we tacked towards the centre
ground, and from people like me who are new to organized political activism,
especially those younger than me, who, for perhaps the first time, have seen
a politician at Westminster who cares about their interests. I’d argue that the
popularity of Jeremy’s campaign is a better indicator of his popularity in the
country at large than the reckons of those party functionaries at headquarters who
for some reason believe that they’ve a miraculous gift to predict the future.
We live in a country with a GDP per head of twenty-five thousand pounds a year,
but a quarter of us live in poverty. We’ve a proud tradition of representative
democracy, but thirty-five percent of the electorate felt so disenfranchised by Westminster
politics that they didn’t even turn up to vote last May. I believe we should
see both of these facts as nothing less than a dereliction of our duty as
a party (and as Labour activists, councillors and MPs) to stand up for the
rights of the working people of this country. I believe that of all the
candidates on the ballot, Jeremy Corbyn is the only one with the vision, and the
track record as a campaigner, to put that right, and put it right we must, both
because it is right in and of itself, and because once we start representing the
working people of this country again, they’ll start voting for us again. Thank you.
22 Jul 2015
For the last two weeks, I was on holiday in France!
I did find the time to read Don Norman’s “The Design of Everyday Things” while
I was away though, which was a really useful introduction to the psychology of
interaction, and refresher of some basics of interaction design, and the
human-centred design process. At times I felt that he places a little too much
faith in the power of the design process to anticipate future needs and problems
(Complex systems produce emergent, unpredictable properties and effects
- something that isn’t mentioned), but in general it’s an excellent work.
There’s only been two days since I got back, so these notes will be brief.
I’ve continued to work on my desktop activity logger -
designing and building a user interface around the logging software I’ve
already developed, which has been a bit of a crash course in Windows GUI
programming. Not really understanding the trade-offs, I fairly arbitrarily
decided to use
of Windows Forms, then subsequently realised this made it more difficult to
create a system tray icon, since this is a Windows Forms component. happily, the
WPF NotifyIcon library provides
a solution, and after finally getting it to load (F# does not automatically load
libraries referenced in XAML files), and working out how to add an icon
image to a Visual Studio project to, I’m now getting along nicely and hope to have the logger software finished this week.
Jiyin has started writing a server application to store the data for our study,
and I’ve been working on deploying it to a server for testing. We’ve also been prototyping
the user interface for our study participants to annotate their activity logs,
and will begin implementing that next week. We’ve also designed the
pre-questionnaire for the study and submitted it for ethical approval.
Interesting links: (A bumper crop since I’ve had plenty of time to read!)
Music: Intergalactic FM - a non stop mix of incredible disco and italo, Richard Skelton (and his ‘The Inward Circles’ alter ego) and Chrononautz.
04 Jul 2015
(A bumper lot this week, as I’m off on holiday this morning and have been rather
busy getting things finished beforehand! From now on, I’m going to write my
weeknotes from Wednesday to Wednesday, in order to have them coincide with my
weekly supervisory meetings, since this is when I typically need to take stock
of my progress and plan the next week.)
This week I’ve spent a load of time working on my task tracker. I’m using the
Windows accessibility API in order to listen for and log UI interactions
- however, I was having real trouble working out how to get information about
the process that fired the event. With some help from Filip (and with reference
to the code that Emine’s Masters student from last year wrote for her own
study), I’ve finally managed to crack it, learning a hell of a lot about
PInvoke, and calling C APIs from F# in the process. I’m now working on building
a simple user interface for the logger, and am going to then leave it running in
the background for a few days (while using my windows VM for everyday tasks) and
look at the distribution of events captured in order to work out where I should
focus my efforts next. I’m also going to look into writing a version for OSX too
- since that’s what I use day to day, it’ll make it a lot easier to evaluate on
my own usage.
I’ve also reviewed the literature on exploratory search - this was prompted by
reading the findings in one of Manisha and Emine’s
publications which suggested that
the technique they were proposing performed better on exploratory queries as
opposed to focused ones. I’d been thinking about this distinction for a while
(especially with reference to Siân Lindley et al’s classification of daily
was very glad to find a way into the existing literature on the topic. White
and Roth’s book on the
has proved to be really useful, and has tied together several disparate strands
of my current research. In particular, there’s two arguments that they make that
are especially relevant to my research: Firstly, that systems which support
exploratory search behaviours should treat search behaviours as a long-term,
iterative and synergistic process (which chimes very nicely with my main
research focus - the use of holistic, long-term activity logs for information
retrieval). Secondly, White and Roth make reference to the trade-off between
Recall and Precision, and identify the fact that exploratory search requires
a higher level of recall compared to typical focused search tasks. I’ve been
thinking for a while about how active learning techniques could be incorporated
into search and retrieval systems, and one of the main challenges I’ve come
across is reconciling the user’s need to be shown the most-likely relevant
results, while the active learning system requires the user to judge the
most-uncertain results - this is roughly analogous to the explore-exploit
trade-off of reinforcement learning. White and Roth’s assertion that recall
matters more than precision to users engaged in exploratory search activities is
particularly interesting in light of this, as it suggests a situation in which
the user’s interests and the learning systems interest would be more closely
aligned - since the user privileges recall at the expense of precision, the
system can elicit judgements on uncertain examples without loss of user
satisfaction. This process could even be seen as a form of computer-aided
serendipity in which the user and system jointly explore the frontier of
In other Active-Learning news, I also reviewed a couple of papers on active
semi-supervised clustering which elicit pairwise must-link and cannot-link
constraints from users in order to arrive at an optimal clustering. I can see
a couple of challenges for my use case - in particular, my particular
application would really benefit from an online learning algorithm - since
a users activity log is longitudinal, with new activities (and new judgements)
arriving sequentially, it makes sense to use an online algorithm, rather than
re-training from scratch periodically. In addition, we can reasonably expect the
number of clusters to vary dynamically over time - while both approaches I read
about allow for the number of clusters to be inferred, the time-varying nature
of our particular problem is a little more subtle. Intuitively, the problem of
assigning interactions or resources to ongoing activities sounds a lot like
a Chinese Restaurant Process, so I’m planning on looking into the literature on Dirichlet process mixture models, in order to see if there’s anything useful there. In particular, Andreas Vlachos (a member of Sebastian’s lab who are also based in our office) has done some work on active learning for DPMMs, which might well be relevant here - I will look into this in the coming weeks.
Jiyin and I continued preparing for our field study - making amendments to our
ethics committee application, and working out how we will store the activity log
data we collect for further analysis. We’ve devised a simple schema for log
messages which incorporates all the data produced by both our loggers, and are
now working on a system to persist it. Given the log data is just a stream of
JSON objects, delivered over HTTP, I suspect Logstash (with the HTTP input) might well do what we need without us actually having to do any development work, which would be nice. I’ll be looking into this when I’m back from holiday.
As part of the process of completing our ethics application, we also had to go
back over our proposed methodology, which highlighted a couple of shortcomings.
In particular, in soliciting task labels from users, we’ve opted to allow users
to describe their tasks with free text labels, in order to reflect our lack of
prior knowledge about the participants' activities. However, this has one important
drawback, in that it does not allow us to make comparisons between
participants, given that there’s no common set of descriptors for tasks. In
order to address this, we could come up with a higher-level set of ‘task
categories’ shared between users, to which they can assign each task - however,
the issue then is deciding upon a principled set of categories which provides
adequate coverage. Another approach (which I prefer), would be to elicit
commonalities between participants as part of the analysis process, or in
interviews after the study. Grounded Theory offers a useful methodology for this, and the Affinity diagram would be a useful technique to elicit task categorisations in the post-study interview. I’m going to read up more on this and formulate a more concrete methodology next week.
We had the first meeting of our office Machine Learning interest group on the
2nd, including staff and students from UCL, and BBC R&D. It was great to find
out more about others interests and activities, and to explore useful ways
of supporting each other. We decided to get started by agreeing on a paper to
read and discuss in a ‘journal club’ format, and to reconvene in a month. We’re
still deciding on our first paper, but the general consensus was that we’d like
to go back to some of the original publications on deep learning, and develop an
understanding of it from the ground up.
Music: Petrels, Tim
Hecker, Vessels, Holly Herndon, Lorenzon Senni, Alessandro Cortini, Fennesz, Some incredible South African disco, Laurel Halo, Julius Eastman, Arthur Russell, Sparks, Pet Shop Boys, East India Youth and Shit and Shine.