Interaction Design
for Novel Applications
Hyowon Lee (home)
This page shows a selection of my interaction designs over the years, in a reverse-chronnological order. I design user experience and user interaction that show-case a variety of novel scenarios and applications that exploit emerging computational and interactive technologies, and many of them have resulted in full-fledged applications and user-evaluated, some of them in a long-term usage environment. While the high-level concepts, understanding of the technological possibilities and technical realisations come from the team efforts, the detailed interaction designs (creating UI concepts, interactive strategy, layout, visual design including icons, labels, data visualisation, etc.) were done by myself. My main challenges in these design activities are (1) there are no exemplar or precedences of successful designs that I can follow, thus requiring creativity and judgement, (2) there is no existing user-base from which requirements could be established, making the initial design stage shaky, and (3) often the back-end system can be unreliable and inaccurate (as the incorporated 'engines' of the system are the computational components that are currently under research and development). Creativity, visual literacy, understanding of emerging technologies and their characteristics, sensitivity to potential usage context, and the ability to work comfortably when there is not enough specifications/information available to the problem are required in designing this type of interactions.
Learning a 2nd language is such a tough experience, especially when it is time to keep practicing the conversations with native speakers. After an hour of conversation practice, how do you know how well you did (compared to previous practice sessions) and how you could improve? Working with my colleagues here in Insight Centre and with pedagogy experts in SALIS, the online platform we developed together has the UI that I designed and refined over time to visualise the conversation metrics so that the students can review and reflect. This system has been deployed to 13 participating universities in Europe and actively used by over 1,370 students to reflect on their 2nd language learning since 2021.
Dey-Plissonneau A, Gomez Soler I, Lee H, Liu M, Scriney M and Smeaton A. Supporting second language learner reflection in teletandem videoconferencing through the visualisation of conversation metrics. Journal of Virtual Exchange, 6(SI-TransferableSkills), 12-25, 2023.
The state-of-the-art UIs for multi-touch and voice-operated devices still bring a lot of legacy features from the desktop UI era. One of the common fitfalls of these is the ‘click/touch-then-show-new-info’ (the fundamental idea of hyperlinking), requiring the user to re-orient and to face a new layout/page of information for each click/touch. Using this observation as starting point, I designed a UI in which there is no sudden appearing/disappearing of information at all – everything is contained in one screen and works by dragging up some vertical cards: no widgets, no buttons, no sliders/scroll bars, no tiny icons/lables, no full-screen mode... removed legacy features (almost) completely. The result is a UI with a 'leaflet rack' metaphor that applies strong constraints on what can happen on screen, rather limited, but priority was on learnability and ease of use, not efficiency.
Inspiration came from how a leaflet rack is inviting and easy to use
Multitouch devices are great in enhancing the overall usage barrier for beginners, but a typical usability issue with them is "fat-finger problem" or finger occlusion. In order to check how some of the basic finger occlusion-avoiding techniques have usability consequences in the subsequent user actions after the targeting/positioning action, my team designed a full-fledged tablet sketch app, every feature addressing finger occlusion issue. In designing the UI, I took particular care on the widgets on the screen - chunky, large buttons are all on the sides (near the edges of the screen) and all featuring immediate feedback to give hints on what is happening. In order to reduce the sense of visual clutter (prone when widgets need to be large), I separated two layers of widgets: (1) main, most important features are to have physical, real button-like qualities, and (2) secondary, more trancient types of widgets that are semi-transparent and blend as part of the textured canvas. The result is a visually simple but interesting shades and harder items overlaid. The app went through a usability testing with 25 participants, from which observations and insights have been collected and will be reflected in the next version.
See a 1.5-min video showcasing the UI: Sketch app with finger occlusion-free features (YouTube, 2m 50s)
Lee H, Ko I, Tan Y.R, Zhang D and Li C. Usability Impact of Occlusion-free Techniques on Commonly-used Multitouch Actions. 5th International Conference on ACM In Cooperation HCI and UX Conference (CHIuXiD 2019), Bali, Indonesia, 1-9 April 2019.
User-maintained interaction works by the user tapping a button but effective only while she keeps pressing it down. Upon releasing, the effect of the function disappears and gets back to the original state/mode. Such an interaction, while quite ubiquitous in physical world (e.g. laser pointer and piano pedal), is very seldom featured in digital world. We identify the kinds of situations such user-maintained interaction can be more useful than the normal 'persist' buttons, and design a few prototype tablet apps that incorporate the feature to test its usability.
See a 1.5-min video showcasing the UI: UMM interaction for apps (YouTube, 1m 30s)
Fennedy K, Lee H, Ko I, Tan Y.R, Zhang D and Li C. Augmenting User-Maintained Interaction Through Mode Locking and Reversing. 32nd British Computer Society Human Computer Interaction Conference, Belfast, Northern Ireland, 2-6 July 2018.
While tablet interaction did significantly enhance the overall usability of computing, there are still categories of people who have difficulty using multi-touch applications, partly due to the finger dexterity required and partly due to overwhelming or overlly-subtle visual feedback on the screen interface. Making the touch interaction that typically considered easy-to-use even easier and simpler, our project aimed at designing a simple game for the people with dementia and the healthy elderly. The resultant interactive game is a reminiscence-inspiring, culutrally-relevant puzzle game tailored for senior people in Singapore, and is the result of extensive discussions with relevant stakeholders including geriatric care unit doctors, nurses and therapists and a series of playtests throught the development amounting to over 400 participants.
Westphal B, Lee H, Cheung N, Teo C and Leong W. Experience of Designing and Deploying a Tablet Game for People with Dementia. 29th Australian Conference on Human-Computer Interaction (OzCHI 2017), Brisbane, Australia, 28 Nov - 1 Dec 2017.
This work envisages what our future application for video consumption would be like, based on a series of interviews with researchers from various technology fields. A novel video browsing interaction was designed as a result, and illustrates how far more sophisticated and fine-grained levels of crowdsourcing on the web and a combination of emerging computational technologies will allow searching and browsing in an online video repository such as YouTube, but with the meaningfully segmented chunks of videos as the main unit of retrieval, resulting in easy concatenation of chunks to create longer videos and share.
To project what the UI for such a system would be like, I assumed that the widgets and panels on screen are not rigid but semi-floating (to more naturally react to finger/hand touch), and also assumed the device itself will be a thin, transparent, AR-enabled device.
See a 5-min video summarising the concept and the UI: EduBang Video Sharing (YouTube, 5 minutes)
Lee H, Poon KW and Balasubramanian G. EduBang: Envisioning the Next Generation Video Sharing. Video Showcase at: ACM SIGCHI International Conference on Human Factors in Computing Systems (CHI 2017) Denver, CO, USA, 6-11 May 2017.
Re-thinking the way we interact with our everyday IT devices and applications today, I explored hand squeezing as a possible alternative way in which a user could engage in as the primary interaction modality. In doing so, I constructed a conceptual design space to orient myself and systematically brainstorm design possibilities for squeezable interaction. I selected various points in the design space to identify novel usage scenarios and application ideas that exploit squeezing interaction. The shaping of the design space was guided by physically instrumenting sensor-embedded squeezable objects and applications and by undertaking evaluation with 20 test users one by one. By cheaply and nimbly testing out and modifying our prototypes through a series of small user testing, I demonstrated how future interaction modalities could be explored in a systematic, pragmatic and designerly way that is cognizant of existing literature yet offers ample room for innovation.
The Squeeze Game works by the player holding a sponge ball - squeeze harder and the on-screen ball goes up, unsqueeze and the ball drops: the goal is to navigate the maze without hitting the blocks (screen shot on the left). I designed all aspects of the game: overall idea, interaction, game mechanics, graphics, the path overview, etc.
Lee H, Yoong A, Lui S, Vaniyar A and Balasubramanian G. Design Exploration for the "Squeezable" Interaction. 28th Australian Conference on Human-Computer Interaction (OzCHI 2016), Tasmania, Australia, 29 November - 2 December 2016.
3D modeling task is a complex challenging task in any interactive platforms. In this tablet-based UI, a viewpoint change mode was designed in so that in that mode the user can comfortably but temporarily focus on shifting the camera angle and also shift back to history. In the screen shot beside, a skeuomorphic version of my design is shown.
Rod J, Li C, Zhang D and Lee H. Designing a 3D Modelling Tool for Novice Users. 28th Australian Conference on Human-Computer Interaction (OzCHI 2016), Tasmania, Australia, 29 November – 2 December, 2016.
More and more scientific evidences are appearing that show how heavily relying on our increasingly more powerful and increasingly smarter applications and systems can undermine our cognitive capabilities (memory, arithmetic/spatial ability, vigilance, decision-making, etc.) when used over time. Instead of just saying "try not to use it too much if possible" as if our technology is some kind of necessary evil, or even falling back on technological pessimism, we should start designing our apps taking this issue into consideration from the start. Envisioning how our day-to-day apps might look different when we re-design based on this awareness, a new set of UIs were sketched in a team effort (see co-authors in the reference below, who all participated in the UI brainstorming and sketching). Many of them take on some simple gamification elements to keep our cognitive functions active while not compromising the accuracy/outcomes/productivity too much. A number of generic issues were identified in the UI sketching exercises and from these (initial set of) high-level design principles were compiled.
Balasubramanian G, Lee H, Poon KW, Lim WK and Yong WK. Towards Establishing Design Principles for Balancing Usability and Maintaining Cognitive Abilities. Book Chapter in: Design, User Experience and Usability: Theory, Methodology and Management. LNCS Vol. 10288 (19th International Conference on Human-Computer Interaction, Vancouver, Canada, 9-14 July 2017), 2017.
Reviewing all major texting apps available today (WhatsApp, WeChat, iPhone Messages, KakaoTalk, etc.) show that they are basically all same in terms of how functionalities are offered and more critically what functionalities are offered. I re-designed a typical texting app by incorporating a simple visualisation of the texting threading and its history. A user can see the miniature threading of texting on the left side of the main texting UI (see left), and also slide right to see these miniature threading visualisation from past sessions (see right). Such design does not require any rocket science to implement/program, yet enhances the visibility of the threads and enhances the access to past threads of chats.
Using 20,000 lifelog photos, I miniaturised each photo to small thumbnail-size photo, and projected on a large, high-resolution, public display wall that has multi-touch capability. In the figure, two of my students were demonstrating interactively investigating the photos while at the same time seeing a particular pattern (the yellow stream in the middle) which characterises the lifestyle of the photo collection owner.
See a short video showcasing the UI: Visualising the photos on the Wall (YouTube, 1 minute)
Kelly P, Kumar A, Doherty A, Lee H, Smeaton A.F, Gurrin C and O'Connor N. The Colour of Life: Interacting with SenseCam Images on Large Multi-Touch Display Walls. SenseCam Symposium 2010, Dublin, Ireland, 16-17 September 2010.
Video summarisation has been one of Multimedia's long-term research topics. While automatically extracting snippets of "important" of a long video is certainly an appealing idea, but when the original video is not a clean, professionally editted video (such as a movie or TV series) but an unstructured, repetitive or boring(?) video (such as a home movie), then the automatic summarisation doesn't work very well. Working with my colleague Saman who's been developing sophisticated video summarisation that are tailored for unstructured/uneditted video contents, we came up with this application that starts with showing automatic summarisation result and then easily customise which parts of video to be included/excluded. The key design feature is a clear indication of the source and target duration which changes as the user drags parameter slider bars, and a hierarchical timeline that can help the user quickly see the inclusion/exclusion in the video on overall as well as detailed view.
A User-Centric System for Home Movie Summarisation. Cooray S, Lee H and O'Connor N. MMM 2011 - The 17th International Conference on Multimedia Modeling, Taipei, Taiwan, 5-7 January 2011.
Large, public multi-touch display is still not commonly sighted, but I envisage that such devices will become more widely seen in urban settings in near future: at a bus stop, in a shopping mall, airports, and in the middle of city square. For different purposes there will be different ways to provide information and menus, but what would be a reasonably generic menu system that could be used for such public multi-touch walls across application areas? Consider a typical menu style for desktop PC applications... library management system, word processor and web browser - even though their application areas are very different, a common style menu system can be used just because of the characteristics of the interaction device (keyboard and mouse, a monitor and the fact that the user is most likely sitting in front of a desk). I designed a generic menu system (seen on the left) that recognises the characteristics of public multi-touch wall device, ended up quite different from conventional menu widget as we know: two different groups of menues - Global menu with shared and awareness features, and Local menu with transient and temporary nature - it may be a long shot but I believe it is a good starting point!
See a short video showcasing the UI: Generic menu system for multitouch wall (YouTube, 1m 7s)
Designing Interaction for a Multi-touch Wall. Wang C, Lee H and Smeaton A.F. iHCI 2010 - Irish Human-Computer Interaction Conference, Dublin, Ireland, 2-3 September 2010.
CLARITY Centre I work in has been doing some great work in using various sensors deployed in houses, collecting the energy usage data around Ireland. Data includes 16 home users' home energy usage for 1 year and gets collected 24/7 and the live stream of data are managed centrally. I designed a web portal where this stream of data can be nicely visualised and monitored. I love working for such a large-scale, technologically-innovative research centre so that I can design for such novel applications that haven't been attempted by anybody else before. For this particular design, I used a slanted panel that closes and opens as the user selects one of the 3 types of the collected data (the left panel on the screen shot beside).
Green Multimedia: Informing People of their Carbon Footprint through Two Simple Sensors. Doherty A, Qiu Z, Foley C, Lee H, Gurrin C and Smeaton A.F. ACM Multimedia 2010 - ACM International Conference on Multimedia, Firenze, Italy, 25-29 October 2010.
Visualisation of blogosphere... this has been usually a word cloud-type of visualisation, and time-graphs that show sentiment changes over time. I wanted to do something different - how about making the visualisation style as if it is a web search-like interface where a user searches, queries, browses the results, refines the query, etc.? I designed a novel 'visual unit representation' of companies ('company' being the recepient of opinions and sentiment), and then stacking them as search result. As a result, this became a highly interactive, familiar web-search-like visualisation. Graphs that show trends over time are a part of the interface, a panel that slides up and down when the user needs.
Integrating Interactivity into Visualising Sentiment Analysis of Blogs. Lee H, Ferguson P, O'Hare N, Gurrin C and Smeaton A.F. IVITA 2010 - 1st International Workshop on Intelligent Visual Interfaces for Text Analysis, at the ACM International Conference on Intelligent User Interfaces (IUI) 2010, Hong Kong, China, 7 February 2010.
Designing for In-Home Display (IHD) is yet another example of a whole different (and new) set of design implications arising from the interaction characteristics for its usage. These include (1) always-on, (2) essential information on the initial screen (no 'main menu' screen), (3) dark background as to not blind the home user at night or brighten the room (think of a night-friendly bedside clock radio with its black background and bright digits), (4) providing context information along with the data (e.g. history, average, neighbour's data, etc.) to help make sense of the reading, etc. After identifying these issues, I designed a home power monitoring interface (see photo on the left) that uses simple finger touch as its main input and visualising a rich set of data reading in a easy, interesting and informative way.
Green Multimedia: Informing People of their Carbon Footprint through Two Simple Sensors. Doherty A, Qiu Z, Foley C, Lee H, Gurrin C and Smeaton A.F. ACM Multimedia 2010 - ACM International Conference on Multimedia, Firenze, Italy, 25-29 October 2010.
Interactive TV brings in a very distinctive interaction characteristics, namely (1) lean-back, (2) use of remote control, and (3) multiple levels of viewer attention. I studied these characteristics and turned them into a set of design strategies (for these characteristics provide a number of design implications), and designed a complete iTV interface that balances the powerful functionalities in Multimedia technology with extremely simple and streamlined viewer-TV interaction with a conventional remote control (see photo on the left). Literature review, compiling the available guidelines, fleshing out the design options, coming up with my solutions and sketching - the process took me full-time 4 months of work, but the result was definitely worth-while. In my opinion, the progress in the iTV community is slow, especially in terms of interaction design - no agreed body of design guidelines or knowledge, no standardised widget sets for TV, nobody knows what is the best way for iTV interface. During creating my own solution, I ended up helping adding more interaction design knowledge to the community - check my paper below.
Balancing the Power of Multimedia Information
Retrieval and Usability in Designing Interactive TV. Lee H, Ferguson P, Gurrin C, Smeaton A.F, O'Connor N and Park H. uxTV 2008 - International Conference on Designing Interactive
User Experiences for TV and Video, Mountain View, CA, 22-24 October 2008.
A car user installs an in-car video recorder that records the scene in front of the car along with GPS device. Once a road trip is done with the car, GPS data and video data are put together to provide a map-based browsing and searching interface. I came up with the idea of map browsing as the base interface where basically the whole UI is the map, and upon request relevant panels slide in and out on top of the map. A 'trip' is represented as a single colour line on the map, which threads a set of 'points' on the line. Selecting a point then slides in a playback panel and shows video captured at that point onwards. A snapshot on the left.
A Content-based Retrieval System for UAV-like Video and Associated Metadata. O'Connor N, Duffy T, Ferguson P,
Gurrin C, Lee H, Sadlier D, Smeaton A.F and Zhang K. SPIE Defence and Security Conference 2008, Orlando, FL, 16-20 March 2008.
A museum visitor taking a number of photos of the exhibited artefacts comes back home, and gets confused which photo is which artefact, cannot remember what artefact had which history or what title, etc. She uploads all these photos to the system, and it automatically identifies same artefacts from many photos, categorises them, and links them to the museum website's authentic photos and information. The user can manually correct if any categorisation was done incorrectly, by easily dragging items around to different groupings - the dynamic Flash interface doesn't flash; only to enhance the usability in a quiet way. Front-end interface designed by me (snapshop on the left), back-end engine researched and developed by Michael, Flash interface implemented by Sorin.
Mo Músaem Fíorúil: A Web-based Search and Information Service for Museum Visitors.
Blighe M, Sav S, Lee H and O'Connor N. ICIAR 2008 - International Conference on Image Analysis and
Recognition, Povoa de Varzim, Portugal, 25-27 June 2008.
A soccer player wired with BodyMedia device that captures a number of body response data along with a GPS device and two video streams capturing his movements gets his data all recorded and analysed at the end of the game. Data is analysed, synchronised and presented to an interface that displays sensed data beside video and soccer field location of the player all in one synchronised way. Juxtaposition of different sources of data with different characteristics (but connected by time dimension) can reveal many interesting and potentially useful facts - the main benefit of any visualisation. Snapshot on the left.
Aggregating Multiple Body Sensors for Analysis in Sports. Smeaton A.F, Diamond D, Kelly P,
Moran K, Lau K, Morris D, Moyna N, O'Connor N and Zhang K. pHealth 2008 - International Workshop on
Wearable, Micro and Nano Technologies for the Personalised Health, Valencia, Spain, 21-23 June 2008.
Hundreds of holiday photos... uploading them all to Flickr is fine but what about the time-consuming manual annotation and captioning on each photo? I designed a web service interface that leverages our group's automatic face detection and recognition technique, whereby the interface starts with showing 'my friends' faces' list (automatically generating by detecting and recognising faces in photos, then cropping into small 'face icons' - rather than a set of photos, and presents photos organised by face icons... thus the interaction style is quite different from conventional photo management services - in this design I tried to exploit the face recognition tool to the maximum. See snapshop on the left, this design took me about 3 months full-time.
User-Feedback on a Feature-Rich Photo Organiser. Sadlier D, Lee H, Gurrin C, Smeaton A.F and O'Connor N. WIAMIS 2008 - 9th International Workshop on Image Analysis for Multimedia Interactive Services, Klagenfurt, Austria, 7-9 May 2008.
SenseCam is a wearable digital camera with sensors wired in, automatically triggering photo capture when something happens in the surrounding. In a usual day, wearing a SenseCam will result in 1,500 - 3,000 photos, in effect visually archiving a person's day. Once there are this many photos, organising, annotating and retrieving becomes a headache. In CDVP we develop automatic organisation tools to group the photos by individual events of the day, to determine repeating or unique patterns of the events, and to establish similarities among those events. I designed this web-based interface, called My Visual Diary, that presents the photos as an interactive comic-book style interface, as a result of the automatic analysis of the photos. In technical Multimedia circle, mapping the 'importance' with a size of an image is something that not many people can come up with. See a screen shot on the left.
Constructing a SenseCam Visual Diary as a Media Process. Lee H, Smeaton A.F, O'Connor N, Jones G, Blighe M, Byrne D, Doherty A and Gurrin C. Multimedia Systems Journal, Special Issue on Canonical Processes of Media Production, Vol. 14, No. 6, pp341-349, 2008.
This Web 2.0 application is an online photoware that uses automatic photo analysis techniques to index and
organise a large amount of personal photos very easily. Photo content analysis such as face detection and
labelling, building detection, combined with context analysis such as the GPS data and shutter speed recorded on
the photo are used to automatically annotate the photos. The user can then see the automatic annotations and confirm or correct
them. Mobile interface for browsing and searching for photos on this system has been also developed and experimented. A snapshot on the left.
MediAssist: Using Content-Based Analysis and Context to Manage Personal Photo Collections.
O'Hare N, Lee H, Cooray S, Gurrin C, Jones G, Malobabic J, O'Connor N, Smeaton A.F and Uscilowski B.
CIVR 2006 - 5th International Conference on Image and Video Retrieval. Springer Lecture Notes in
Computer Science Vol. 4071/2006. Tempe, AZ, 13-15 July 2006. (pp529-532)
Moving out of the conventional mouse, keyboard, monitor for single-user video searching, Físchlár-DT is a TableTop video search interface on top of Físchlár Digital Video System. The Tabletop is based on DiamondTouch table, with DiamondSpin software toolkit. Designing a collaborative interface on which two users working together to search for video shots require consdering interesting issues such as division of the task between the users, workspace awareness, and widget/object coordination policy between the users. I explored these issues by designing, implementing and experimenting with users and assessing in terms of search performance, amount/kinds of interaction bewteen the users and their personality type matching. The interface design was done collaboratively between me, Colum and Sinéad, and demonstrated at TRECVid2005 at Gaithersburg, Maryland, in November 2005.
Collaborative Video Searching on a Tabletop. Smeaton A.F, Lee H, Foley C and McGivney S.
Multimedia Systems Journal, Springer, 2006
It is good to provide Relevance Feedback feature so that some relevant image/object in the database can be used for formulating subsequent query... but the starting point (initial query) is often not supported and relies on text-query. BBC Rushes Explorer allows the user to use any image external search engine (e.g. Google Image Search) and incorporate those images directly into the system to formulate initial query, as well as using the database images/objects (50 hours of BBC rushes footage, in participation to TRECVid2005 activity). The incorporated images can be further segmented to object by the user, and the object is then used for RF query. Using interactive object segmentation interface and object-based RF feature of this system, the overall interaction is smooth, user's RF is open to *any* image or object outside of the database. To me, this application is one of those that use most novel yet realistic (in terms of the current status of Multimedia research) scenarios... A snapshot on the left.
Interactive Experiments in Object-Based Retrieval. Sav S, Jones G, Lee H, O'Connor N and Smeaton A.F. CIVR 2006
- 5th International Conference on Image and Video Retrieval, Tempe, AZ, 13-15 July 2006.
'Scene detection' is still not fully explored area except special genre such as news stories or sports event scenes. The system automatically finds where exciting (action) scenes, dialogue, or montages scenes in films are, and helps the user quickly spot those particular scenes. These three particular scenes are a 'prescribed' set of parameters pre-defined for the users, and more generic query formulation is provided in which the user can adjust these parameters to customise the query formulation beyond action/dialogue/montage scenes. A snapshot on the left.
Searching Movies based on User Defined Semantic Events. Lehane B, O'Connor N and Lee H. SIGMAP 2006 -
International Conference on Signal Processing and Multimedia Applications, Setubal, Portugal, 7-10 August 2006.
This is advanced version of the previous interface. When the user adds more and more example objects (and their features) as query formulation, if the added examples are not very similar to each other it tends to confuse the system and results in poor retrieval result. However, the addition of any example objects is a legitimate action allowed by the interface, yet we do not want a user to add semantically very different objects... so the solution provided in this interface is the system suggests the user the possible clusters among the query objects. Separated clusters then can be individually searched, allowing a more focused searching thus the result is better. A snapshot on the left.
Using Segmented Objects in Ostensive Video Shot Retrieval. Sav S, Lee H, Smeaton A.F. and O'Connor N. AMR 2005 -
3rd International Workshop on Adaptive Multimedia Retrieval, Glasgow, U.K., 28-29 July 2005.
This system allows the user to compose a query based on objects and their low-level
attributes (colour, texture and shape). Query formulation incorporates unit representation idea in L'OEUVRE interaction design
as above, but the cumulation of user queries become a relevance feedback, to incrementally refine the automatic object
classification in the database. I put a particular attention on the feature feedback buttons, as these are the most important
query element of the system. See a snapshot on the left.
Interactive Object-based Retrieval Using Relevance Feedback. Sav S, Lee H, O'Connor N and Smeaton A.F. Acivs 2005 -
Advanced Concepts for Intelligent Vision Systems, Antwerp, Belgium, 20-23 September 2005.
Again based on objects and unit representation, this one is applied to security and surveillance system. DCU's CCTV cameras keep recording and archiving a large volume of footages, and a security staff needs to search through this archive when a theft has happened, for example. Knowing only a approximate time/date and location still takes long time to search, possibly missing all other potentially useful information that has been captured in nearby cameras around that time, all of which could have been valuable in re-constructing the event (forensic analysis). My design (see a screen shot on the left) efficiently indicates wanted people/objects in the archive, searches the same person/object found in other nearby cameras and visualises in map and highlighting their trails by time, supporting an efficient forensic analysis of an event.
User-Interface to a CCTV Video Search System. Lee H, Smeaton A.F, O'Connor N and Murphy N. ICDP 2005 - IEE
International Symposium on Imaging for Crime Detection and Prevention, London, U.K., 7-8 June 2005.
This is based on L'OEUVRE interface above, but advanced in the sense that various object classifications (people's name, object, object classes, scene/background, and actions) are also automatically done and filtering based on them. Quite futuristic in terms of automatic features.
This is my initial conceptual sketch on how the object-level interaction should be projected to the user. Here I used small, oval buttons that represent each of the detected objects in a keyframe, and thus interacting with objects is enabled by mouse-over or clicking on the button indirectly. I used this concept in my further object-based interfaces, integrated in a more realistic context for all the following designs I did for object-based interaction.
Interaction with all of the above interfaces are based on whole keyframes (or segments of a video). Sub-keyframe level is only enabled on the above TREC2004 system in which regional colour and edge can be specified by the user in querying, but they still remain low-level features. L'OEUVRE Project, which investigates object-based operations in video, brings forward the whole state of the art, and the interface design has also started moving forward accordingly. Following designs allow the users the ability to interact with objects detected in video contents: viewing what has been detected; selecting one of the detected objects in a keyframe; using a selected object for subsequent querying. Based on the bottom-level interaction with objects as a 'unit representation', overall interfaces have been designed for different contexts and possible usage.
This experimental video searching application uses video clip content-based relevance feedback... the user uses a group of video clips to query the video clip database, rather than typing in text query. In terms of interface, I gave a lot of thoughts in making all the text, images, video clips and other administrative elements on the screen as well organised as possible, aesthetically pleasing, and the saved shot list is added on the right end of the screen showing what the user has collected so far. This is probably one of the best examples in which a great amount of different kinds of information is displayed on a single screen but they are properly grouped, separated and colour-coded in such a way that it succeeds to look not too cluttered or complex. One important key was to use different background colour for 'Saved Shots' area on the left of the screen - just this simple visual trick made the interface look 30-35% less complicated. I used this trick in some of my other object-based interfaces (see below). A screen shot on the left.
Físchlár-TRECVid2004: Combined Text- and Image-Based Searching of Video Archives. O'Connor N, Lee H, Smeaton A.F, Jones G,
Cooke E, Le Borgne H and Gurrin C. ISCAS 2006 - IEEE International Symposium on Circuits and Systems, Kos, Greece, 21-24 May 2006.
An experimental system that can search through videos by indoor/outdoor, existence of face, people, audio, in-video captions (open caption), and also text transcribed by automatic speech recognition. Using a 4-colour scheme assigned to the grouping of the features and icons for each feature, I tried to combine the complicated and varied elements into an organised, coherent visualisation and interaction. The system was demonstrated at TRECVid 2002 Workshop (NIST, Gaithersburg, Maryland) and drew a lot of attention. Designing this interface resulted in a number of alternative solutions for some of the features, which I used later for other design problems.
Design, Implementation and Testing of an Interactive Video Retrieval System. Gaughan G, Smeaton A.F, Gurrin C,
Lee H and Mc Donald K. MIR 2003 - 5th ACM SIGMM International Workshop on Multimedia Information Retrieval, Berkeley,
CA, 7 November 2003.
Based on the automatic news story segmentation, the system automatically records, indexes and provides
news story-level of access to its users.
More than 6,500
news stories (2003-2004) were in the database,
all searchable, browseable and playable on a web browser. In Spring 2004, I conducted an extensive longitudinal study with
this system, monitoring 16 users who regularly used the system for 1 month while they noted down a diary.
The system also comes with a mobile interface for PDA such as iPAQ and XDA: its
primary access method is personalised story list, to reduce intensive interaction on a mobile environment.
User Evaluation of Físchlár-News: An Automatic Broadcast News Delivery System. Lee H, Smeaton A.F, O'Connor N and
Smyth B. ACM TOIS - ACM Transactions on Information Systems, 2006.
This is a variation of the above system, but contains a fixed number of nursing-related materials for teaching and learning in the School of Nursing. The overview of a programme is manually generated to provide a good Table of Contents of each video. Interaction stages follow exactly same as Físchlár-TV.
Físchlár-Nursing: Using Digital Video Libraries to Teach Processes to Nursing Students. Gurrin C, Browne P,
Smeaton A.F, Lee H, Mc Donald K and MacNeela P. WBE 2004 - IASTED International Conference on Web-Based Education,
Innsbruck, Austria, 16-18 February 2004.
The very first Físchlár system I designed its user-interface for was deployed to our university campus for over a 4-year trial period, which allowed its users to request recording of any TV programmes from 8 channels and once recorded, processed and indexed to allow browsing and playing on a web browser. The system enjoyed more than 2,000 registered users during its trial period. In designing this interface, I started with browsing vs. recording features as the two main features, and took up the look-and-feel of a black VCR box as a metaphor. As it was being used by a large number of real users, I had to be careful in upgrading or plugging-in any new features, a screen shot on the left. The interface features various within-video browsers, which was a subject of my PhD thesis.
This was long before Web 2.0... I used complex frame structure (not recommended for today's Web scene) and JavaScript to make the web interface more interactive and look more proper application-like.
Designing the User Interface for the Físchlár Digital Video Library. Lee H and Smeaton A.F. Journal of
Digital Information, 2(4), 2002 (JoDI online).