Atanas Bozhkov Human-Computer Interaction 26 October 2013

of an Internet Enabled Fridge Introduction

Implementation

The idea presented in the previous assignment was that of a smart internet-enabled refrigerator. After discussing the ideas behind the IoT and the smart refrigerator, I shall now present a plan for the implementation of such a device.

Hardware Implementation
The hardware of the fridge can be divided into several categories. The categories discussed here will only be the ones that are being added to the hardware to a normal household refrigerator. In other words - we shall omit the basic refrigerator hardware - only touching on the smart aspects of it.

Fig.1: Main Hardware Components

The categories can be broken up as follows:! 1. Main Computer Controller 2. Sensors 3. Actuators

HCI COURSEWORK 2

!1

Main Computer Controller

In order for the refrigerator to gather, process and transmit the data, it needs a computational core to carry out those tasks. The main computer controller can be a computer machine running an operating system based on Linux. A perfect choice for this task are little embedded computers similar to the Raspberry Pi. Specications:

The raspberry Pi has several strong advantages that make it perfect for the task. Size: 8.5cm x 5.4cm x 1.7cm. This is perfect since the low prole allows for easy integration with any type of fridge layout. It also allows for connecting several units in parallel in case more computational performance is needed. Power Consumption: Because of the Raspberry Pi is built on top of the ARM architecture, it only draws 2.14Watts of power (assuming 90% load) with a display connected. This makes it really efcient in terms of electricity and ultimately has no impact on the overall power consumption of the whole refrigerator unit. GPIO: The unit also has General Purpose Input/Output connectors which can be used to connect to the refrigerator control board. This will allow the unit to control various aspects of the refrigerators behaviour such as temperature, power consumption, lighting as well as the weight scales built into the shelves. Audio/Video Ports: These will allow for connecting to the touch screen panel placed on the front door of the fridge as well as the loudspeakers that are going to be used for voice feedback. USB Port: The presence of a dedicated USB port makes connecting the cameras on the shelves a breeze since no additional boards and components are going to be required. (With

HCI COURSEWORK 2

!2

the exceptions of a USB hub, that is going to be used for connecting multiple cameras to a single port). It will also allow for the addition of a WiFi card that is going to be used to connect the fridge to the internet.

It is also worth noting that the Raspberry Pi itself is not a powerful computing platform, which ultimately means that computationally heavy tasks such like image and voice recognition will take a long amount of time. This is why the best approach will be to have the unit send the gather data to a dedicated server on the internet that will process the data quickly and return it as qualitative information back. It will be the Raspberry Pis task to control to ow and representation of that data via audio, video and messaging.

Sensors
In order for the refrigerator to gather and provide information about the environment it needs dedicated sensors from which to extract and analyse data. There are going to be two main sensors - cameras (for recognising items inside the fridge), scales (for weighting of different and microphones (for voice assistance).

Cameras: The cameras are placed on each shelf of the refrigerator. There will be four cameras per shelf (one on the left, right, back sides as well as one on the top shelf facing down). The cameras will have a wide angle of view allowing them to capture an image of the whole shelf. They will not be directly intractable with. Instead they will be triggered on each door close so that the changes between the last snapshot can be detected via the processing server and then transformed into metadata.

Wide Angle Cameras

!
HCI COURSEWORK 2

Computer Generated Image of camera placements

!3

Weight Detectors: Weight detectors will be placed under each shelf. This will allow the controller to gather data after each added item. It can also monitor the remaining quantities of item in boxes using weighting (e.g. a box of orange juice is placed, the optical recognition qualies it as a 1litre carton, but the scales only detect 150grams of added weight, thus the clear conclusion is that the juice is almost empty).

Shelf

Scales placed underneath each corner of the shelf Microphones: The microphones will be used as assistive devices (e.g: for conrmation of questions, or for providing feedback for visually impaired people) There will be two microphones one placed on the front of the refrigerator door, and one placed on the inner side of it. This will provide constancy with regards to quality when the refrigerator door is open or closed. The microphones will be connected to the main controller unit, which will gather the data and use an online based service for voice recognition. The results can then be parsed by the main controller, after which a decision for action can be taken.

Front Door Touchscreen: The large spacious front door of the refrigerator can easily house a relatively small touchscreen (11inch diagonal) that will present useful information about current stock, power consumption, maintenance information, while also allowing for easy interaction with the users via a familiar interface. The touchscreen will be connected to the Raspberry Pi, which ships with controlling software for touch-screens.

HCI COURSEWORK 2

!4

!
Software Implementation
The software implementation for this device can be done on two levels - a client level (the refrigerator unit itself) and server level (a set of computational services that aid the unit in its actions). Client Level: The client level software is comprised of:
Client Level Software Description

Interface Refrigerator Control Component Control

Includes visual and audio interactions, connections to online shopping etc. Control of the main refrigerator functions Control for cameras, microphones, loudspeakers and scales Includes the protocol for communicating to the server, social networks and other online services.

Communication Protocol

Interface: The interface can easily be built on top of the Linux based operating system residing on the main controller module. It will be design so that restricts the user from any interactions other than the ones concerning the interface (in other words the operating system will be hidden). Since touch screens are becoming common, the interface can use existing guidelines for designing touch interfaces (e.g. Apples Mobile Interface Guidelines, Android Mobile Interface Guidelines). Using these best practices the product will have a familiar look and feel to it, that will naturally appeal to users.

Refrigerator Control: The controls for the main refrigerator functions can be regulated on the basis of what products are currently residing inside. In order to control these functions the one has to implement a low-level control software for sending electrical signals through the GPIO pins of the main controller to the logic board of the refrigerator. These aspects can be achieved by using a programming language like C or Python which are bundled with libraries for carrying out such tasks.

Component Control: The component control module can be implemented by two main submodules. One for the low-level interactions such as capturing a photo and saving it to a le or recording audio data. And one module for higher level functions such like - playing a sequence of sounds or switching between microphones.

HCI COURSEWORK 2

!5

This is why such functionality needs both a low and high level programming. There are plenty of open-source software that will allow for this, but if necessary they can be reimplemented.

Server Level: The server level programming can be broken up into these components:
Server Level Software Description

Analysng Planning Prediction

Performing quantitative analysis on audio and visual information sent by the client Includes planning for power consumption, food supplies and shopping orders Suggestions based on the observed user behaviour

Analysing: This component is the most essential component for the operation of the smart refrigerator. This component is responsible for recognising different types of food, classifying them, looking up their Best Before date and sending metadata about the nding back to the client. This component needs to be created in two phases. - Learning: A database of different types of food must be created and then labeled and " " processed so that the learning algorithms can be learned1. This will be used as " " foundation for the image recognition module. It is also worth noting that the learning " should not be a one time process. The algorithm has to be able to account for " " feedback for new items, or items that were detected incorrectly.
"

- Image recognition: A well performing component that quickly analyses and extracts metadata " from a series of photos (in our case there will be four per batch - because there are " four cameras per shelf). There are a lot of open-source solutions that can be used for " this task, but permanence in terms of speed and accuracy is crucial. Therefore an " " extensive research of all possible solutions should be carried out prior to implementing " a specic one.

Planning: The planning component is used to receive a devices metadata (the metadata can be about the content, user behaviour history etc.) and then useful conclusions can be made based on the data. Once a conclusion is taken, the original refrigerator unit is notied of the decision and it is up to the main computer controller to act upon it.

!
! 1

Learned is a term used for training a machine learning algorithm on a specic dataset. !6

HCI COURSEWORK 2

Prediction: Predictions are a useful way of notifying the user about future events. Predictions can be in various forms (e.g. power consumption prediction, food depletion prediction, etc.). Such predictions are very useful, because they can be fed back into the planner component, so that it can act on them. Predictions are relatively easy to make, because they are made on historical data. Having said that however, historical data is now always relevant to future events, so each prediction should come out with a weight factor, which adjusts according to the certainty of the prediction.

! !

Conclusions

Having a smart internet enabled refrigerator is useful. But ultimately we have to see how this product will t into the Internet of Things. How it can utilise information from other devices and act on it. We also have to look into the different ways for the device to provide a feed for other Internet Enabled devices, so that they can act on the information contained and gathered by it.

Making use of data from the refrigerator: Since the refrigerator is only storing metadata in the form of XML, it can t large amounts of it before running out of space. This fact means that for every person (or even family) can essentially have its refrigerator history stored inside. There are many ways in which this can come in useful, but here are some of the notable ones:

-Analysing food habits:! ! Keeping track of what, when and how one eats can come in handy when a food habit prole is created. This can be based on the past year(s) of data. Thus for example a visit to your dietician (who was an internet enabled device) can reveal the following (sample) prole: Consumption of Sugars
90g 80g 70g 60g 50g

April

May GPA

June Consumed

July

HCI COURSEWORK 2

!7

Which quickly reveals a tendency in eating too many sugars per month. This information can then be used by your dietician in order to create a personalised diet affecting only the foods you eat too much of. The same algorithm can be applied across whole families - if a family is consuming too much then theres clearly someone who is not well regimented.

- Shopping Lists:! ! It is quite often that a person nds out that they have to go home, but they are next to a shopping centre having no idea what needs to be bought. This problem can easily be eliminated by a smartphone app that reads in the data from your fridge. Food in stock, items in low quantity, items with nearing Best Before date - the smartphone can easily process all this data and display it in an orderly manner, allowing you to plan ahead and decide on what needs to be bought.

- Power Consumption: " Lets say that a person is trying to minimise his/her the costs of living. An application that samples all internet enabled devices can read in the data for power consumption over the past few months and create an easily interpretable report of which appliances consume most electricity. Digging a little bit further - each appliance can then explain the reasons behind the power consumption. In the case of the fridge - it can reveal that an item that has to be kept at minimum temperature has been sitting in it, without being used. The user can then act on this information - ultimately helping him/her to understand where costs are being incurred from.

- Recipe Ideas: ! During or before a shopping sessions it is always good to have in mind what you will be cooking when you go back home. Well sometimes even having a list of ingredients can be really unhelpful. An app that reads in your stocked products can suggest new things too cook and try. The app can also read your past history of food items, to check what your preferences are in order to give you a custom tailored experience just for you.

! !

Making use of data from other sources: - A meal to t your schedule:! " The refrigerator can allowed access to your personal calendar. Knowing that you will have only half an hour for lunch it can match recipes that you have the ingredients for which will t your schedule.

HCI COURSEWORK 2

!8

Only 30 minutes for Lunch? Try the following:

Meal Omelette Stir-Fry Sheppards Pie Preparation Time 15mins 19mins 23mins
Sample Output

!
- Diet from a dietician

" Following the example of the dietician, who has prepared a specialised diet for you or your family, the refrigerator can then take that diet into account by accessing it remotely. It can then suggest recipes and shopping lists accordingly.

- Cheapest shopping plan: ! If the refrigerator suggest a shopping list and you decide to accept it from the touch screen panel, the device can then connect to different shopping centres and query their online servers about the estimated price of your shopping list. Based on the prices that are returned it can then present you with different options that you can select.

I have made a list of estimate pricing for your oder

Provider Tesco ASDA Sainsurys Cooperative Price 23.20 24.80 28.60 Out of Stock (Leek)

!
In the same way the refrigerator can detect possible foods that you are running out. Scanning through the daily deals it might nd a match for a specic shop and thus prompt you to save. I have detected that your supply of eggs is almost over. I have found that Tesco has a promotion on eggs: 24 Large Eggs - 1.50 Delivered.! Would like me to order it for you? Yes No

!
HCI COURSEWORK 2 !9

-Suggestions Based on Yelp Ratings: The fridge can learn your dietary habits not only from statics that it makes, but also from other available resources. It can for example connect to your Yelp2 prole and look at your restaurant reviews. It can then aim to detect a preference for a certain kitchen. These preferences can then be used in suggestions for recipes etc.
"

2 !

Yelp is an online urban city guide that helps people nd places to eat. !10

HCI COURSEWORK 2