“Lifelogging” and neurophysiological computing: Will we forget how to forget?

One of the most famous examples of reminiscence includes a
madeleine dipped in tea, which lead to almost 3,000 pages of recollection by
the narrator in the beginning of Marcel Proust's novel In Search of Lost Time,
and we have all experienced these sensory triggers to a particular memory. Remembering
the past helps us to re-examine our lives, make choices, and share personal
accomplishments. We often use external devices to help us remember
events big and small, and with advances in technology, we often record and make
plans using a variety of digital devices such as iPhones, Microsoft’s Outlook,
and even smart watches. We have the capability to store a lifetime of data with
these advanced technologies, and with the advent of Facebook, Twitter, “selfies”,
and blogs it has become routine for many people to document their lives on a
daily basis in a digital form, a practice that has been referred to as
“lifelogging.” The outcome of documenting activities digitally are human
digital memories (HDM), which have been defined as “a combination
of many types of media, audio, video, images, and many texts of textual content [1].




The concept of recording and then later having the ability
to review certain documents was first proposed by Dr. Vannevar Bush in 1945
when he described the “Memex”
(a combination of “memory” and index”) in an issue of Atlantic Monthly [2]. As described in the article,
a Memex was “a device in which an individual stores all his books, records, and
communications, and which is mechanized so that it may be consulted with
exceeding speed and flexibility. It is an enlarged intimate supplement to his
memory.”The device would look like
a desk where documents were either recorded via microfilm or photography.

From u-tx.net

Since that time, many
similar devices have been developed, but a revolutionary advance was seen with
Microsoft’s SenseCam a wearable camera with a wide-angle lens and multiple
sensors, including an infrared sensor to detect the presence of other people.
The camera takes a photo every 30 seconds, resulting in up to 2,500 photos a
day and is capable of storing 30,000 images in total. Photos can then uploaded
to a computer and viewed later using a Microsoft application [3]. SenseCam was developed as a memory aid and there has been over 50 research institutions
that have used the device in a variety of studies involving memory and behavior [4]. Notably the SenseCam has shown promising results in studies where it
was used as memory aid for a child with anterograde amnesia [5] and with adult patients that were
suffering from amnesia [3]. Aside from the medical
purpose that a camera such as SenseCam
could potentially serve, “lifelogging” has become more socially acceptable as
we live in a digital age where Facebook posts and Twitter feeds are consumed
constantly, and “selfies”
are a regular occurrence at most events.

The SenseCam. From microsoft.com

However, our memories and our experiences are made up of more
than static images. Our memories are composed of sensory information, such as
temperature and smells, and especially emotions and physiological signals. The
next step of HDM would involve going beyond just digital images, and instead
would include physiological information that has been captured with sensors.
This type of information could allow for a more vivid recall and potentially
could remind us how we felt at any point in time [4]. Wearable systems that
incorporate sensors which collect data for future review have been the focus of
many researchers. One example of this type of system is the “Physiological Sensor Suite (PSS),” which
collects electrocardiogram (ECG), electromyogram (EMG), electrooculogram (EOG)
and through-hair electroencephalogram (EEG) that is then sent wirelessly to a
data sensor [6]. Although the sensors of this
suite did not require a gel to be applied on the skin first, the most
effortless way to use sensors to record our daily activities would be smart
fabrics, such as “smart shirts” that combine textiles and wireless sensors
to monitor heart rate, angle of inclination, body temperature, and location [7]. Shirts of this type were originally
designed for a hospital setting as a noninvasive method to monitor patients,
but in the age of personal computing, technologies like this could potentially
be used outside of a hospital.

An example of a "smart shirt". From howstuffworks.com

Of course, these
types of sensors that record physiological data such as heart rate or body
temperature would be essentially meaningless alone; it would be difficult to
reconstruct any memory, even a simple memory, based on physiological data since
it is so ambiguous. However, if physiological data were supplemented with more
data, such as a photo of the user, the location where the data was recorded,
and the temperature, these clues together may help to trigger certain memories
or parts of memories. One example of a system that works to combine data from a
variety of sources is the AffectAura, an emotional prosthetic
where data is collected from devices such as a microphone, Microsoft’s
Kinect, and a webcam to predict emotional states such as engagement,
valence, and arousal. Six participants were recorded over 4 days, and based on
the data collected, users were able to reconstruct stories about their days [8]. There are still multiple
challenges associated with creating accurate systems that could reveal
emotional aspects of a memory, especially when systems and devices are created
that go beyond only capturing photos. However, sensors will only become smaller
and smaller in the coming years, and as a society we have a great interest in
recording events for cultural reasons, so it is not unreasonable that one
point, most people will participate in the act of “lifelogging” and the
creation of HDM.




It is important to note though that physiological data or
even photos can only act as triggers to a memory, as there is no method to
actually capture an exact “memory” or a “thought.” Additionally, not only would
an HDM need to incorporate data from a variety of sensors and then correctly
corroborate and translate this data into meaningful information (a significant
challenge), but an entire lifetime of
memories would need be recorded for an accurate reflection and then a database
that is searchable would also be required.




If however we could accurately record and then disseminate
data that could compose a memory, does this documentation act as crutch?
Reminiscing and sharing personal stories with families and friends is a basic
human experience that acts as way to connect with others. If instead of
memorializing a lost relative through stories or laughing with friends over a
childhood experience, we could just push “play” on a device, how would that
change us? We already live in a society where any question can be answered with
a quick Google search on a phone requiring no discussion between two people, but
how would our interactions with others change if we could just “Google” how a
past experience played out or made us feel? (MIT professor Dr. Sherry Turkle has been studying how
technology impacts people for over 15 years and has written numerous articles
and books on the topics, and given this recent interesting TED talk on
the subject). Devices that use information from HDM are meant to help us with
the reminiscence process, but what if these devices are actually making us lose
that ability, or at the very least, fundamentally altering the memory process?




Additionally,
just as remembering is central to our existence, so is forgetting. Just because
at some point in the future we may be able to document a person’s entire
lifetime with a wealth of data, including physiological data, should we? Just
this year, researchers from the University of Basel discovered the musashi
protein, a protein that appears to inhibit molecules that stabilize synaptic
connections. These connections are important for the development of memories,
and based on this discovery, it appears that forgetting
is an active biological process. The biological processes behind
remembering and forgetting appear to work together, and forgetting is not just
a passive process [9]; we most likely forget for a
reason, even if more research is necessary to discover why.






References




1. Kelly L. The Information Retrieval
Challenge of Human Digital Memories. Proceedings of the 1st BCS IRSG Conference
on Future Directions in Information Access [Internet]. Swinton, UK, UK: British
Computer Society; 2007 [cited 2014 Aug 19]. p. 17–17. Available from:
http://dl.acm.org/citation.cfm?id=2227895.2227913

2.Bush V. As We May
Think. The Atlantic [Internet]. 1945
Jul [cited 2014 Aug 19]; Available from:
http://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/


3. Hodges S, Williams L, Berry E, Izadi S, Srinivasan J, Butler A, Smyth G, Kapur N, Wood E. SenseCam: A
Retrospective Memory Aid. UbiComp 2006 4206:
177-193.


4. Fairclough, S.H., and Gilleade, K. (2014). Capturing Human Digital Memories for Assisting Memory Recall In Advances in Physiological Computing, S.H. Fairclough, and K. Gilleade, eds.(Springer London), pp. 211-234.

5. Pauly-Takacs K,
Moulin CJA, Estlin EJ. SenseCam as a rehabilitation tool in a child with
anterograde amnesia. Mem Hove Engl.
2011, 19(7): 705–12.

6. Matthews R,
McDonald NJ, Hervieux P, Turner PJ, Steindorf MA. A wearable physiological
sensor suite for unobtrusive monitoring of physiological and cognitive state. Conf Proc IEEE Eng Med Biol Soc 2007, 2007:
5276–81.

7. López G, Custodio
V, Moreno JI. LOBIN: E-textile and wireless-sensor-network-based platform for
healthcare monitoring in future hospital environments. IEEE Trans Inf Technol Biomed Publ 2010, 14(6): 1446–58.

8. Mcduff D, Karlson
A, Kapoor A, Roseway A, Czerwinski M. AffectAura: An Intelligent System forEmotional Memory.

9. Hadziselimovic N,
Vukojevic V, Peter F, Milnik A, Fastenrath M, Fenyves BG, et al. Forgetting Is
Regulated via Musashi-Mediated Translational Control of the Arp2/3 Complex. Cell, 2014, 156(6): 1153–66.

Want to cite this post?




Strong, K. (2014). “Lifelogging” and neurophysiological computing: Will we forget how to forget? The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2014/08/lifelogging-and-neurophysiological.html

“Pass-thoughts” and non-deliberate physiological computing: When passwords and keyboards become obsolete

Imagine opening your email on your computer not by typing a number code, a password, or even by scanning a finger, but instead by simply thinking of a password. Physical keys and garage door openers could also become artifacts of the past once they are replaced with what could be referred to as pass-thoughts. Just last year, researchers at UC Berkley used EEG signals emitted from subjects as biomarker identifiers to allow access to a computer. The entire system – the headset, the Bluetooth device, and the computer – had an error rate of less than 1%.¹ While wearing EEG headsets to open our devices may seem futuristic, this type of scenario could become more prevalent in the future due to advances in physiological computing (PC). Physiological computing is a unique form of human computer interactions because the input device for a computer is any form of real-time physiological data, such as a heart-rate or EEG signal. This is in stark contrast to the peripheral devices that we are familiar with today, such as a keyboard, remote, or mouse.²



The field of physiological computing is still quite new, but research has suggested that different physiological computers require varying degrees of intentionality from the human user, and that the devices can be placed on a spectrum.³

Via physiologicalcomputing.net

On one end of the spectrum are technologies where users can deliberately interact with input devices based on voluntary muscle movement such as electrooculography (EOG) to direct the movement of a cursor (shown in 2 on the spectrum).⁴ In contrast, brain-computer-interfaces (BCI)­ such as the exoskeleton showcased at the recent first kick for the 2014 World Cup, bypass this step­ since BCIs are often developed for those with diminished movement capacities and disabilities. However, in both cases the general principle is the same: the interface is ultimately translating a neural signal that the user has specifically and deliberately directed to complete a task.⁵

Via cbsnews.com

Non-deliberate PC, on the other hand, bypasses any voluntary input, and instead involves a “biocybernetic” approach where spontaneous physiological changes, such as a heart rate or brain electrical signals are recorded via an electrocardiogram (EKG) or an electroencephalogram (EEG), respectively. These signals are then correlated to meaningful information, such as the case mentioned above where specific EEG signals act as identifying information to allow access to a computer. These types of technologies are able to associate recorded physiological changes with the motivational, cognitive or emotional state of the user. Once the interface determines the user’s emotional state, it can often adapt in an attempt to promote a specific type of positive mentality or negate a potentially hazardous emotional state. For example, if a computer calculates that the user is stressed, it can play soothing music or offer to help to diffuse the negative situation. The long-term recording of physiological data usually for learning purposes is referred to as ambulatory monitoring.⁶

Via thenextweb.com

Technologies that incorporate aspects of physiological computing, such as the recently released Kinect 2 from Microsoft, have recently become prevalent in consumer products. Using technology similar to that developed at MIT and referred to as Eulerian Video Modification,⁷ the camera on the Kinect detects small changes in skin color pigmentation and monitors heart rate optically (although pulse rate can be an indicator for an emotional state, at this time the Kinect 2 focuses on monitoring heart rates during physical activity, but does not correlate this data to an emotional state).

Portable, wireless sensors that are able to not only record, but also convert raw EEG signals into some form of meaningful information are currently available. EPOC by Emotiv and MindWave by NeuroSky have developed and currently sell wireless headsets that act as EEG sensors. Since certain EEG signals could be used as an indicators of a specific emotional state, such as frustration,⁸ the interface can label or adapt to a user in real-time. That said, while these EEG sensors give the impression that the user can execute commands with seemingly only the power of thought, these technologies are not yet able to comprehend intentions or mimic emotions (but, see recent data on AI recently passing Turing Test). For an interface to recognize intentions, first a system, similar to a dictionary, must be created so that the computer records the EEG data for a series of tasks that the interface will be able to recognize later. Not to mention, “intent” is still not clearly understood mechanistically through neuroscience.

Pertinent ethical issues include those related to ownership and privacy. Raw EEG or electrocardiogram (ECG) data is powerful information, especially when linked to changes in an emotional state. Emotiv will provide the raw EEG data from its users for an additional fee, but NeuroSky does not provide this information. Do we have any claim over our own (neuro-)physiological data once it leaves us? Even if raw EEG signals are worthless without an algorithm to decipher the meaning, the data still originated from only one, original source. Until it was pulled for ownership issues (NASA wanted to ensure that the data was no longer federal property), the EKG of Neil Armstrong’s heart as he took the first steps on the moon was to be auctioned off last year.⁹ But did NASA ever have a right to lay claim to this information, even if without an algorithm the EKG is seemingly meaningless? Or, does Neil Armstrong (or in this case, his family) have any right to claim ownership since NASA paid for and played a role in developing the technology that enabled this collection? These will be the types of questions that need to be addressed as more and more people continue to offer up their physiological data by using these types of technologies and popular commercial venues.

Via time.com

It seems inevitable that one day enough people will participate in the use of these EEG sensors and a massive database of neurological signals will begin to develop. Having a large dataset of neurological data that can potentially be correlated to disease states is already the goal of well established companies such as Lumosity ¹⁰ and BrainResource.¹¹ Additionally, the United States government recently launched PCORnet: The National Patient-Centered Clinical Network Project with the intention of building a national health-data system by combining data from 29 different health data networks.¹² The United Kingdom has met ethical conflicts with the introduction of a similar system, care.data,¹³ and the United States already has a history of alleged National Security Agency privacy violations, but government backed organizations are moving forward with the massive collection of medical records and perhaps one day, extensive physiological data. A precedent for having a dataset of extensive, personal information is the company 23andMe, which provided information based on DNA analysis. Nothing is protecting the users of 23andMe’s service from having their personal information sold,¹⁴ but the Genetic Information Nondiscrimination Act (GINA) passed in 2008 protects people from having their genetic information interfere with insurance policies and employment. This type of law does not exist for neurological data. Regulations and discussions should be taking place now before companies like Emotiv or NeuroSky have 5 years’ worth of data from their customers whose privacy is not protected in the slightest.




Already specific EEG signals can be used to characterize neurological disorders. With the collection of more data, we have the potential to be able to recognize and use specific signals as “brain signatures” for other neurological disorders or even tendencies toward certain behaviors (The well-established company Brainwave Science is a proponent of using EEG technology to test guilt or innocence). This ability, while incredibly powerful, has a high risk for abuse in terms of covert monitoring of individuals.¹⁵ Of course, if a patient has epilepsy, a discrete EEG sensor that has the power to be predictive for seizure activity could greatly increase the health, safety, and quality of life for these patients.¹⁶ Would it be appropriate to monitor a person who has been given a neurological diagnosis that has rendered them emotionally unstable if the EEG sensor could detect a very high or low state though? If that EEG sensor means that they are deemed stable enough for certain activities they were once denied, such as driving, does that make the constant monitoring worth what many would consider a violation of privacy?






References




(1) New Research: Computers That Can Identify You by Your Thoughts http://www.ischool.berkeley.edu/newsandevents/news/20130403brainwaveauthentication (accessed Jun 26, 2014).


(2) Fairclough, S. H. Fundamentals of Physiological Computing. Interact. Comput. 2009, 21, 133–145.


(3) Physiological Computing F.A.Q. Physiological Computing Blog. http://www.physiologicalcomputing.net/?page_id=227 (assessed on June 28, 2014).


(4) Allanson, J.; Fairclough, S. H. A Research Agenda for Physiological Computing. Interact. Comput. 2004, 16, 857–878.


(5) Allison, B. Z.; Wolpaw, E. W.; Wolpaw, J. R. Brain-Computer Interface Systems: Progress and Prospects. Expert Rev. Med. Devices 2007, 4, 463–474.


(6) Fairclough, S.H., and Gilleade, K. (2014). Meaningful Interaction with Physiological Computing. In Advances in Physiological Computing, S.H. Fairclough, and K. Gilleade, eds. (Springer London), pp. 1–16.


(7) Wu, H.-Y.; Rubinstein, M.; Shih, E.; Guttag, J.; Durand, F.; Freeman, W. T. Eulerian Video Magnification for Revealing Subtle Changes in the World. ACM Transactions on Graphics (Proc. SIGGRAPH 2012 2012, 31.


(8) Kapoor, A.; Burleson, W.; Picard, R. W. Automatic Prediction of Frustration. Int. J. Hum.-Comput. Stud. 2007, 65, 724–736.


(9) Pearlman, R. Z. Neil Armstrong’s “Heartbeat,” Apollo Joystick Pulled from Auction http://www.space.com/21228-neil-armstrong-apollo-artifacts-auction.html (accessed Jun 26, 2014).


(10) Sternberg, D. A.; Ballard, K.; Hardy, J. L.; Katz, B.; Doraiswamy, P. M.; Scanlon, M. The Largest Human Cognitive Performance Dataset Reveals Insights into the Effects of Lifestyle Factors and Aging. Front. Hum. Neurosci. 2013, 7.


(11) McRae, K.; Rekshan, W.; Williams, L. M.; Cooper, N.; Gross, J. J. Effects of Antidepressant Medication on Emotion Regulation in Depressed Patients: An iSPOT-D Report. J. Affect. Disord. 2014, 159, 127–132.


(12) Collins, F. S.; Hudson, K. L.; Briggs, J. P.; Lauer, M. S. PCORnet: Turning a Dream into Reality. J. Am. Med. Inform. Assoc. 2014, amiajnl–2014–002864.


(13) Callaway, E. UK Push to Open up Patients’ Data. Nature 2013, 502, 283–283.


(14) Seife, C. 23andMe Is Terrifying, but Not for the Reasons the FDA Thinks. Scientific American, Nov. 27, 2013. http://www.scientificamerican.com/article/23andme-is-terrifying-but-not-for-reasons-fda/ (accessed Jun 26, 2014).


(15) Deceiving the Law. Nat. Neurosci. 2008, 11, 1231–1231.


(16) Jouny, C. C.; Franaszczuk, P. J.; Bergey, G. K. Improving Early Seizure Detection. Epilepsy Behav. EB 2011, 22 Suppl 1, S44–48.






Want to cite this post?




Strong, K. (2014). “Pass-thoughts” and non-deliberate physiological computing: When passwords and keyboards become obsolete. The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2014/06/pass-thoughts-and-non-deliberate.html