Disrupting diagnosis: speech patterns, AI, and ethical issues of digital phenotyping

By Ryan Purcell, PhD

Jim Schwoebel, presenter at April The Future Now: (NEEDS)

Diagnosing schizophrenia can be complex, time-consuming, and expensive. The April seminar on The Future Now: (NEEDs) Neuroscience and Emerging Ethical Dilemmas at Emory focused on one innovative effort to improve this process in the flourishing field of digital phenotyping. Presenter and NeuroLex founder and CEO Jim Schwoebel had witnessed his brother struggle for several years with frequent headaches and anxiety, and saw him accrue nearly $15,000 in medical expenses before his first psychotic break. From there it took many more years and additional psychotic episodes before Jim’s brother began responding to medication and his condition stabilized. Unfortunately, this experience is not uncommon; a recent study found that the median period from the onset of psychotic symptoms until treatment is 74 weeks. Naturally, Schwoebel thought deeply about how this had happened and what clues might have been seen earlier. “I had been sensing that something was off about my brother’s speech, so after he was officially diagnosed, I looked more closely at his text messages before his psychotic break and saw noticeable abnormalities,” Schwoebel told Psychiatric News. For Schwoebel, a Georgia Tech alum and co-founder of the neuroscience startup accelerator NeuroLaunch, this was the spark of an idea. Looking into the academic literature he found a 2015 study led by researchers from Columbia University who applied machine learning to speech from a sample of participants at high risk for psychosis. They found that the artificial intelligence correctly predicted which individuals would transition to psychosis over the next several years.

Image Courtesy of Pixabay user mohamed_hassan.

Schwoebel went on to found NeuroLex Laboratories, which is developing technology to analyze speech samples for diagnostic purposes. For NeuroLex, schizophrenia is only one of several neuropsychiatric disorders that may be diagnosable by AI-mediated linguistic analysis. In their early stages, depression, Alzheimer’s (AD) and Parkinson’s disease (PD) also may affect the brain in unseen ways that algorithms can identify before a clinician. Early diagnosis of these conditions could have a profound impact on patient outcomes and the development of future treatments. While there are no disease-reversing cures available for any of these disorders currently, preventing psychotic episodes is a major goal in schizophrenia treatment and early diagnosis at least provides the opportunity for intervention. For neurodegenerative diseases, the case for early diagnosis may be even more compelling. By the time many patients are diagnosed with a neurodegenerative disease such as AD or PD, the disease has already done catastrophic and perhaps irreversible damage to the brain. Clinical trials that include such patients would therefore be doomed before they even begin. Knowing about the disease earlier could provide more opportunity for treatment, and would also have practical benefits for families wanting to plan for future care. 

NeuroLex is far from the only startup in the digital phenotyping space. Jeff Arnold, the founder of WebMD co-founded Sharecare, which offers analysis of phone calls to determine stress level among other personalized medical services. The former director of the US National Institute of Mental Health, Dr. Thomas Insel co-founded Mindstrong, which measures countless aspect of smartphone use as indicators of mental and emotional health. We know of Facebook’s efforts to identify users who may be suicidal and, it is safe to assume, Facebook and Google are interested in (if not already performing) other digital phenotyping analyses. 

Image Courtesy of Max Pixel

More efficient, more accurate, and earlier diagnosis of neuropsychiatric diseases could provide real benefits for patients, their families, and researchers. However, there are also real ethical issues that need to be considered. First, like any application involving AI, there is an increasing appreciation that bias may be a major hurdle. Related to speech, Schwoebel noted that there are obvious regional (think Brooklyn vs. Birmingham) as well as racial, ethnic, and gender differences in how people speak and the words that they choose. He emphasized the importance of a diverse training set, which would hopefully head off harmful and embarrassing situations like when a Google Photos algorithm generated some of the most racist tags imaginable. Yet the really scary part may be that the public will likely never know about most of the discrimination that happens in the background as we browse the web. Diverse training in the research phase and transparent computations would likely help avoid systemic biases but it is doubtful that they could be eliminated completely. 

Image Courtesy of Pexels user Andres Urena

Speech and voice data may contain everything from the most mundane to the very personal and sensitive and thus privacy issues could also present a significant challenge, both legally and ethically. From a legal perspective in the United States, different states have varying wiretapping and voice recording statutes. In some states, it is legal to record a conversation with the consent of only one party, in others the consent of all parties is required. This may seem like an easy ethical judgment – simply go with the stricter regulation and get everyone’s consent – but it probably is not that simple in practice. Another important consideration is how the speech data is collected. AI-powered home assistants like Google Home and Amazon Echo and even many smartphones and televisions are listening, and the owner of the device, not to mention other people within earshot, may not know exactly what they have consented to having recorded. Just recently, Amazon was asked to explain why exactly an Echo emailed an audio recording of a conversation that a Portland woman had at home to one of her husband’s co-workers. 

Lastly, there are concerns about how this technology could be used. Predictive data related to an individual’s likelihood to develop neuropsychiatric conditions that may result in long periods of disability would be very valuable to insurance companies and employers, as only two examples. In a one party voice recording consent state, an applicant for insurance or a job would not even need to agree to submit to this sort of analysis. In this era of deregulation, without obvious appetite for increased oversight at the Federal level, it will likely be up to the private sector to police itself and decide on ethical principles to guide the development and implementation of these technologies. After all, the more incidents of ugly AI bias and gross disregard for privacy that make it into the public view, the less interest there will be in adopting these technologies, which could seriously hamper their potential for good. There is little doubt that digital phenotyping in its many forms has the potential not only for improved efficiency at drastically lower cost, but also to enhance the ability and extend the reach of clinicians. Thoughtfully considering and addressing these concerns and should improve chances of reaching that potential, not hinder them.

Want to cite this post?




Purcell, R. (2018). Disrupting diagnosis: speech patterns, AI, and ethical issues of digital phenotyping. The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2018/06/disrupting-diagnosis-speech-patterns-ai.html 

Downloading Happiness

By Sorab Arora

Sorab Arora is currently a Master’s in Public Health student at Emory University, specializing in Healthcare Management and Policy. He has researched health technology design and strategy focused on behavioral medicine, most recently at Northwestern University’s Center for Behavioral Intervention Technologies. Arora is a graduate of both the University of Chicago (Summer Business Scholar – 2017) and Grinnell College (2016), where he has bridged social entrepreneurship with mobile technologies and medical innovation. 

With median adult smartphone ownership rising to nearly 70% in advanced markets, individuals ranging from wealthy millennials to homeless youth have unprecedented access to mobile technologies (Poushter, 2016; Ben-Zeev et al., 2013). From “swiping” potential soulmates to ordering prescription glasses to one’s door, the proliferation of opportunities for immediate gratification through mobile applications only continues to grow. In what economists have now termed the “Fourth Industrial Revolution,” this period of integrated consumer technologies focuses on human-centered design and improved efficiency across global sectors (Schwab, 2017). In healthcare especially, mobile health (mHealth) platforms offer an innovative new element to how medicine can be conceptualized, delivered, and implemented. 

The melding of mHealth technologies in the field of psychotherapy offers a marriage of tremendous promise. As Fitbits, Apple Watches, and the like have taken center stage as wearables to enhance wellness, data collected from these sources offers a wealth of vital, longitudinal information. Predictive analytics allow healthcare providers (and consumers) to gain a more precise understanding of their health through personalized strategies based on their past health trends (Siegel, 2013). Coupled with artificial intelligence and machine learning, predictive analytics can offer improved differentiations in care by closely analyzing biological and behavioral markers. The value in doing so is a greater understanding of healthcare patterns at both the individual and the societal level, driving more insightful strategies for improvement. In other words, mobile technologies have opened the door to data-mining metrics that were once nearly impossible to quantify– including behaviors, cognitions, and emotions (Mohr, Zhang & Schueller, 2017). 

Image courtesy of Wikimedia Commons.

As it currently stands, smartphones have access to a tremendous amount of personal data ranging from location, movement, circadian rhythms, exercise, diet, and even ambient light. Applying analytics to the world of mental health can have dramatic impacts not as a replacement for current treatments, but by offering more accurate indicators of what and how to treat. Telemedicine -- usage of technology to remotely deliver healthcare – has also played a key role in psychiatry by facilitating innovative symptom tracking and communication between patients and providers (Mermelstein et al., 2017). Aligned with these strides in telepsychiatry, recent studies indicate efficacy in correlating behavioral markers to clinical disorders in efforts to more precisely pinpoint once-overlooked symptomology (Harari et al., 2016). 

Implementing smartphone usage in clinical settings has been a recent focus for The National Center for Telehealth & Technology through their efforts in improving mood and anxiety disorders, especially for veteran populations (Luxton et al., 2011). The move to focusing on everyday individuals using and benefiting from similar mobile apps, however, comes with its own string of unique legal, ethical, and medical concerns. 

Koko is a crowd-sourced platform for providing positive, constructive feedback to others that was based on cognitive-behavioral therapy to change conceptualization of challenging events ranging from teen bullying to work stress in efforts to build resilience. Woebot helps facilitate conversation and track moods through quantitative and qualitative measures utilizing machine learning. Headspace is a popular mobile app across Android and iPhone systems that teaches meditation and mindfulness through short, daily modules. These applications just scratch the surface of how healthtech innovation and human-centered interactions have fused in recent years. But with this plethora of opportunity comes a slew of related questions. What are the related ethical concerns and how is privacy safeguarded? How would one measure adherence and incentivize actual usage of these apps? Would these new technologies yield clinically significant improvements in psychiatric populations? All these questions (and more) beg to be answered, but the issue of efficacy takes center stage. 

Image courtesy of Wikimedia Commons.

As mental health and neurotechnologies are brought to market, there have been minimal barriers to entry in creating mobile apps with apparent face value. From ideation stages to actual product launch, several healthtech designs are relatively unregulated and untested regarding their actual validity as medical devices or supplements (Mohr, Zhang & Schueller, 2017). Because of this lack of quality control, individuals may be managing their stress and mental health disorders in ways that do more harm than good. 

Calling for evidence-based mental health apps and screening related technologies through more efficacious standards paves the path for creating clinically significant improvements long-term for at-risk patients (Lui, Marcus & Barry, 2017). While resources like PsyberGuide evaluations, the American Psychiatric Associations (APA) App Evaluation Model, and similar criteria have helped equip individuals with skills to discern between mental health apps by providing holistic evaluation criterion, a fundamental issue remains. Too many apps lack efficacy to be hailed as breakthroughs in the current climate of healthtech design and innovation– especially those in the fields of mental health and neuroscience. 

With privacy concerns as a major player in healthcare data storage, requirements for novel healthtech apps, wearables, and software go beyond simply achieving gold standards for randomized clinical trials or patient satisfaction. These technologies handle highly sensitive personal information, making bioethics and legal considerations key factors in data storage and analysis. Concerns over GPS and raw audio data already introduce a design challenge for individuals who cannot allow these systems to run in workplace settings or confidential meetings, reminiscent of “Big Brother” collecting too much information of daily activities (Klasjna et al., 2009). 

As the ability to collect and synthesize more intimate data emerges, the process of ensuring data security through deep learning software must draw on psychologists to effectively collaborate with colleagues across healthcare, computer science, and engineering. If used as a medical technology supplement to face-to-face therapy or psychotherapeutic interventions - compliant with The Health Information Technology for Economic and Clinical Health Act (HITECH Act) – the need for ensuring patient confidentiality and privacy becomes the responsibility of more than just the provider (Luxton et al., 2011). 

Image courtesy of Wikimedia Commons.

As wellness based health-tech products are launched to market, evaluating their efficacies from a more rigorous clinical and legal standpoint becomes crucial. Because advanced technologies have significantly altered daily interactions at a personal and professional level, leveraging human-computer interactions in the field of behavioral medicine offers tremendous potential for enhancing short- and long-term treatment strategies. Tracking emotional states through novel frameworks can therefore serve as a tool for “downloading” a healthier state of mind – given adherence to these applications as if they were tangible medication itself. The potential benefits of improved patient-provider relationships, decreased per capita cost, and increased access to care make discussing cross-disciplinary strategy, limitations, and directions invaluable with regards to emerging neurotechnologies. 

References

 

Ben-Zeev, D., Davis, K. E., Kaiser, S., Krzsos, I., & Drake, R. E. (2013). Mobile technologies among people with serious mental illness: opportunities for future services. Administration and Policy in Mental Health and Mental Health Services Research, 40(4), 340-343. 

Harari, G. M., Lane, N. D., Wang, R., Crosier, B. S., Campbell, A. T., & Gosling, S. D. (2016). Using smartphones to collect behavioral data in psychological science: Opportunities, practical considerations, and challenges. Perspectives on Psychological Science, 11(6), 838-854. 

Klasnja, P., Consolvo, S., Choudhury, T., Beckwith, R., & Hightower, J. (2009). Exploring privacy concerns about personal sensing. Pervasive Computing, 176-183. 

Lui, J. H., Marcus, D. K., & Barry, C. T. (2017). Evidence-based apps? A review of mental health mobile applications in a psychotherapy context. Professional Psychology: Research and Practice, 48(3), 199. 

Luxton, D. D., McCann, R. A., Bush, N. E., Mishkind, M. C., & Reger, G. M. (2011). mHealth for mental health: Integrating smartphone technology in behavioral healthcare. Professional Psychology: Research and Practice, 42(6), 505. 

Madan, A., Cebrian, M., Lazer, D., & Pentland, A. (2010, September). Social sensing for epidemiological behavior change. In Proceedings of the 12th ACM international conference on Ubiquitous computing (pp. 291-300). ACM. 

Mermelstein, H., Guzman, E., Rabinowitz, T., Krupinski, E., & Hilty, D. (2017). The Application of technology to health: The evolution of telephone to telemedicine and telepsychiatry: A historical review and look at human factors. Journal of Technology in Behavioral Science, 1-16. 

Mohr, D. C., Zhang, M., & Schueller, S. M. (2017). Personal sensing: Understanding mental health using ubiquitous sensors and machine learning. Annual Review of Clinical Psychology, 13, 23-47. 

Poushter, J. (2016). Smartphone ownership and internet usage continues to climb in emerging economies. Pew Research Center, 22. 

Schwab, K. (2017). The fourth industrial revolution. Crown Business. 

Siegel, E. (2013). Predictive analytics. Hoboken: Wiley.

Want to cite this post?




Arora, S. (2018). Downloading Happiness. The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2018/03/downloading-happiness.html

Big data and privacy on the Web: how should human research be conducted on the Internet?

“They said, ‘You can’t mess with my emotions. It’s like messing with me. It’s mind control.'” That’s what Cornell communication and information science professor Jeffrey T. Hancock reported in a recent New York Times article about the public outcry over the now infamous Facebook emotional manipulation study (read on for details). Hancock was surprised and dismayed over the response. He sees the advent of massive-scale sociology and psychology research on the Internet as a “new era” and he has a point. The days of mostly relying on college students as research subjects may be coming to an end. But how should research be conducted in this new online setting? Is it even appropriate to use data from web sites as it is collected now with little, if any, user knowledge and informed consent existing only in the form of privacy policies that nobody reads?¹ In this post I argue that the Internet is not the Wild West and therefore internet-based research should not be allowed to side step established practices of informed consent. Furthermore, significant changes must be made so that these new research opportunities are maximized in the best way possible.

Via Linkedin

 Earlier this year Facebook, the social network with well over 1 billion users, found itself in hot water after publishing a study in collaboration with academic researchers (including Hancock above) that sought to measure “emotional contagion” online.²

In January of 2012 Facebook researchers altered the News Feed content of nearly 700,000 users without their knowledge or explicit consent. Users in a control group had random posts withheld from their News Feeds, irrespective of emotional content, whereas others either had some posts with a positive valence removed or some with a more negative tone hidden. They found that overall, omitting News Feed content – either negative or positive – seemed to affect the emotional valence of subsequent posts. And all those Facebook users who saw fewer emotion-tinged posts subsequently posted fewer updates. In effect, Facebook researchers were successfully able to manipulate the emotions of hundreds of thousands of users simply by altering their News Feed content. The researchers were able to study a phenomenon – emotional contagion – which can be very difficult to study in person due to a variety of potential experimental confounds and they were also able to achieve extraordinary statistical power. But by not explicitly asking for consent many users felt well, used.



In the paper, the authors state that would-be subjects gave informed consent for this research when creating an account. But this may not be true according to a report, which alleges that the word “research” was actually added to the privacy policy months after the study. Anyway, in my case, consent would have been given in the summer of 2005, nearly a decade ago. Even if I did agree to participate in research at that moment in time, I certainly don’t remember it and I could not have had any idea what would be possible on the Internet of 2014. Would you provide your consent today for a study of your Internet behavior in January of 2023? The journal that published the study, Proceedings of the National Academy of Sciences, updated the article with an attached “Editorial Expression of Concern” wherein the Editor-in-Chief Inder M. Verma noted that, as a private company, Facebook is not beholden to the US Government’s Department of Health and Human Services “Common Rule”, which guides human research best practices and advises that, in addition to documented informed consent, subjects should have the ability to opt out of a trial. This, however, is where journals need to step in to enforce ethical standards regardless of the research setting.



In another example, Christian Rudder, one of the founders of the free online dating site OkCupid.com, drew fire recently for revealing the details of an internal study. The website essentially told users who appeared to be poorly matched that they were in fact well-suited and vice versa. His reaction to it all, which may be indicative of the current gulf between traditional brick-and-mortar institution researchers and Silicon Valley entrepreneurs (who may, pardon the pun, have their heads in the cloud) was, effectively, “what’s the big deal?”



The prevailing business model of Web-based companies – offering services for free in exchange for information which they then hope to monetize – inherently disagrees with the ethos of human research. Skipping (for the moment) the issue of informed consent, researchers should first and foremost avoid harming the study participants physically, emotionally or otherwise and should endeavor to protect their subjects’ privacy rather than feed off of it. There does however seem to be an important difference between studying data that is passively collected from users – who should expect that they are forfeiting some amount of privacy in order to get a free service like Facebook in return – and actively manipulating the information that users see and observing their responses.

Via EyeWire

These studies are not limited to social networks and dating sites, however and are quite relevant to neuroscience. Lumosity.com, perhaps the largest brain training website with more than 50 million users around the world, has embarked on the Human Cognition Project. Preliminary results from this project were published in a proof-of-concept study that had N’s well over 100,000.³ However, Lumosity (which I have written about previously on this blog) is very up-front about the Project and seems to be tapping into a widespread desire among the public to get involved in neuroscience and cognitive science research.



Academic researchers are taking advantage of crowdsourcing in neuroscience as well. MIT computational neuroscientist Sebastian Seung has found a solution to the image analysis bottleneck that was slowing down the progress of his group’s research. He developed Eyewire which, in their words, is “a game to map the brain...Anyone can play and you need no scientific background. Over 130,000 people from 145 countries already do. Together we are mapping the 3D structure of neurons; advancing our quest to understand ourselves.” This work has already produced a milestone paper in Nature.⁴



The landscape of Internet research is rapidly changing and, while it certainly presents a number of ethical challenges in terms of privacy and informed consent, there is vast potential for breakthrough studies – of a kind that have never been possible before – to answer important questions. The first question, though, is how to create a safe, effective, environment for research in a setting that is increasingly seen as insecure and where personal information is the most common currency. A necessary first step is to provide would-be subjects with the choice to opt out of a study. In addition, Internet privacy expert Helen Nissenbaum suggests there needs to be a fundamental shift in norms for conceptualizing privacy online. It is not reasonable to expect users to read all privacy policies for websites they visit and, even if they could, it is unlikely they could predict all the potential implications of those often nebulous terms.⁵ Moreover, some sites (such as Facebook) have become so widely used and so necessary for modern communication without any comparable competitors, that it could be argued that users do not really have a choice to take their business elsewhere, and are therefore implicitly coerced to participate.



The Internet offers incredible research opportunities, particularly to understand human behavior. But just because participants never actually step into the lab does not necessarily mean they don’t deserve the same protections as those who do. Besides, the literature is filled with clever, elegant experiments designed to prevent the participants from knowing the goals and hypotheses of the study – there is no reason that couldn’t also be done online. A major challenge facing academia is how to best partner with private companies who may be primarily interested in improving website functionality and in so doing collect large datasets that may be of interest to researchers. They must also address the issue of whether it is ethical to use data collected by private companies without real informed consent or the choice to opt out to answer academic questions. Internet connectedness has opened up incredible new opportunities for research into human behavior and cognition as well as for “citizen scientists” to help expedite progress in neuroscience. Now is the time to codify ethical standards for these studies to ensure that growth in this area continues in the best way possible.





 References 



1. Kelley, P. G., Bresee, J., Cranor, L.F., Reeder, R.W. in SOUPS '09 Proceedings of the 5th Symposium on Usable Privacy and Security. (ACM).



2. Kramer, A. D., Guillory, J. E. & Hancock, J. T. Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences of the United States of America 111, 8788-8790, doi:10.1073/pnas.1320040111 (2014).



3. Sternberg, D. A. et al. The largest human cognitive performance dataset reveals insights into the effects of lifestyle factors and aging. Front Hum Neurosci 7 (2013).



4. Kim, J. S. et al. Space-time wiring specificity supports direction selectivity in the retina. Nature 509, 331-336, doi:10.1038/nature13240 (2014).



5. Nissenbaum, H. A Contextual Approach to Privacy Online. Daedalus 140, 32-48 (2011).





Want to cite this post?




Purcell, R. (2014). Big data and privacy on the Web: how should human research be conducted on the Internet? The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2014/09/big-data-and-privacy-on-web-how-should.html

“Lifelogging” and neurophysiological computing: Will we forget how to forget?

One of the most famous examples of reminiscence includes a
madeleine dipped in tea, which lead to almost 3,000 pages of recollection by
the narrator in the beginning of Marcel Proust's novel In Search of Lost Time,
and we have all experienced these sensory triggers to a particular memory. Remembering
the past helps us to re-examine our lives, make choices, and share personal
accomplishments. We often use external devices to help us remember
events big and small, and with advances in technology, we often record and make
plans using a variety of digital devices such as iPhones, Microsoft’s Outlook,
and even smart watches. We have the capability to store a lifetime of data with
these advanced technologies, and with the advent of Facebook, Twitter, “selfies”,
and blogs it has become routine for many people to document their lives on a
daily basis in a digital form, a practice that has been referred to as
“lifelogging.” The outcome of documenting activities digitally are human
digital memories (HDM), which have been defined as “a combination
of many types of media, audio, video, images, and many texts of textual content [1].




The concept of recording and then later having the ability
to review certain documents was first proposed by Dr. Vannevar Bush in 1945
when he described the “Memex”
(a combination of “memory” and index”) in an issue of Atlantic Monthly [2]. As described in the article,
a Memex was “a device in which an individual stores all his books, records, and
communications, and which is mechanized so that it may be consulted with
exceeding speed and flexibility. It is an enlarged intimate supplement to his
memory.”The device would look like
a desk where documents were either recorded via microfilm or photography.

From u-tx.net

Since that time, many
similar devices have been developed, but a revolutionary advance was seen with
Microsoft’s SenseCam a wearable camera with a wide-angle lens and multiple
sensors, including an infrared sensor to detect the presence of other people.
The camera takes a photo every 30 seconds, resulting in up to 2,500 photos a
day and is capable of storing 30,000 images in total. Photos can then uploaded
to a computer and viewed later using a Microsoft application [3]. SenseCam was developed as a memory aid and there has been over 50 research institutions
that have used the device in a variety of studies involving memory and behavior [4]. Notably the SenseCam has shown promising results in studies where it
was used as memory aid for a child with anterograde amnesia [5] and with adult patients that were
suffering from amnesia [3]. Aside from the medical
purpose that a camera such as SenseCam
could potentially serve, “lifelogging” has become more socially acceptable as
we live in a digital age where Facebook posts and Twitter feeds are consumed
constantly, and “selfies”
are a regular occurrence at most events.

The SenseCam. From microsoft.com

However, our memories and our experiences are made up of more
than static images. Our memories are composed of sensory information, such as
temperature and smells, and especially emotions and physiological signals. The
next step of HDM would involve going beyond just digital images, and instead
would include physiological information that has been captured with sensors.
This type of information could allow for a more vivid recall and potentially
could remind us how we felt at any point in time [4]. Wearable systems that
incorporate sensors which collect data for future review have been the focus of
many researchers. One example of this type of system is the “Physiological Sensor Suite (PSS),” which
collects electrocardiogram (ECG), electromyogram (EMG), electrooculogram (EOG)
and through-hair electroencephalogram (EEG) that is then sent wirelessly to a
data sensor [6]. Although the sensors of this
suite did not require a gel to be applied on the skin first, the most
effortless way to use sensors to record our daily activities would be smart
fabrics, such as “smart shirts” that combine textiles and wireless sensors
to monitor heart rate, angle of inclination, body temperature, and location [7]. Shirts of this type were originally
designed for a hospital setting as a noninvasive method to monitor patients,
but in the age of personal computing, technologies like this could potentially
be used outside of a hospital.

An example of a "smart shirt". From howstuffworks.com

Of course, these
types of sensors that record physiological data such as heart rate or body
temperature would be essentially meaningless alone; it would be difficult to
reconstruct any memory, even a simple memory, based on physiological data since
it is so ambiguous. However, if physiological data were supplemented with more
data, such as a photo of the user, the location where the data was recorded,
and the temperature, these clues together may help to trigger certain memories
or parts of memories. One example of a system that works to combine data from a
variety of sources is the AffectAura, an emotional prosthetic
where data is collected from devices such as a microphone, Microsoft’s
Kinect, and a webcam to predict emotional states such as engagement,
valence, and arousal. Six participants were recorded over 4 days, and based on
the data collected, users were able to reconstruct stories about their days [8]. There are still multiple
challenges associated with creating accurate systems that could reveal
emotional aspects of a memory, especially when systems and devices are created
that go beyond only capturing photos. However, sensors will only become smaller
and smaller in the coming years, and as a society we have a great interest in
recording events for cultural reasons, so it is not unreasonable that one
point, most people will participate in the act of “lifelogging” and the
creation of HDM.




It is important to note though that physiological data or
even photos can only act as triggers to a memory, as there is no method to
actually capture an exact “memory” or a “thought.” Additionally, not only would
an HDM need to incorporate data from a variety of sensors and then correctly
corroborate and translate this data into meaningful information (a significant
challenge), but an entire lifetime of
memories would need be recorded for an accurate reflection and then a database
that is searchable would also be required.




If however we could accurately record and then disseminate
data that could compose a memory, does this documentation act as crutch?
Reminiscing and sharing personal stories with families and friends is a basic
human experience that acts as way to connect with others. If instead of
memorializing a lost relative through stories or laughing with friends over a
childhood experience, we could just push “play” on a device, how would that
change us? We already live in a society where any question can be answered with
a quick Google search on a phone requiring no discussion between two people, but
how would our interactions with others change if we could just “Google” how a
past experience played out or made us feel? (MIT professor Dr. Sherry Turkle has been studying how
technology impacts people for over 15 years and has written numerous articles
and books on the topics, and given this recent interesting TED talk on
the subject). Devices that use information from HDM are meant to help us with
the reminiscence process, but what if these devices are actually making us lose
that ability, or at the very least, fundamentally altering the memory process?




Additionally,
just as remembering is central to our existence, so is forgetting. Just because
at some point in the future we may be able to document a person’s entire
lifetime with a wealth of data, including physiological data, should we? Just
this year, researchers from the University of Basel discovered the musashi
protein, a protein that appears to inhibit molecules that stabilize synaptic
connections. These connections are important for the development of memories,
and based on this discovery, it appears that forgetting
is an active biological process. The biological processes behind
remembering and forgetting appear to work together, and forgetting is not just
a passive process [9]; we most likely forget for a
reason, even if more research is necessary to discover why.






References




1. Kelly L. The Information Retrieval
Challenge of Human Digital Memories. Proceedings of the 1st BCS IRSG Conference
on Future Directions in Information Access [Internet]. Swinton, UK, UK: British
Computer Society; 2007 [cited 2014 Aug 19]. p. 17–17. Available from:
http://dl.acm.org/citation.cfm?id=2227895.2227913

2.Bush V. As We May
Think. The Atlantic [Internet]. 1945
Jul [cited 2014 Aug 19]; Available from:
http://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/


3. Hodges S, Williams L, Berry E, Izadi S, Srinivasan J, Butler A, Smyth G, Kapur N, Wood E. SenseCam: A
Retrospective Memory Aid. UbiComp 2006 4206:
177-193.


4. Fairclough, S.H., and Gilleade, K. (2014). Capturing Human Digital Memories for Assisting Memory Recall In Advances in Physiological Computing, S.H. Fairclough, and K. Gilleade, eds.(Springer London), pp. 211-234.

5. Pauly-Takacs K,
Moulin CJA, Estlin EJ. SenseCam as a rehabilitation tool in a child with
anterograde amnesia. Mem Hove Engl.
2011, 19(7): 705–12.

6. Matthews R,
McDonald NJ, Hervieux P, Turner PJ, Steindorf MA. A wearable physiological
sensor suite for unobtrusive monitoring of physiological and cognitive state. Conf Proc IEEE Eng Med Biol Soc 2007, 2007:
5276–81.

7. López G, Custodio
V, Moreno JI. LOBIN: E-textile and wireless-sensor-network-based platform for
healthcare monitoring in future hospital environments. IEEE Trans Inf Technol Biomed Publ 2010, 14(6): 1446–58.

8. Mcduff D, Karlson
A, Kapoor A, Roseway A, Czerwinski M. AffectAura: An Intelligent System forEmotional Memory.

9. Hadziselimovic N,
Vukojevic V, Peter F, Milnik A, Fastenrath M, Fenyves BG, et al. Forgetting Is
Regulated via Musashi-Mediated Translational Control of the Arp2/3 Complex. Cell, 2014, 156(6): 1153–66.

Want to cite this post?




Strong, K. (2014). “Lifelogging” and neurophysiological computing: Will we forget how to forget? The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2014/08/lifelogging-and-neurophysiological.html

“Pass-thoughts” and non-deliberate physiological computing: When passwords and keyboards become obsolete

Imagine opening your email on your computer not by typing a number code, a password, or even by scanning a finger, but instead by simply thinking of a password. Physical keys and garage door openers could also become artifacts of the past once they are replaced with what could be referred to as pass-thoughts. Just last year, researchers at UC Berkley used EEG signals emitted from subjects as biomarker identifiers to allow access to a computer. The entire system – the headset, the Bluetooth device, and the computer – had an error rate of less than 1%.¹ While wearing EEG headsets to open our devices may seem futuristic, this type of scenario could become more prevalent in the future due to advances in physiological computing (PC). Physiological computing is a unique form of human computer interactions because the input device for a computer is any form of real-time physiological data, such as a heart-rate or EEG signal. This is in stark contrast to the peripheral devices that we are familiar with today, such as a keyboard, remote, or mouse.²



The field of physiological computing is still quite new, but research has suggested that different physiological computers require varying degrees of intentionality from the human user, and that the devices can be placed on a spectrum.³

Via physiologicalcomputing.net

On one end of the spectrum are technologies where users can deliberately interact with input devices based on voluntary muscle movement such as electrooculography (EOG) to direct the movement of a cursor (shown in 2 on the spectrum).⁴ In contrast, brain-computer-interfaces (BCI)­ such as the exoskeleton showcased at the recent first kick for the 2014 World Cup, bypass this step­ since BCIs are often developed for those with diminished movement capacities and disabilities. However, in both cases the general principle is the same: the interface is ultimately translating a neural signal that the user has specifically and deliberately directed to complete a task.⁵

Via cbsnews.com

Non-deliberate PC, on the other hand, bypasses any voluntary input, and instead involves a “biocybernetic” approach where spontaneous physiological changes, such as a heart rate or brain electrical signals are recorded via an electrocardiogram (EKG) or an electroencephalogram (EEG), respectively. These signals are then correlated to meaningful information, such as the case mentioned above where specific EEG signals act as identifying information to allow access to a computer. These types of technologies are able to associate recorded physiological changes with the motivational, cognitive or emotional state of the user. Once the interface determines the user’s emotional state, it can often adapt in an attempt to promote a specific type of positive mentality or negate a potentially hazardous emotional state. For example, if a computer calculates that the user is stressed, it can play soothing music or offer to help to diffuse the negative situation. The long-term recording of physiological data usually for learning purposes is referred to as ambulatory monitoring.⁶

Via thenextweb.com

Technologies that incorporate aspects of physiological computing, such as the recently released Kinect 2 from Microsoft, have recently become prevalent in consumer products. Using technology similar to that developed at MIT and referred to as Eulerian Video Modification,⁷ the camera on the Kinect detects small changes in skin color pigmentation and monitors heart rate optically (although pulse rate can be an indicator for an emotional state, at this time the Kinect 2 focuses on monitoring heart rates during physical activity, but does not correlate this data to an emotional state).

Portable, wireless sensors that are able to not only record, but also convert raw EEG signals into some form of meaningful information are currently available. EPOC by Emotiv and MindWave by NeuroSky have developed and currently sell wireless headsets that act as EEG sensors. Since certain EEG signals could be used as an indicators of a specific emotional state, such as frustration,⁸ the interface can label or adapt to a user in real-time. That said, while these EEG sensors give the impression that the user can execute commands with seemingly only the power of thought, these technologies are not yet able to comprehend intentions or mimic emotions (but, see recent data on AI recently passing Turing Test). For an interface to recognize intentions, first a system, similar to a dictionary, must be created so that the computer records the EEG data for a series of tasks that the interface will be able to recognize later. Not to mention, “intent” is still not clearly understood mechanistically through neuroscience.

Pertinent ethical issues include those related to ownership and privacy. Raw EEG or electrocardiogram (ECG) data is powerful information, especially when linked to changes in an emotional state. Emotiv will provide the raw EEG data from its users for an additional fee, but NeuroSky does not provide this information. Do we have any claim over our own (neuro-)physiological data once it leaves us? Even if raw EEG signals are worthless without an algorithm to decipher the meaning, the data still originated from only one, original source. Until it was pulled for ownership issues (NASA wanted to ensure that the data was no longer federal property), the EKG of Neil Armstrong’s heart as he took the first steps on the moon was to be auctioned off last year.⁹ But did NASA ever have a right to lay claim to this information, even if without an algorithm the EKG is seemingly meaningless? Or, does Neil Armstrong (or in this case, his family) have any right to claim ownership since NASA paid for and played a role in developing the technology that enabled this collection? These will be the types of questions that need to be addressed as more and more people continue to offer up their physiological data by using these types of technologies and popular commercial venues.

Via time.com

It seems inevitable that one day enough people will participate in the use of these EEG sensors and a massive database of neurological signals will begin to develop. Having a large dataset of neurological data that can potentially be correlated to disease states is already the goal of well established companies such as Lumosity ¹⁰ and BrainResource.¹¹ Additionally, the United States government recently launched PCORnet: The National Patient-Centered Clinical Network Project with the intention of building a national health-data system by combining data from 29 different health data networks.¹² The United Kingdom has met ethical conflicts with the introduction of a similar system, care.data,¹³ and the United States already has a history of alleged National Security Agency privacy violations, but government backed organizations are moving forward with the massive collection of medical records and perhaps one day, extensive physiological data. A precedent for having a dataset of extensive, personal information is the company 23andMe, which provided information based on DNA analysis. Nothing is protecting the users of 23andMe’s service from having their personal information sold,¹⁴ but the Genetic Information Nondiscrimination Act (GINA) passed in 2008 protects people from having their genetic information interfere with insurance policies and employment. This type of law does not exist for neurological data. Regulations and discussions should be taking place now before companies like Emotiv or NeuroSky have 5 years’ worth of data from their customers whose privacy is not protected in the slightest.




Already specific EEG signals can be used to characterize neurological disorders. With the collection of more data, we have the potential to be able to recognize and use specific signals as “brain signatures” for other neurological disorders or even tendencies toward certain behaviors (The well-established company Brainwave Science is a proponent of using EEG technology to test guilt or innocence). This ability, while incredibly powerful, has a high risk for abuse in terms of covert monitoring of individuals.¹⁵ Of course, if a patient has epilepsy, a discrete EEG sensor that has the power to be predictive for seizure activity could greatly increase the health, safety, and quality of life for these patients.¹⁶ Would it be appropriate to monitor a person who has been given a neurological diagnosis that has rendered them emotionally unstable if the EEG sensor could detect a very high or low state though? If that EEG sensor means that they are deemed stable enough for certain activities they were once denied, such as driving, does that make the constant monitoring worth what many would consider a violation of privacy?






References




(1) New Research: Computers That Can Identify You by Your Thoughts http://www.ischool.berkeley.edu/newsandevents/news/20130403brainwaveauthentication (accessed Jun 26, 2014).


(2) Fairclough, S. H. Fundamentals of Physiological Computing. Interact. Comput. 2009, 21, 133–145.


(3) Physiological Computing F.A.Q. Physiological Computing Blog. http://www.physiologicalcomputing.net/?page_id=227 (assessed on June 28, 2014).


(4) Allanson, J.; Fairclough, S. H. A Research Agenda for Physiological Computing. Interact. Comput. 2004, 16, 857–878.


(5) Allison, B. Z.; Wolpaw, E. W.; Wolpaw, J. R. Brain-Computer Interface Systems: Progress and Prospects. Expert Rev. Med. Devices 2007, 4, 463–474.


(6) Fairclough, S.H., and Gilleade, K. (2014). Meaningful Interaction with Physiological Computing. In Advances in Physiological Computing, S.H. Fairclough, and K. Gilleade, eds. (Springer London), pp. 1–16.


(7) Wu, H.-Y.; Rubinstein, M.; Shih, E.; Guttag, J.; Durand, F.; Freeman, W. T. Eulerian Video Magnification for Revealing Subtle Changes in the World. ACM Transactions on Graphics (Proc. SIGGRAPH 2012 2012, 31.


(8) Kapoor, A.; Burleson, W.; Picard, R. W. Automatic Prediction of Frustration. Int. J. Hum.-Comput. Stud. 2007, 65, 724–736.


(9) Pearlman, R. Z. Neil Armstrong’s “Heartbeat,” Apollo Joystick Pulled from Auction http://www.space.com/21228-neil-armstrong-apollo-artifacts-auction.html (accessed Jun 26, 2014).


(10) Sternberg, D. A.; Ballard, K.; Hardy, J. L.; Katz, B.; Doraiswamy, P. M.; Scanlon, M. The Largest Human Cognitive Performance Dataset Reveals Insights into the Effects of Lifestyle Factors and Aging. Front. Hum. Neurosci. 2013, 7.


(11) McRae, K.; Rekshan, W.; Williams, L. M.; Cooper, N.; Gross, J. J. Effects of Antidepressant Medication on Emotion Regulation in Depressed Patients: An iSPOT-D Report. J. Affect. Disord. 2014, 159, 127–132.


(12) Collins, F. S.; Hudson, K. L.; Briggs, J. P.; Lauer, M. S. PCORnet: Turning a Dream into Reality. J. Am. Med. Inform. Assoc. 2014, amiajnl–2014–002864.


(13) Callaway, E. UK Push to Open up Patients’ Data. Nature 2013, 502, 283–283.


(14) Seife, C. 23andMe Is Terrifying, but Not for the Reasons the FDA Thinks. Scientific American, Nov. 27, 2013. http://www.scientificamerican.com/article/23andme-is-terrifying-but-not-for-reasons-fda/ (accessed Jun 26, 2014).


(15) Deceiving the Law. Nat. Neurosci. 2008, 11, 1231–1231.


(16) Jouny, C. C.; Franaszczuk, P. J.; Bergey, G. K. Improving Early Seizure Detection. Epilepsy Behav. EB 2011, 22 Suppl 1, S44–48.






Want to cite this post?




Strong, K. (2014). “Pass-thoughts” and non-deliberate physiological computing: When passwords and keyboards become obsolete. The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2014/06/pass-thoughts-and-non-deliberate.html