Category: Bio-sensing

Cognitive Augmentation of Knowledge Workers

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Simulation

The Flow Choice Architecture (FCA) is a biofeedback-based behavior modification tool that was designed for operation by human knowledge workers (KWs) before, during, and after the performance of work tasks.

FCA utilizes human-aware artificial intelligence (AI) that senses patterns in its operator’s bio-signals to derive their cognitive and affective states. The states are contextualized using relevant information about the operator, task, and work timeline. The contextualized operator states are then used to recommend effective nudges that orient the human KW towards the flow state.

This dissertation outlines the state-of-the-art on flow, neurofeedback, and AI then explains how FCA was designed, developed, and tested using a hybrid research methodology that combined design thinking and agile development. There is an in-depth discussion of the results from interviews, questionnaires, cognitive walkthroughs, heuristics evaluations, agent-based simulations, randomized controlled trials (RCT), and a longitudinal playtest.

The RCT featured a visual search task with and without a target in the stimulus patterns. The task was first tested by artificial KW agents in an agent-based simulation then later by human KWs. In the human-in-the-loop experiment, there were statistically significant outcomes in the hypotheses that were tested. Task demand was found to affect certain operator states and performance when the target was present in and absent from the stimulus patterns. Operator performance decreased when the target was absent.

There was evidence to confirm that nudges caused the operator to transition to desired states, and in some instances, the transitioning of operator state improved operator performance. Results from the ecological playtest demonstrated that the operator accepted and rejected nudges even though accepting nudges more frequently. However, the study was inconclusive about whether or not the nudges improved operator performance in the wild. FCA was highly usable from the perspective of the operators.

The latest FCA prototype was developed as a neurotechnology AI and deployed as a personalized recommender system with a gamified and conversational interface. The ethical issues surrounding this type of technology were discussed with the vision of commercializing a safe and responsible AI that proactively limits abuse from employers.

The dissertation concluded with an outlook on future research to be conducted along with the following three major contributions: (1) a contribution to flow research through bio-sensing and bio-feedback cognitive augmentation; (2) a contribution to human-centered AI design in the form of an integrated design thinking and agile development methodology; and (3), a contribution to cognitive economics in the form of a novel choice architecture.

The impact of these contributions was discussed within the broader context of knowledge work as an enabling service that is driving socio-economic development now and into the future.

Policy-based Recommendations in a Flow Choice Architecture

Troy W. Artificial Intelligence, Bio-sensing, Cognitive Economics, Human-Centered Design, Innovation, Machine Learning, Robotics

Human agents that are involved in work activities employ their attention and decision making resources in exchange for experiences and rewards that may be physically, socially, economically and emotionally valued. When the human agent attains the flow state during work, their perception of physical effort diminishes, and their perception of emotional rewards increases.

In this work, we develop a computational model called the flow choice architecture, and simulate different approaches to policy-based utility-maximization using targeted choice recommendations. The results demonstrate a flow choice architecture that provides timely nudges to human agents in order to establish and maintain the flow state in a way that increases their productivity and sense of work accomplishment.

Opportunities for Case-Based Reasoning in Personal Flow and Productivity Management

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Machine Learning

Knowledge workers can benefit from tools to support them in performing deep, concentrated work. Research in biofeedback has shown success in training relaxation, but not in directly influencing task performance. One reason for this may be the difficulties users have in contextualizing biofeedback signals for different task situations. This presents an opportunity to leverage the strengths of case-based reasoning to select the feedback mechanism that will produce the best response.

This paper describes initial research into the Adaptive Choice Case-Based Reasoning (ACCBR) system, that learns from and interacts with a user to assist them in achieving greater concentration and productivity.

Eskridge, T. C., & Weekes, T. R. (2020, June). Opportunities for Case-Based Reasoning in Personal Flow and Productivity Management. In International Conference on Case-Based Reasoning (pp. 349-354). Springer, Cham.

Neurofeedback Humanoid to Support Deep Work

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Machine Learning, Robotics

High performance is desired in the workplace, even with swarms of robots on their way in the Fourth Industrial Revolution. Our research focuses on the population of knowledge workers, who are typically expected to sit in one space for extended periods while performing deep, intellectual and creative work. For those who work predominantly using computers, there is growing scope to augment task performance using artificial virtual agents. This trend is evident in the adoption of voice-based, and gesture-based applications that allow users to issue vocal or gestural commands while their hands are occupied on primary tasks. Even though multimodal interaction may yield more productivity than solely mouse and keyboard interactions, it may still impose a significant cognitive load on the user.

We propose the modeling of a smart motivational humanoid assistant that is personalized to interact with human users without explicit commands, and instead via wireless sensors that can perceive the operator’s brain activity. The humanoid engages with the human using effective nudges through neurofeedback.

Weekes, T. R., & Eskridge, T. C. (2020, May). A neurofeedback-driven humanoid to support deep work. In Proceedings of the 33rd Florida Conference on Recent Advances in Robotics (pp. 14-16).

Nudging into Flow: Optimizing Productivity with a Choice Architecture

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Machine Learning, Visualization

When humans are performing well, the information they attend to and action choices they make naturally maximize productivity and enable effective completion of tasks. The agent’s dynamic tradeoffs under constraints and uncertainty of the work environment, and their own cognitive state constitute rational economic behavior.

We propose a theoretical framework for a choice architecture that can be used to efficiently transform an agent’s lagging work experience into productive work outcomes. The choice architecture operates within the interacting contexts of the human emotional and behavioral states and the work environment, and is the source of feelings of success, stress and the emergence of other emotions. Changes in these contexts will elicit changes in the response of the choice architecture, and corresponding changes in human behavior and productivity.

Our framework is based on the hypothesis that if the emotional and behavioral states that are conducive to effective work can be identified, then well-timed “nudges” in the work environment that alter human emotions or behavior in predictable ways without eliminating options or significantly modifying economic incentives can optimize the utility of the choice architecture.

By doing so, the choice architecture may guide the human decision maker towards states of flow as defined by Csikszentmihalyi. Flow is a mental state related to a state of high motivation that is characterized by the dynamic equilibrium of challenge and skill in a task with clear goals and immediate feedback. With the goal of maintaining flow for as long as necessary, our nudges personalize default settings, propose key anchors, incentivize the agent, and provide triggers that activate and sustain flow. When an agent is operating in flow and the emotional and behavioral contexts are aligned with the work environment, the agent can be deeply involved with work tasks while making the work itself effortless and enjoyable.

Weekes, Troy, and Eskridge, Thomas C. Nudging into Flow: Optimizing Productivity with a Choice Architecture. In Cognitive Economics Workshop 2019

Naturalized Artificial Intelligence for Language Amplification

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Machine Learning

Knowledge elicitation can be mediated by technology interfaces (Boy, 1997). The process of creativity relies on ideating and deciding among choices, which gives birth to new conditions. Design thinking provides a streamlined framework under which the creative process can be continuously practised to optimize choices into marketable innovation. Good ideas are intellectual assets, and in today’s world of business, the design of new products, features, processes or brand names can be monetized through patents, copyrights and trademarks. But how are creative ideas generated, converted into practical knowledge, and managed to derive customer value and business intelligence?

Our work in this analysis has been built upon previous research conducted in the field of participatory design and group decision making theory. Traditionally, focus groups and brainstorming sessions are used to elicit ideas. The main challenges of these approaches and their variants are predicated on the complex social dynamics that are inherent to group decision making. Both focus groups and brainstorms are limited by the number of participants that can contribute within a bounded timeframe and physical space. Both are synchronous, time-consuming methods that do not scale and are relatively difficult to document.

Our target users are design team members or creative individuals who are primarily responsible for the ideation process in research and development projects whether in corporations, governments or less formal organizations. In this Uber-competitive landscape, agile methods of knowledge elicitation that satisfy the needs of end users and stakeholders are imperative to generate innovation.

Historically, design teams rely on feedback from disparate user groups using research methods to elicit ideas and draw conclusions for design. However, some design ideas plagued by outdated processes and infected by unavoidable politics. Consequently, these phenomena lead to inefficient design sessions as well as products that amplify the biases of the more dominant and extroverted participants from design sessions.

In our previous research, two realistic scenarios were conducted that generated useful results. The participants were members of design teams, including the facilitators and participants. Similar to the target users, the participants were familiar with web applications, email, computers and mobile devices. The exploratory research leveraged use cases with different levels of interaction and different group compositions (homogeneous vs. heterogeneous).

The first scenario was conducted using the traditional GEM, and involved a research team of graduate students who sought to redesign an optimized organizational structure for delivering university services. The second scenario involved a NASA Hackathon design team that was seeking to answer the design question around enhancing space crew health that could be sustained from outer space and be controlled through remote communications on earth. The design team comprised a multidisciplinary team of human-centered designer, mechanical engineer and computer engineer.

The new solution was modeled from the successes and failures of e-GEM, which sought to enhance the traditional GEM. From the outset, the long-term goal of this research project is to lay a foundation for an operational “Usability and UX methods bank” that can be commercialized for industrial applications. However, that vision has been pivoted in order to scale within a better-primed market.

The e-GEM application was web-based, and so is the new solution. However, the problem statement was revised based on evidence that emerged from the work done on the previous e-GEM as well recent accounts from a targeted sequence of interviews and meetings. How can innovators generate ideas leveraging smart voices inside and outside their head?

The proposed solution will help teams and individuals to add value to their innovations through natural conversations with artificial agents. The new solution, called NAILA, exploits Artificial Intelligence in the form of natural language understanding and generation. It was designed to fully abstract e-GEM, and subsume other brainstorming and brainwriting ideation predecessors.

Design Thinking was utilized as an empathetic human-centered framework to structure the creative process by mitigating risks of creating a misaligned prototype. Our core research driver was that of empathy. Research methods used in the process included Questions, Options, Criteria to make decisions on the following design questions after careful consideration of the practical options using a set of relevant criteria.

User Personas were reviewed and tweaked for the key end users involved in the GEM i.e. the facilitator and the participant. The Scenario-based Design approach was used to test the suitability of the user personas and to construct practical use cases and scenarios that informed the development of the storyboard and wireframes that were used to mockup the system design in order to facilitate rapid prototyping and generate a visual representation.

In conclusion, our new solution will generate an agent-based participatory design methodology and a modular service-oriented web platform capable of scale. The competitive advantage of the solution lies in the naturalized agent-based emulation, which makes the end user immune to spatiotemporal discontinuity. The solution aims to allow end users to work with teams of conversational AIs.

AVA, an Affective Virtual Assistant for the Workplace

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Machine Learning

No matter how much training, skill, motivation, or experience, humans are compromised by reduced attention or impaired alertness. Depression, stress, fatigue, and burnout are all consequences of unhealthy, and unsafe work conditions. Our research focuses on the population of desk-bound employees, who are typically expected to sit in one space for periods of two hours, and more while performing a specific activity or sequence of tasks. For those who work predominantly using computers, there is growing scope to augment task performance using artificial virtual agents.

We propose AVA, an interactive smart assistant, that is designed to playfully augment user performance in the workplace without the user having to say anything. Universally, humans express emotions and feelings through their faces (Ekman, 1997). Our smart assistant observes a human user, infers emotional state, and reasons to effect relevant feedback. The assistant embodies an affective model of the subject that is used for advising more effective actions, behaviors, and work practices.

A system of productions was formulated from expected features that were found in the video streams, which represent the emotional state. The Markov Decision Process (MDP) is defined as a tuple (B, A, τ, r, γ) modeled in discrete-time and that the agent was unable to directly observe the underlying human emotional state (only expressions of emotion). Therefore, the model was required to maintain a probability distribution over the set of possible states, based on a set of observations, observation probabilities, and the underlying MDP.

We implemented a real-time facial expression recognition algorithm. The live video streams are processed as discrete frames with 34 AUs per face. AVA detects and extracts facial features according to the Facial Action Coding System (FACS). Classified emotions are used to generate intelligent actions, advice, and prompts. AVA integrates facial expression with objective EEG & eye-tracking data. In this research project, we simulated and tested a human-in-the-loop model within the context of a real-world joint-cognitive system.

We plan to integrate our facial expression model with objective measures of physiological arousal, brain activity, and eye-tracking to generate a comprehensive model of behavior in the office workplace.