Category: Innovation

Cognitive Augmentation of Knowledge Workers

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Simulation

The Flow Choice Architecture (FCA) is a biofeedback-based behavior modification tool that was designed for operation by human knowledge workers (KWs) before, during, and after the performance of work tasks.

FCA utilizes human-aware artificial intelligence (AI) that senses patterns in its operator’s bio-signals to derive their cognitive and affective states. The states are contextualized using relevant information about the operator, task, and work timeline. The contextualized operator states are then used to recommend effective nudges that orient the human KW towards the flow state.

This dissertation outlines the state-of-the-art on flow, neurofeedback, and AI then explains how FCA was designed, developed, and tested using a hybrid research methodology that combined design thinking and agile development. There is an in-depth discussion of the results from interviews, questionnaires, cognitive walkthroughs, heuristics evaluations, agent-based simulations, randomized controlled trials (RCT), and a longitudinal playtest.

The RCT featured a visual search task with and without a target in the stimulus patterns. The task was first tested by artificial KW agents in an agent-based simulation then later by human KWs. In the human-in-the-loop experiment, there were statistically significant outcomes in the hypotheses that were tested. Task demand was found to affect certain operator states and performance when the target was present in and absent from the stimulus patterns. Operator performance decreased when the target was absent.

There was evidence to confirm that nudges caused the operator to transition to desired states, and in some instances, the transitioning of operator state improved operator performance. Results from the ecological playtest demonstrated that the operator accepted and rejected nudges even though accepting nudges more frequently. However, the study was inconclusive about whether or not the nudges improved operator performance in the wild. FCA was highly usable from the perspective of the operators.

The latest FCA prototype was developed as a neurotechnology AI and deployed as a personalized recommender system with a gamified and conversational interface. The ethical issues surrounding this type of technology were discussed with the vision of commercializing a safe and responsible AI that proactively limits abuse from employers.

The dissertation concluded with an outlook on future research to be conducted along with the following three major contributions: (1) a contribution to flow research through bio-sensing and bio-feedback cognitive augmentation; (2) a contribution to human-centered AI design in the form of an integrated design thinking and agile development methodology; and (3), a contribution to cognitive economics in the form of a novel choice architecture.

The impact of these contributions was discussed within the broader context of knowledge work as an enabling service that is driving socio-economic development now and into the future.

Foraging for Flow in Knowledge Work

Troy W. Human-Centered Design, Innovation

The most valuable asset of a 21st-century institution, whether business or nonbusiness, will be its knowledge workers and their productivity (Drucker, 1999). Knowledge work includes “non-routine problem solving” that requires considerable amounts of concentration and creativity to perform. The complexity of knowledge work is often caused by interruptions, context-switching, and high workload conditions. Given this complexity among other demands in work settings, how do knowledge workers explore and exploit their tasks to optimize flow in their work? How does work productivity impact on their health and wellbeing? In this project, we conducted pilot interviews and an informal focus group with knowledge workers in order to: identify heuristics, biases, rituals and habits that are adopted by knowledge workers; compare patterns and trends among groups of knowledge workers; generate user personas and anti-personas that are relevant to productivity multiplier tools; and, recommend knowledge work practices that yield high quality outcomes, good health and wellbeing.

We found word clouds to be helpful for visualizing results from our qualitative study with a lot of rich information. The top themes for favorite activities before, during and after knowledge work were music, read, socialize, relax, exercise, videos, and games. There was a clear preference of coffee over water.

When changing their work state towards flow, respondents tend to use music for arousal. They prefer to use breathing exercises, affirmations, and chatting with colleagues for relaxation. For motivation, people think about rewards and accomplishments.

In order to optimize their work flow, KWs adopt strategies based on their tasks, resources and emotions. Respondents use external support mechanisms, and the awareness of internal signals to work at their best. When KWs experience flow, they report of productive outcomes, and increases in well-being and satisfaction, which motivates them to perform even better. When there is a lack of productivity, the KW becomes dissatisfied and well-being declines. For future work, we plan to refine the interview schedule of questions, and generate a survey instrument for approved research of the Flow Choice Architecture.

Policy-based Recommendations in a Flow Choice Architecture

Troy W. Artificial Intelligence, Bio-sensing, Cognitive Economics, Human-Centered Design, Innovation, Machine Learning, Robotics

Human agents that are involved in work activities employ their attention and decision making resources in exchange for experiences and rewards that may be physically, socially, economically and emotionally valued. When the human agent attains the flow state during work, their perception of physical effort diminishes, and their perception of emotional rewards increases.

In this work, we develop a computational model called the flow choice architecture, and simulate different approaches to policy-based utility-maximization using targeted choice recommendations. The results demonstrate a flow choice architecture that provides timely nudges to human agents in order to establish and maintain the flow state in a way that increases their productivity and sense of work accomplishment.

Opportunities for Case-Based Reasoning in Personal Flow and Productivity Management

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Machine Learning

Knowledge workers can benefit from tools to support them in performing deep, concentrated work. Research in biofeedback has shown success in training relaxation, but not in directly influencing task performance. One reason for this may be the difficulties users have in contextualizing biofeedback signals for different task situations. This presents an opportunity to leverage the strengths of case-based reasoning to select the feedback mechanism that will produce the best response.

This paper describes initial research into the Adaptive Choice Case-Based Reasoning (ACCBR) system, that learns from and interacts with a user to assist them in achieving greater concentration and productivity.

Eskridge, T. C., & Weekes, T. R. (2020, June). Opportunities for Case-Based Reasoning in Personal Flow and Productivity Management. In International Conference on Case-Based Reasoning (pp. 349-354). Springer, Cham.

Neurofeedback Humanoid to Support Deep Work

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Machine Learning, Robotics

High performance is desired in the workplace, even with swarms of robots on their way in the Fourth Industrial Revolution. Our research focuses on the population of knowledge workers, who are typically expected to sit in one space for extended periods while performing deep, intellectual and creative work. For those who work predominantly using computers, there is growing scope to augment task performance using artificial virtual agents. This trend is evident in the adoption of voice-based, and gesture-based applications that allow users to issue vocal or gestural commands while their hands are occupied on primary tasks. Even though multimodal interaction may yield more productivity than solely mouse and keyboard interactions, it may still impose a significant cognitive load on the user.

We propose the modeling of a smart motivational humanoid assistant that is personalized to interact with human users without explicit commands, and instead via wireless sensors that can perceive the operator’s brain activity. The humanoid engages with the human using effective nudges through neurofeedback.

Weekes, T. R., & Eskridge, T. C. (2020, May). A neurofeedback-driven humanoid to support deep work. In Proceedings of the 33rd Florida Conference on Recent Advances in Robotics (pp. 14-16).

Nudging into Flow: Optimizing Productivity with a Choice Architecture

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Machine Learning, Visualization

When humans are performing well, the information they attend to and action choices they make naturally maximize productivity and enable effective completion of tasks. The agent’s dynamic tradeoffs under constraints and uncertainty of the work environment, and their own cognitive state constitute rational economic behavior.

We propose a theoretical framework for a choice architecture that can be used to efficiently transform an agent’s lagging work experience into productive work outcomes. The choice architecture operates within the interacting contexts of the human emotional and behavioral states and the work environment, and is the source of feelings of success, stress and the emergence of other emotions. Changes in these contexts will elicit changes in the response of the choice architecture, and corresponding changes in human behavior and productivity.

Our framework is based on the hypothesis that if the emotional and behavioral states that are conducive to effective work can be identified, then well-timed “nudges” in the work environment that alter human emotions or behavior in predictable ways without eliminating options or significantly modifying economic incentives can optimize the utility of the choice architecture.

By doing so, the choice architecture may guide the human decision maker towards states of flow as defined by Csikszentmihalyi. Flow is a mental state related to a state of high motivation that is characterized by the dynamic equilibrium of challenge and skill in a task with clear goals and immediate feedback. With the goal of maintaining flow for as long as necessary, our nudges personalize default settings, propose key anchors, incentivize the agent, and provide triggers that activate and sustain flow. When an agent is operating in flow and the emotional and behavioral contexts are aligned with the work environment, the agent can be deeply involved with work tasks while making the work itself effortless and enjoyable.

Weekes, Troy, and Eskridge, Thomas C. Nudging into Flow: Optimizing Productivity with a Choice Architecture. In Cognitive Economics Workshop 2019

Naturalized Artificial Intelligence for Language Amplification

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Machine Learning

Knowledge elicitation can be mediated by technology interfaces (Boy, 1997). The process of creativity relies on ideating and deciding among choices, which gives birth to new conditions. Design thinking provides a streamlined framework under which the creative process can be continuously practised to optimize choices into marketable innovation. Good ideas are intellectual assets, and in today’s world of business, the design of new products, features, processes or brand names can be monetized through patents, copyrights and trademarks. But how are creative ideas generated, converted into practical knowledge, and managed to derive customer value and business intelligence?

Our work in this analysis has been built upon previous research conducted in the field of participatory design and group decision making theory. Traditionally, focus groups and brainstorming sessions are used to elicit ideas. The main challenges of these approaches and their variants are predicated on the complex social dynamics that are inherent to group decision making. Both focus groups and brainstorms are limited by the number of participants that can contribute within a bounded timeframe and physical space. Both are synchronous, time-consuming methods that do not scale and are relatively difficult to document.

Our target users are design team members or creative individuals who are primarily responsible for the ideation process in research and development projects whether in corporations, governments or less formal organizations. In this Uber-competitive landscape, agile methods of knowledge elicitation that satisfy the needs of end users and stakeholders are imperative to generate innovation.

Historically, design teams rely on feedback from disparate user groups using research methods to elicit ideas and draw conclusions for design. However, some design ideas plagued by outdated processes and infected by unavoidable politics. Consequently, these phenomena lead to inefficient design sessions as well as products that amplify the biases of the more dominant and extroverted participants from design sessions.

In our previous research, two realistic scenarios were conducted that generated useful results. The participants were members of design teams, including the facilitators and participants. Similar to the target users, the participants were familiar with web applications, email, computers and mobile devices. The exploratory research leveraged use cases with different levels of interaction and different group compositions (homogeneous vs. heterogeneous).

The first scenario was conducted using the traditional GEM, and involved a research team of graduate students who sought to redesign an optimized organizational structure for delivering university services. The second scenario involved a NASA Hackathon design team that was seeking to answer the design question around enhancing space crew health that could be sustained from outer space and be controlled through remote communications on earth. The design team comprised a multidisciplinary team of human-centered designer, mechanical engineer and computer engineer.

The new solution was modeled from the successes and failures of e-GEM, which sought to enhance the traditional GEM. From the outset, the long-term goal of this research project is to lay a foundation for an operational “Usability and UX methods bank” that can be commercialized for industrial applications. However, that vision has been pivoted in order to scale within a better-primed market.

The e-GEM application was web-based, and so is the new solution. However, the problem statement was revised based on evidence that emerged from the work done on the previous e-GEM as well recent accounts from a targeted sequence of interviews and meetings. How can innovators generate ideas leveraging smart voices inside and outside their head?

The proposed solution will help teams and individuals to add value to their innovations through natural conversations with artificial agents. The new solution, called NAILA, exploits Artificial Intelligence in the form of natural language understanding and generation. It was designed to fully abstract e-GEM, and subsume other brainstorming and brainwriting ideation predecessors.

Design Thinking was utilized as an empathetic human-centered framework to structure the creative process by mitigating risks of creating a misaligned prototype. Our core research driver was that of empathy. Research methods used in the process included Questions, Options, Criteria to make decisions on the following design questions after careful consideration of the practical options using a set of relevant criteria.

User Personas were reviewed and tweaked for the key end users involved in the GEM i.e. the facilitator and the participant. The Scenario-based Design approach was used to test the suitability of the user personas and to construct practical use cases and scenarios that informed the development of the storyboard and wireframes that were used to mockup the system design in order to facilitate rapid prototyping and generate a visual representation.

In conclusion, our new solution will generate an agent-based participatory design methodology and a modular service-oriented web platform capable of scale. The competitive advantage of the solution lies in the naturalized agent-based emulation, which makes the end user immune to spatiotemporal discontinuity. The solution aims to allow end users to work with teams of conversational AIs.

AVA, an Affective Virtual Assistant for the Workplace

Troy W. Artificial Intelligence, Bio-sensing, Human-Centered Design, Innovation, Machine Learning

No matter how much training, skill, motivation, or experience, humans are compromised by reduced attention or impaired alertness. Depression, stress, fatigue, and burnout are all consequences of unhealthy, and unsafe work conditions. Our research focuses on the population of desk-bound employees, who are typically expected to sit in one space for periods of two hours, and more while performing a specific activity or sequence of tasks. For those who work predominantly using computers, there is growing scope to augment task performance using artificial virtual agents.

We propose AVA, an interactive smart assistant, that is designed to playfully augment user performance in the workplace without the user having to say anything. Universally, humans express emotions and feelings through their faces (Ekman, 1997). Our smart assistant observes a human user, infers emotional state, and reasons to effect relevant feedback. The assistant embodies an affective model of the subject that is used for advising more effective actions, behaviors, and work practices.

A system of productions was formulated from expected features that were found in the video streams, which represent the emotional state. The Markov Decision Process (MDP) is defined as a tuple (B, A, τ, r, γ) modeled in discrete-time and that the agent was unable to directly observe the underlying human emotional state (only expressions of emotion). Therefore, the model was required to maintain a probability distribution over the set of possible states, based on a set of observations, observation probabilities, and the underlying MDP.

We implemented a real-time facial expression recognition algorithm. The live video streams are processed as discrete frames with 34 AUs per face. AVA detects and extracts facial features according to the Facial Action Coding System (FACS). Classified emotions are used to generate intelligent actions, advice, and prompts. AVA integrates facial expression with objective EEG & eye-tracking data. In this research project, we simulated and tested a human-in-the-loop model within the context of a real-world joint-cognitive system.

We plan to integrate our facial expression model with objective measures of physiological arousal, brain activity, and eye-tracking to generate a comprehensive model of behavior in the office workplace.

Mass Production of Radiation Shields In Situ

Troy W. Human Spaceflight, Innovation

The Radiation Problem

Acute and chronic exposure to radiation can increase risks to astronaut health and well being.

 


 

 

 

The Solution

A cost-effective prewalk robotic mission that autonomously fabricates radiation shield components for generating localized magnetospheres in order to deflect cosmic radiation and enhance well-being of astronauts and biological habitability on Mars.

 


 

 

 

How We Built It

Using design thinking and concept maps, the team collaboratively ideated the solution to meet the end goal of reducing radiation impacts on astronauts. Feasibility and Risk Analysis of fabricating the radiation shield components was utilized from a human-system integration perspective. The tools and techniques used include the following:

 

  • Design-thinking
  • Concept-maps
  • Idea-visualization
  • Analytical-hierarchical-processing
  • Systems-thinking
  • Human-system-integration

 

Challenges Encountered

The breadth and technical complexity of the system. For example, the goal setting tangible interactive system (TIS) integrated with Augmented Reality, which is illustrated below.

 


 

 

 

Accomplishments that we’re proud of

Ideation of an end-to-end system with a process that was validated by the Buzz Aldrin VR story.

 

 

Lessons Learned

The journey to Mars will take extraordinary innovation and engineering.

 

 

What’s next for Mass Production of Radiation Shields In Situ

Derivation of high level requirements that can be tested on earth first.

 

 

Embraer: A Legacy of Innovation in Aviation

Troy W. Aviation, Innovation

On September 22, 2017, I walked into the Embraer facilities at the Melbourne plant. Embraer is a global company headquartered in Brazil with highly integrated businesses in commercial, executive and military aviation (Embraer 2017). Embraer designs, develops, manufactures and markets aircraft, and provides customer support and services. The company implements human-integrated systems, programmable automation and Kaizen to ensure the agile delivery of high quality end-to-end solutions that are customized to meet the distinct needs and preferences of their downstream customers.

Embraer was founded in 1969 by Ozires Silva who later divested executive control to dynamic entrepreneurs, Mauricio Botelho and Frederico Fleury Curado (Embraer 2017). Following its success in commercial aviation, Embraer began exploring the executive aviation market in 2000 with its Legacy jet (Embraer 2017). Embraer followed up with its popular Phenom 100 and the more robust Phenom 300, which was certified in late 2009 (Robb Report 2010). By 2017, Embraer had developed a full line of business aircraft ranging from the entry-level Phenom 100 executive jet to ultra-large cabin Lineage 1000E business aircraft.

Embraer is a complex socio-technical system, which organizes its employees into cells and business units of its corporate value chain. In 2015, the company employed 23,050 people (Statista 2017). This growth in employment represented approximately 4% since 2010. The company has adopted Kaizen as a business philosophy for continuous improvement in its business units. Under the Embraer Enterprise Excellence Program (P3E), the company functions as a lean enterprise with the goals of reducing waste (muda) of resources, maximizing productivity, and, ultimately, the endgame of total customer satisfaction.

In August 2014, Melbourne International Airport and Embraer signed a multi-year lease for a new 250,000-square-foot, three-building facility with the goal of creating 600 jobs by 2016 (Barth 2014a). Additionally in 2014, Embraer opened an engineering Center of Excellence in Melbourne with the expectation of creating more 200 jobs (Barth 2014b). One year later, Embraer expanded by building a new manufacturing facility in Titusville for the production of seating solutions creating 150 new jobs (Richardson 2015).

Embraer’s rapid growth in Florida may be enabled by (1) its relatively close proximity to Brazil; (2) the ease of logistics between Embraer’s production facilities and parts distribution centers; (3) the direct access to the Melbourne International Airport and I-94; and (4) the nearby talent markets provided by the Florida Institute of Technology, Embry Riddle Aeronautical University and the University of Central Florida.

The Melbourne plant houses aircraft assembly, painting and delivery of the Embraer’s newest line of Legacy jets including all Phenoms and some Legacy 450/500s (Barth 2014a; Trautvetter 2017). On September 27, 2017, Embraer delivered its first midsize Legacy 500 that was assembled at its Melbourne plant in Florida (Trautvetter 2017). Besides the first 2017 Legacy handover, the Florida facility has assembled and delivered two Legacy 450s and nearly 260 Phenoms (Trautvetter 2017). I was privileged to see the newly painted Legacy 500 in the flight-prep area the week before its delivery.

The Melbourne production facility (gemba) has a systematic layout with structured assembly lines for the different aircraft models. The plant layout features a range of ergonomic artifacts such as proper signage, clearly marked areas, emergency systems, a smart equipment “vending machine”, and a sorted registry for aircraft components (and possibly line replacement units).

The aircraft (gembutsu) begin from shells and major components that are shipped from Brazil and other suppliers from across the world. The procurement and logistics involved is complicated but streamlined with tremendous efficiency through Embraer’s business processes. After unpacking from the cargo bay, technicians go through step-by-step assembly so that the aircraft flows from one stage of manufacturing to another. The technicians generally work in small teams and business units that are coordinated by supervisors and managers.

Each aircraft bears a conspicuous label with a production number and other pertinent identification facts. Technicians have specialized workstations situated around the assembly lines and in close proximity to other specialists in the assembly team. Throughout the assembly line, the airframe is completed, the avionics systems are integrated along with the powerplant and other subsystems, and the interiors are fitted. At various points in the engineering process, meticulous inspections are conducted according to internal and external certification standards and regulations.

At the end of the assembly line, the aircraft are taken to a neighboring facility where they are painted. Following painting, more technical inspections are conducted in the flight-prep area. The aircraft are flight tested by experienced test pilots and engineers before delivery to their buyers. This simplified process model highlights the high degree of teamwork and human-technology integration that is required in order for Embraer to fulfill its business model. The process is very risky but Embraer utilizes process standardization techniques and risk mitigation principles to guarantee the production of safe and reliable products.

During the tour at the Embraer Engineering and Technology Center, I engaged with engineers and designers who expressed satisfaction with their jobs during the conversations that we shared. Embraer prides itself on diversity and corporate culture. The company leverages diversity as a platform for innovation, which is a hallmark of the Embraer organization. The company has a clear vision that is shared by its employees on all levels. The employees demonstrate this vision through their perseverance towards excellence, their unwavering customer service, and their boldness in creativity and sustainability.

The company motivates its employees through unique Employee Appreciation Awards that are augmented with a Global Trainee Program, an Engineering Specialization Program, a Designer Program, a Young Apprentice Program, a Learning Program For People With Disabilities, and a Good Ideas program (Embraer 2017). These programs highlight different aspects of Embraer’s people-centered approach to business. The Good Ideas program, in particular, is a gamification concept whereby employees generate valuable ideas and are meaningfully rewarded through “special” parking lots, peer recognition on visual message boards, job enrichment and other valuable perks.

In addition to the programs outlined above, Embraer conducts intensive workshops and features a corporate education model to offer continued learning as a way for its professionals to build on their expertise and support corporate growth (Embraer 2017). The training is gamified in the form of abstracted levels and challenges. For example, criteria of excellence are clearly defined and reinforced to continuously induce the gradual improvement in skills that lead to better performance. Employees earn bronze, silver and gold credentials along their academic tracks as they improve and contribute to Embraer’s process and management maturity. If that wasn’t enough to motivate employees, Embraer provides cafeterias and massages on-site.

It is not only great to work at Embraer because of the beneficial programs that it offers; the company has an impeccable reputation in the aviation industry. In the 2017 Product Support Survey by Aviation International News, magazine readers rated Embraer as the top aircraft manufacturer with the highest combined average ratings of newer and older business aircraft (Thurber 2017). Such brand recognition can motivate employees when they tell others about where they work, and can instill self-esteem and self-efficacy in the work they perform.

From the perspective of corporate social responsibility and leaving the world a better place, Embraer gives back to the community. The Embraer Foundation raises funds for social and environmental projects, and inspires employees to volunteer in local initiatives with social organizations (Embraer 2016). One jet at a time, Embraer makes a sustainable impact in the lives of its employees, customers and the environment.

During a surprise ceremony where an aircraft, which featured the name of the company’s founder, was unveiled, Mr. Silva stated “I transfer to each of you this honor for what you are doing now. You deserve this… When we put this plane as a symbol of what we are going to do for the future, I would like you to think big in that direction and continue the work you are doing with confidence.” Such an empathetic statement from the head of company gives a glimpse into the heart of Embraer. Embraer does not only make great technology in the form of aircraft; Embraer also makes great people.

References

Barth, Cindy. (2014a). Embraer bringing 600 jobs to Melbourne airport. Orlando Business Journal. https://www.bizjournals.com/orlando/news/2014/08/28/embraer-bringing-600-jobs-to-melbourne-airport.html

Barth, Cindy. (2014b). Embraer’s $24.2M Center of Excellence opens in Melbourne. Orlando Business Journal. https://www.bizjournals.com/orlando/news/2014/09/08/embraers-24-2m-center-of-excellence-opens-in.html

Embraer. (2016). Embraer Establishes Foundation in the United States. http://embraerfoundation.org/embraer-establishes-foundation-in-the-united-states/

Embraer. (2017). Award-winning History With Executive Jets. http://www.embraerexecutivejets.com/en-us/about-embraer/pages/history.aspx

Imai, Masaaki. (1986). Kaizen: The Key To Japan’s Competitive Success 1st Edition. McGraw-Hill Education.

Richardson, Matthew. (2015). Aircraft manufacturer expands in Brevard, creates dozens of jobs. https://www.bizjournals.com/orlando/news/2015/06/15/aircraft-manufacturer-expands-in-brevard-creates.html

Robb Report. (2010). Embraer Phenom 300. Robb Report. http://robbreport.com/motors/aviation/embraer-phenom-12862/

Statista. (2017). Number of Embraer employees from FY 2010 to FY 2015. https://www.statista.com/statistics/281360/number-of-employees-at-embraer/

Thurber, Matt. (2017). Product Support Survey Part 1: Aircraft. Aviation International News. https://www.ainonline.com/product-support-survey-aircraft-2017

Trautvetter, Chad. (2017). Embraer Delivers First Florida-assembled Legacy 500. Aviation International News. https://www.ainonline.com/aviation-news/business-aviation/2017-09-27/embraer-delivers-first-florida-assembled-legacy-500