|
AI4PI - Artificial Intelligence for Performance Improvement is an ISPI platform to encourage an evidence-based dialogue and practical applications of AI to performance.
|
AI4PI Framework
The initial AI4PI Framework describes a framework to design and implement efficient, effective, human-centered AI4PI Solutions.
It consists of four principles and six process steps.
Four AI4PI Principles:
AI4PI Principle #1: Ensure Human Oversight of AI4PI Solutions
AI4PI Principle #2: Enforce Data Integrity and Privacy in AI4PI Solutions
AI4PI Principle #3: Evaluate and minimize AI-Induced Biases
AI4PI Principle #4: Establish AI4PI Solution’s Governance and Risk Management Strategies
Six AI4PI Process Steps:
AI4PI Process Step #1: Determine Needs, Organization’s Readiness and Potential Applications (Discovery)
AI4PI Process Step #2: Define Efficient and Effective Human-Centered AI4PI Solutions
AI4PI Process Step #3: Design and Develop AI - Human Collaboration
AI4PI Process Step #4: Document AI Accuracy and Trustworthiness
AI4PI Process Step #5: Deploy and Audit the AI4PI solution
AI4PI Process Step #6: Demonstrate Outcome and Overall Impact
Human-Centered AI for Performance Improvement Principles
AI4PI Principle #1: Ensure Human Oversight of AI4PI Solutions
Key concepts: Transparency, explainability, demystification, human-AI collaboration, performance excellence, ability to interrupt
Ensuring that AI decisions are transparent and understandable to all stakeholders is crucial. This involves demystifying AI processes and outcomes so that everyone involved can comprehend and trust the AI's recommendations. Additionally, the ability to interrupt AI processes ensures that human oversight can be maintained at all times.
So clients:
- Can confidently rely on AI4PI solutions, knowing that AI decisions are transparent and understandable.
- Feel empowered to make informed decisions with the understanding that AI4PI solutions are demystified and accountable.
- Can trust the AI recommendations with transparent, evidence-based decision-making.
Examples:
- An AI-powered customer service tool that provides clear explanations for its recommendations, allowing service agents to understand and validate the AI's decisions before communicating with customers.
- A manufacturing AI system that collaborates with human workers to optimize production processes, providing real-time insights and explanations for its suggestions.
- An AI4PI solution in a healthcare setting that works alongside medical professionals to diagnose patients, offering transparent reasoning and references for its diagnostic recommendations and allowing doctors to make well-informed decisions.
AI4PI Principle #2: Enforce Data Integrity, Protection and Privacy
Key concepts: data protection, confidentiality, internal versus public servers, data provenance, evidence-based recommendation, data preparation, data privacy, data security
Protecting the source of data and their confidentiality is paramount, from ensuring the traceability of AI decision-making to protecting personal and organizations’ confidential data. Users of the AI4PI solution should be able to obtain the sources and reference data that inform AI recommendations. Neither organizational data nor personal data should not be shared with public Large Language Models or publicly accessible AI tools but should remain protected within the organization’s firewall.
So clients:
- Can confidently rely on AI4PI solutions, knowing the origin of the data and logic behind AI decisions.
- Are assured that their personal and company confidential data are secure and not shared with the public internet.
- Feel empowered to make informed decisions with the understanding that AI4PI solutions are transparent and accountable.
- Can trust the AI recommendations with transparent, evidence-based decision making
Examples:
- An HIPAA-compliant, AI-powered health diagnostic tool that provides clinicians with the actual source documents used for the diagnostic and recommendations.
- A financial AI-application that tracks and reports the actual data used for its recommendations for investments.
- An AI4PI system operates within the organization’s network firewall and without any access to the public internet. It uses company data to optimize operations while enforcing strict data confidentiality protocols.
AI4PI Principle #3: Evaluate and minimize AI-Induced Biases
Key concepts: potential ethical biases, diversity, non-discrimination, fairness, equity, inaccuracy, bias mitigation
AI4PI solutions should be proactively analyzed to mitigate potential biases that could lead to skewed results, thereby preventing AI from perpetuating societal injustices, generating inaccuracies or engaging in potentially discriminatory behaviors.
So clients:
- Are aware and able to recognize the various types of potential biases (Shestakova, 2023).
- Can depend on AI to enhance workplace productivity without introducing or perpetuating bias.
- Have a reasonable confidence that AI-generated insights and recommendations are derived from balanced and fair algorithms.
- Benefit from AI tools that promote equality and objectivity in performance-related assessments.
Examples:
- An AI performance monitoring tool is regularly checked to ensure it does not unfairly evaluate employees based on age, gender, or cultural background, maintaining a focus on individual merits and contributions.
- A AI4PI learning platform used for professional development is refined to eliminate biases in training material recommendations, allowing for a personalized and equitable learning experience.
- An AI-driven project management assistant is adjusted to ensure equitable distribution of tasks and recognition of team members’ contributions based on results or accomplishments (not activities), thereby enhancing team spirit and productivity.
- A study uncovered racial bias in a hospital algorithm that was supposed to flag patients needing extra care management. It mistakenly identified patients with higher healthcare costs, who were predominantly white, due to historical disparities in spending on Black patients' healthcare (published in the Wall Street Journal, 2 December 2023).
AI4PI Principle #4: Establish AI4PI Solution’s Governance and Risk Management Strategies
Key concepts: governance, leadership, risk categorization, risk mitigation, banned use or misuse, harmful systems, accountability, worker and consumer protection, self-declaration, transparency, framework
Creating a risk-aware governance framework that categorizes AI operations and ensures AI that practices adhere to ethical standards and protect human fundamental rights.
The governance framework should ensure that AI4PI solutions are evaluated and categorized according to the level of risk they pose to individuals, organizations and the society (the three levels are defined in Standard 5):
- Low or minimal risks: These AI4PI solutions are unlikely to cause any harm to individuals, organizations, or the society. For example, a chatbot within the organization’s firewall that helps users find shared, non-confidential information on the internal servers would be considered to have low or minimal risks.
- High risks, still manageable: These AI4PI solutions have the potential to cause harm, but the risks can be mitigated through careful design and implementation. For example, an AI system that is used to make decisions about who receives a performance bonus could have high risks.
- Unacceptable risks: These AI4PI solutions are considered to be too risky to be used. For example, an AI system that is used to make decisions about who gets life-saving medical treatment or any other high-consequence decisions would be considered to have unacceptable risks if a human user is not ultimately making the final decision.
It is important to note that the level of risk associated with an AI4PI solution can change over time. For example, an AI system that was originally considered to have low or minimal risks may become more risky if it is used in a new way or if the underlying data changes: in the above example, the internal chatbot could gain access to personal or confidential data that should not be shared with the entire workforce.
Therefore, it is important to regularly review AI4PI solutions to ensure that they continue to meet acceptable risk levels
AI applications must be transparent about their nature (self-declaration):
- Clearly declaring themselves as non-human agents when interfacing with human users, or when creating new media (sound, picture, video).
- Displaying the level of risks of the AI application.
The uses of AI that compromise universal values or infringe on fundamental human rights, including techniques that could manipulate individuals subconsciously or exploit the vulnerabilities of protected groups, leading to potential psychological or physical harm, must be prohibited. Likewise, the indiscriminate uses of AI such as social scoring, biometric identification, fraud, discrimination, disinformation, disempowering workers, stifling competition, increasing the risks to national security and other misuse of AI, are explicitly banned.
AI4PI solutions promote individual, societal and environmental wellbeing:
- Societal Well-being involves ensuring AI systems benefit society, enhance social skills, respect democratic values, and do not harm physical or mental health. It focuses on AI’s positive contribution to social dynamics and the fair treatment of all human beings.
- Environmental Well-being is about minimizing AI systems' environmental impact, focusing on sustainable development and ecological responsibility throughout their lifecycle, from development to deployment, including their supply chain.
So clients:
- Have the ability to identify high-risk versus low risk applications of AI early in the design and development process.
- Have clarity on the ethical considerations and risk level of AI4PI solutions they use.
- Can trust that AI applications will be transparent and declare their nature as tools rather than human equivalents.
- Are protected from AI practices that could lead to manipulation, discrimination, or harm.
- Engage in a dialogue and pursue AI4PI solutions that promote individual, organizational and societal wellbeing.
Examples:
- Banned use: An AI-driven employee assessment tool is governed by a biased dataset that could lead to discriminatory practices, thus not ensuring that employee evaluations are solely based on individual performance.
- Declaration: A customer service chatbot is designed to clearly inform users that they are interacting with an AI, establishing realistic expectations and trust.
- Harmful system: A healthcare AI that assists in patient treatment planning operates under strict governance to ensure it does not exploit vulnerabilities or bias treatment recommendations, thereby safeguarding patient welfare and trust.
Human-Centered AI for Performance Improvement Process
Key concepts: needs, opportunities, performance gap, root-cause, performance analysis, three levels of needs or opportunities, worker support, environmental and societal well-being, fear of and resistance to AI, change management, organization's readiness
The deployment of Human-Centered AI should be purposefully aligned to address specific, identified needs or opportunities. The needs analysis should not only consider short-term factors but also analyze the root-causes of the performance gaps and anticipate evolving requirements, ensuring its continuous relevance and value enhancement over time. ISPI’s Performance Improvement standards should be applied to conduct a performance analysis, quantify the performance gaps and reveal the actual root-causes.
So clients:
- Evaluate their organization's readiness and potential challenges to embrace a new AI4PI solution.
- Have identified and articulated the specific need, opportunity and root-causes that the AI4PI solution will be addressing.
- Understand how AI can augment the capabilities of individual workers and teams and improve operational systems.
- Recognize the potential of AI to reorganize workplace systems.
Ai4PI solutions address three levels of needs or opportunities:
- 🧑 Level 1: Worker/Individuals. AI Assistants (or AI Advisors) assist individuals perform their tasks defined by the existing work system and within the existing workplace. AI Assistants assist us in our personal and professional lives.
- 👨👨👧👦 Level 2: Work/Workplace Systems. By taking over specific or repetitive tasks, AI Automators change the way teams are performing work and how the workplace is organized.
- 🌎 Level 3: Society/Mega Impact. AI Transformative Agents transform communities, the society and the environment.
Key concepts: customization, evolving needs, unique requirements, fine-tuning, Retrieval Augmented Generation (RAG), prompt engineering
AI4PI solutions should be customized to serve the specific needs of the organization and/or realize new opportunities. This means that the solution should be tailored to the organization's unique challenges
and use relevant organization data in their responses. For example, if an organization is facing a problem with employee turnover, the solution should be designed to address the specific factors and root-causes
that are causing employees to leave.
Customizing a solution to meet the specific needs of the organization increases the AI accuracy, likelihood of success, solution effectiveness and overall impact.
So clients:
- Benefit from AI4PI solutions that are tailored to their specific needs and requirements.
- Obtain specific, customized responses to their unique challenges.
- Experience a productive, symbiotic relationship between workers, the work, the workplace, and AI.
- Engage with AI technologies that are continuously refined to address evolving individual and organizational needs.
Methodologies to customize an AI4PI solution include:
- Shot prompting: Uses the same API conversation ID. This means that the AI model is able to access the context of the previous conversation, which can help it to generate more relevant and informative responses.
- Retrieval-Augmented Generation (RAG): Extracts relevant knowledge from a dataset and combines it with the AI response to produce a custom output. This can help to make the AI responses more personalized and tailored to the individual user.
- Fine-tuning: Adapts a model by making small adjustments with a small amount of task-specific information. This can help to improve the performance of the AI model on a specific task, such as generating text or answering questions.
Effective Prompt Engineering techniques, to include:
- Clear Instructions: Provide concise and precise instructions to the AI, including details, desired format, and length of output, to get more relevant and accurate answers.
- Reference Text: Supply the AI with reference texts, citations, or examples to guide its responses, particularly when there's a risk of fabricating information.
- Splitting Complex Tasks: Break down complex tasks into simpler steps, using the output of earlier tasks as inputs for later ones, to reduce error rates and improve response accuracy.
- Time to 'Think': Allow the AI to process information thoroughly by structuring prompts that encourage thoughtful responses, which could involve inner monologue sequences or questions that guide the model's reasoning.
- External Tools: Enhance AI capabilities by integrating external tools for tasks better suited to specialized functions, such as code execution engines or embedding-based search for document retrieval.
- Testing: Measure the performance of prompt modifications systematically, using a comprehensive testing methodology, to ensure prompt adjustments lead to overall performance improvement.
Key concepts: user interface (UX), usability, user experience, context, point of need, optimum collaboration, unique human skills
Engineering human-machine user interfaces should be done carefully to ensure an adequate, relevant and contextually appropriate use of the AI4PI solution.
So clients:
- Can interact with AI in a way that feels natural and produces worthy outcomes at the point of need.
- Have confidence in the AI's ability to understand and respond to complex needs with precision.
- Benefit from AI interfaces that are tailored to specific tasks, enhancing user experience and performance.
Examples:
- A company providing AI-powered internal chatbots should ensure that their chatbots are tailored to the actual needs of their employees. For example, a chatbot about pay and benefits should be tested to answer key concerns and questions of the workforce, with the goal of improving employees’ morale.
- A hospital that uses AI-powered medical imaging software ensures that the software is easy to use for radiologists and other healthcare professionals. The software should be able to interpret images quickly and accurately, and provide clear and concise reports that can be effectively used by the medical professionals.
Key concepts: accuracy measurement, reliability testing, stress test, confidence level, hallucination, confusion matrix, technical robustness
The accuracy of the AI output should be systematically assessed using a thorough, step-by-step methodology. The findings of the Accuracy evaluation should be reported to the organization's leadership and AI4PI solution users. The goal is to ensure that AI actions are reliable and conform to the expected standards of operation. A systematic plan should be implemented to measure and improve the AI accuracy (KPI) over time. For instance, instances of AI 'hallucinations' and incorrect responses can be minimized by implementing a feedback loop into the AI4PI solution.
So clients:
- Have confidence in the accuracy and reliability of AI outputs.
- Understand the CI process of improving AI performance over time.
- Are assured of AI4PI solutions' adherence to established standards through regular, transparent accuracy assessments.
- Methodology for measuring and improving AI accuracy over time include:
- Compare expert (SME) recommendations to AI-generated outputs to assess initial AI accuracy.
- Use a confusion matrix to evaluate AI performance, calculating overall accuracy by considering true positives, false negatives, true negatives, and false positives.
- Conduct multiple rounds of testing to fine-tune the AI4PI solution parameters.
- Capture the corrections and insights into a database to help customize and finetune the AI model.
- Conduct a Stress test to validate the Reliability or Robustness of the AI4PI solution.
Top
Key Concepts: testing, audits, accountability, reporting, technical robustness, safety, AI solution levels, implementation plan, change management
The deployment of AI4PI solutions demands a rigorous testing and auditing process to confirm the compliance with ethical standards, operational effectiveness, and alignment with organizational and societal values. Regular and thorough auditing of AI systems is crucial to validate their performance, adherence to ethical norms and organizational principles.
These human-led audits should include extensive evaluations of the AI accuracy, reliability, usability in the users’ actual environment, personal data protection, bias reduction, and the examination of indirect, second-order impacts.
So clients:
- Can trust that AI4PI solutions are functioning as intended and in a manner that upholds ethical standards.
- Have assurance in the AI's alignment with both organizational objectives and broader societal values.
- Receive transparent insights into the AI's performance and impact, fostering a culture of accountability.
Examples:
- Data Protection Audit: An AI-driven health records system undergoes rigorous data privacy audits to ensure patient information is handled securely and in compliance with healthcare regulations.
- Bias Mitigation Audit.
- Risk Audit to properly categorize the risk associated with the AI4PI solution.
- Usability Test.
- Accuracy Tests and Stress Tests.
Top
Key concepts: performance metrics, results and outcome, first and second order impact on individual, organizational and societal well-being, ROI, tangible and intangible benefits, individual, organizational and societal well-being
It is essential to complete a thorough analysis of the AI impact in order to mitigate risks and maximize the cost/benefit ratio. The impact evaluation should encompass the expected tangible and intangible benefits of the AI4PI solution as well as second-order impacts and broader implications of AI on individuals, organizations and the society (three levels described in Standard 5).
Tangible benefits like improved work efficiency allows employees to spend more time on higher-value activities, such as relationship building, consulting, coaching, and mentoring, which improves the impact of their work.
AI initiatives can also result in intangible benefits, such as improved customer satisfaction and brand reputation by allowing employees to spend more time with Customers and less time with administrative duties. Intangible benefits of AI can be difficult to quantify. However, they can be significant and should not be overlooked when evaluating the ROI of an AI initiative.
So clients:
- Gain a holistic view of how AI affects various aspects of work and life.
- Understand both the tangible and intangible benefits AI brings to their operations and overall mission.
- Can make informed decisions about AI investments, considering their full range of impacts.
- Have clear insights into positive like negative second order impacts that could be detrimental to individuals, organizations, or the society.
Examples:
Impact on:
- Individuals: productivity improvement, time to focus on higher-level order tasks (thinking, analyzing, solving, effectiveness improvement)
- Teams/Organizations: time to market, failure rates, revenue increase, operational margin, brand reputation, competitiveness, diversity and inclusion
- The society: net gain/loss in employment, environmental impact, growth opportunities versus widening gaps
- Tangible Benefits: An AI-enhanced project management tool increases efficiency, freeing up team members for high-value tasks like client relations and strategic planning, thereby amplifying the quality of their work.
- Intangible Benefits: Implementing an AI customer service platform leads to higher customer satisfaction and enhanced brand reputation, factors crucial for long-term business success.
Print Page