LLM Model Evaluation & Agent Interface

IEEE sponsors two AI and ADS projects that follow ANSI standardization requirements:

A cheat sheet explanation of how Large Language Models work: pic.twitter.com/CcPujRNvHi

— Abacus.AI (@abacusai) July 23, 2023

Title: IEEE P3119 – Standard for the Procurement of Artificial Intelligence and Automated Decision Systems

Scope: The IEEE P3119 standard establishes a uniform set of definitions and a process model for procuring Artificial Intelligence (AI) and Automated Decision Systems (ADS). It covers government procurement, in-house development, and hybrid public-private development of AI/ADS. The standard redefines traditional procurement stages—problem definition, planning, solicitation, critical evaluation (e.g., impact assessments), and contract execution—using an IEEE Ethically Aligned Design (EAD) foundation and a participatory approach to address socio-technical and responsible innovation considerations. It focuses on mitigating unique AI risks compared to traditional technologies and applies to commercial AI products and services procured through formal contracts.

Purpose: The purpose of IEEE P3119 is to help government entities, policymakers, and technologists make transparent, accountable, and responsible choices in procuring AI/ADS. It provides a framework to strengthen procurement processes, ensuring due diligence, transparency about risks, and alignment with public interest. The standard aims to minimize AI-related risks (e.g., bias, ethical concerns) while maximizing benefits, complementing existing procurement practices and shaping the market for responsible AI solutions. It supports agencies in critically evaluating AI tools, assessing vendor transparency, and integrating ethical considerations into procurement.

Developmental Timelines:

• • September 23, 2021: The IEEE Standards Association (SA) Standards Board approved the project and established the IEEE P3119 Working Group. The Project Authorization Request (PAR) was created to define the scope.
  • 2021–Ongoing: Development continues, with no final publication date confirmed in available sources. As of July 18, 2024, the standard was still in progress, focusing on detailed process recommendations.
  • The standard is being developed as a voluntary socio-technical standard, with plans to test it against existing regulations (e.g., via regulatory sandboxes).

By Whom:

• • Working Group Chair: Gisele Waters, Ph.D., Director of Service Development and Operations at Design Run Group, co-founder of the AI Procurement Lab, and a human-centered design researcher focused on risk mitigation for vulnerable populations.
  • Working Group Vice Chair: Cari Miller, co-founder of the AI Procurement Lab and the Center for Inclusive Change, an AI governance leader and risk expert.
  • IEEE P3119 Working Group: Comprises a global network of IEEE SA volunteers from diverse industries, collaborating to develop standards addressing market needs and societal benefits. The group integrates expertise from government workers, policymakers, and technologists.
  • Inspiration: The standard was inspired by the AI and Procurement: A Primer report from the New York University Center for Responsible AI.

The IEEE P3119 standard is a collaborative effort to address the unique challenges of AI procurement, emphasizing ethical and responsible innovation for public benefit

Title: IEEE P3120 – Standard for Quantum Computing Architecture

Scope: The IEEE P3120 standard defines a general architecture for quantum computers, focusing on the structure and organization of quantum computing systems. It covers the overall system architecture, including quantum hardware components (e.g., qubits, quantum gates), control systems, interfaces with classical computing systems, and software layers for programming and operation. The standard aims to provide a framework for designing interoperable and scalable quantum computing systems, addressing both hardware and software considerations for quantum and hybrid quantum-classical architectures.

Purpose: The purpose of IEEE P3120 is to establish a standardized framework to guide the design, development, and integration of quantum computing systems. It seeks to ensure consistency, interoperability, and scalability across quantum computing platforms, facilitating innovation and collaboration in the quantum computing ecosystem. By providing clear architectural guidelines, the standard supports developers, researchers, and industry stakeholders in building reliable and efficient quantum computers, bridging the gap between theoretical quantum computing and practical implementation.

Developmental Timelines:

• • September 21, 2023: The IEEE Standards Association (SA) Standards Board approved the Project Authorization Request (PAR) for P3120, initiating the project under the IEEE Computer Society’s Microprocessor Standards Committee (C/MSC).
  • 2023–Ongoing: Development is in progress, with no confirmed publication date in available sources. As a standards development project, it involves iterative drafting, review, and consensus-building, typical of IEEE processes, which can span several years.
  • The standard is being developed as a voluntary standard, with potential for testing and refinement through industry and academic collaboration.

By Whom:

• • Sponsor: IEEE Computer Society, specifically the Microprocessor Standards Committee (C/MSC), which oversees standards related to microprocessor and computing architectures.
  • Working Group: The IEEE P3120 Working Group consists of volunteers from academia, industry, and research institutions with expertise in quantum computing, computer architecture, and related fields. Specific chairs or members are not detailed in available sources, but IEEE SA working groups typically include global experts from relevant domains.
  • Stakeholders: The development involves contributions from quantum computing researchers, hardware manufacturers, software developers, and standardization experts to ensure a comprehensive and practical standard.

The IEEE P3120 standard is a critical step toward formalizing quantum computing architectures, aiming to support the growing quantum technology industry with a robust and interoperable framework.