Supply Chain Attack on AI Development Libraries: The LiteLLM Compromise

The LiteLLM compromise, occurring in March 2026, represents a significant supply chain attack targeting AI development libraries, orchestrated by a threat actor group identified as TeamPCP. This incident involved the injection of malicious payloads into legitimate PyPI package versions of LiteLLM, specifically 1.82.7 and 1.82.8, which are widely used as a unified API gateway for various Large Language Model (LLM) providers. The attack demonstrated sophisticated methods for code execution, data exfiltration, and persistence, highlighting critical vulnerabilities in the software supply chain impacting AI and cloud-native environments.

Attack Vector and Execution

The LiteLLM compromise was not an isolated event but rather a critical component of a broader, multi-stage supply chain campaign by TeamPCP. The campaign began with the compromise of other prominent developer tools, including the open-source vulnerability scanner Trivy and Checkmarx KICS, in the days leading up to the LiteLLM incident.

The initial breach for LiteLLM stemmed from the compromise of PyPI publishing credentials. This was achieved by exploiting a vulnerability in LiteLLM's CI/CD pipeline, which utilized a compromised version of Trivy. The Trivy security scanner, when used in the CI/CD workflow, inadvertently exfiltrated the project's PyPI publishing tokens to the attackers.

With the stolen credentials, TeamPCP published two malicious versions to PyPI: 1.82.7 and 1.82.8. These malicious packages bypassed the standard GitHub release protocols, being uploaded directly to PyPI.

Execution Mechanisms:

  • Version 1.82.7: The malicious logic was embedded directly within the litellm/proxy/proxy_server.py file. Any system installing this version and subsequently importing the proxy module within their application would inadvertently execute the attacker's payload. The code was injected in a base64-encoded format for obfuscation.
  • Version 1.82.8: This version employed a more aggressive and stealthy execution mechanism. It included a new file named litellm_init.pth within the package wheel. Python's site documentation specifies that executable lines within .pth files are automatically run during the interpreter's startup. This meant that simply having litellm version 1.82.8 installed in a Python environment would trigger the malware, even without an explicit import litellm statement.

The malicious payload in version 1.82.8 was double base64-encoded, making it difficult to detect through casual inspection or static analysis tools.


# Example of a .pth file content that would execute code
import base64; exec(base64.b64decode('...'))

Malicious Payload Analysis

The malware deployed a sophisticated, multi-stage payload designed for extensive data exfiltration and persistence.

Three-Stage Payload:

  1. Credential Harvesting: The malware meticulously gathered a broad spectrum of sensitive information. This included, but was not limited to:
    • Environment variables (API tokens, secrets).
    • SSH keys (~/.ssh/id_rsa, id_ed25519, authorized_keys, known_hosts, config).
    • Cloud credentials (AWS, GCP, Azure keys, EC2 IMDS tokens).
    • Kubernetes data and secrets, service account tokens.
    • Docker configurations.
    • Shell history.
    • Database credentials.
    • Cryptocurrency wallet files.
    • CI/CD secrets (Terraform, GitLab CI, Jenkins, Drone).
    • SSL/TLS private keys.
    • System information (hostname, user, kernel details, network interfaces).
  2. Kubernetes Lateral Movement: The payload included capabilities to compromise entire Kubernetes clusters by deploying privileged pods across cluster nodes. It could read all cluster secrets across all namespaces.
  3. Persistent Backdoor: The malware aimed to establish persistence on compromised systems. It achieved this by writing a Python script, sysmon.py, to ~/.config/sysmon/ and installing it as a user systemd service named sysmon.service. This backdoor allowed the attackers to maintain access and continuously poll an attacker-controlled domain (e.g., https://checkmarx[.]zone/raw) for second-stage payloads, such as /tmp/pglog, which would then be executed.

The collected data was encrypted using a hybrid AES-256 and RSA-4096 scheme before being exfiltrated. The data was packed into an archive, typically named tpcp.tar.gz, and sent via HTTPS POST to attacker-controlled domains, primarily models.litellm[.]cloud.


# Example of credential harvesting (conceptual, simplified)
import os
import glob

def collect_credentials():
    secrets = {}
    # Environment variables
    secrets['env_vars'] = os.environ.copy()

    # SSH keys
    ssh_paths = glob.glob(os.path.expanduser('~/.ssh/id_*'))
    for path in ssh_paths:
        try:
            with open(path, 'r') as f:
                secrets[f"ssh_key_{os.path.basename(path)}"] = f.read()
        except Exception:
            pass
    
    # ... additional logic for cloud credentials, K8s, etc.
    return secrets

# (In a real attack, this would be encoded and exfiltrated)

Affected Versions and Impact

The malicious LiteLLM versions were available on PyPI for approximately three hours on March 24, 2026, between 10:39 UTC and 16:00 UTC, before PyPI quarantined the project.

Package Affected Versions (PyPI) Execution Mechanism Availability Window
litellm 1.82.7 Malicious code in litellm/proxy/proxy_server.py, executed upon module import. March 24, 2026 (approx. 10:39 - 16:00 UTC)
litellm 1.82.8 Malicious litellm_init.pth file, executed automatically at Python interpreter startup. March 24, 2026 (approx. 10:39 - 16:00 UTC)

The immediate impact was the potential for widespread data exfiltration and system compromise across environments where the malicious versions were installed. LiteLLM is widely used, with millions of downloads monthly, and is often embedded in AI applications, backend services, CI/CD pipelines, and cloud infrastructure, placing it in environments with access to high-value secrets.

Due to the automatic execution mechanism of the .pth file in version 1.82.8, developers or CI/CD pipelines that simply ran pip install litellm without pinning a version, or whose dependencies pulled LiteLLM as a transitive dependency, were at risk.

A critical error in the malicious payload, specifically a fork bomb that spawned runaway processes and pegged CPU at 100%, led to system crashes and ultimately facilitated the early discovery of the attack. Without this bug, the credential stealer could have operated silently for an extended period.

Detection and Mitigation

Organizations that installed either litellm version 1.82.7 or 1.82.8 should treat any affected environment as potentially compromised and undertake immediate remediation.

Immediate Remediation Steps:

  • Isolate Affected Systems: Identify and isolate all hosts, containers, CI jobs, and developer workstations that installed the malicious versions.
  • Remove Malicious Packages: Uninstall compromised LiteLLM versions (1.82.7, 1.82.8) and reinstall a verified clean version (e.g., 1.82.6 or the latest patched release). 
    
pip uninstall litellm
# Purge package manager cache to prevent re-installation from cached wheels
pip cache purge
  • Rotate All Credentials: Assume all credentials present on affected machines are compromised. This includes API keys, cloud credentials (AWS, GCP, Azure), SSH keys, Kubernetes secrets, database passwords, CI/CD tokens, and cryptocurrency wallet keys.
  • Check for Persistence: Look for persistence mechanisms installed by the malware. Specifically, check for ~/.config/sysmon/sysmon.py and ~/.config/systemd/user/sysmon.service. In Kubernetes environments, audit kube-system for unauthorized pods (e.g., matching node-setup-*).
  • Audit Logs and Network Behavior: Review authentication logs for unauthorized access and monitor network traffic for suspicious outbound connections to attacker-controlled domains like models.litellm[.]cloud and checkmarx[.]zone.
  • Rebuild from Baseline: Rebuild affected systems and CI environments from known-good, trusted base images.

Proactive Defense Strategies:

  • Dependency Pinning: Always pin dependencies to specific, known-good versions in requirements.txt or other package manifests to prevent automatic updates to potentially malicious versions.
  • Dependency Scanning: Implement continuous scanning of all third-party dependencies for known vulnerabilities and malicious injections. Tools that monitor package integrity and flag suspicious changes are invaluable.
  • Least Privilege: Enforce the principle of least privilege for all development and deployment environments. Limit permissions for CI/CD pipelines and developer workstations.
  • Software Bill of Materials (SBOM): Maintain clear SBOM visibility across all environments to understand and track software components and their origins.
  • Artifact Verification: Implement mechanisms to verify that distributed artifacts match their upstream source, rather than blindly trusting package registries.
  • Runtime Monitoring: Implement runtime monitoring for unusual process execution, file system changes, and network activity.
  • Ephemeral Credentials: Utilize ephemeral, sender-constrained tokens for connecting agents to sensitive resources, avoiding hardcoding long-lived API keys.
  • Supply Chain Security Solutions: Invest in dedicated software supply chain security platforms that offer comprehensive dependency analysis, integrity checks, and real-time threat intelligence.

The LiteLLM incident, part of the broader TeamPCP campaign, underscores the evolving threat landscape where adversaries target foundational components of the software development lifecycle. Compromising widely adopted libraries like LiteLLM enables attackers to potentially gain access to numerous downstream projects and organizations, emphasizing that the security of AI and LLM deployments is intrinsically linked to the integrity of the underlying software supply chain.