Humanloop
Humanloop is an LLMOps platform for product, engineering, and domain expert collaboration, covering Evaluation, Prompt Management, Observability, Datasets, Logs, Evaluators, Agents, and API workflows. Official documents show that the Humanloop platform has sunset on 2025-09-08 and is suitable for historical product research, migration evaluation, and similar platform selection comparison.
Humanloop
Core parameters and statistics
| Projects | Public Information |
|---|---|
| Product Positioning | Enterprise-level LLM evals platform, covering Evaluation, Prompt Management, Observability |
| Official entrance | https://humanloop.com/ |
| Document entrance | https://humanloop.com/docs/ |
| Default document version | v5.0 |
| Main capabilities | Evaluators, Datasets, Logs, Prompts, Agents, Tools, Flows, API/SDK, Monitoring |
| API sample domain name | https://api.humanloop.com/v5 |
| Platform status | 2025-09-08 sunset |
| Billing status | The official changelog shows that billing has stopped since 2025-07-30 |
| Team Status | The Humanloop team has joined Anthropic |
| Applicable form | Web console API, Python/TypeScript SDK and documented workflow |
Product Boundaries: Humanloop is not a base model provider, nor is it a general-purpose chatbot. Its core position is in the LLM application development and governance layer, helping teams manage prompt files, data sets, logs, evaluators, online monitoring and collaboration processes. Since the platform has sunset, it is currently not suitable as a direct procurement target for new projects. It is more suitable for LLMOps product form research, legacy migration reference, and comparison of similar solutions.
Status Meaning: Humanloop’s official homepage and documentation both refer to “Join Anthropic” and “Platform sunset” as the current factual state. For historical customers, the key issue is data export and migration; for the new selection team, the key value is understanding the product structure that it once combined evals-driven development and collaborative development into one platform.
User and market recognition
Enterprise-level positioning: The official website meta information describes Humanloop as "LLM evals platform for enterprises", and the homepage of the document emphasizes that product teams can use Humanloop to build robust AI features and collaborate with Observability through Evaluation, Prompt Management. This positioning shows that it does not serve a single Prompt experiment, but an enterprise AI function with version, testing, monitoring and team collaboration requirements.
Customer and ecological signals: The case-studies page of the official website publicly displays the customer case entrance, where customer names such as Gusto can be verified in the page content; the YC company page also classifies Humanloop as a W20 company and records its tool positioning for LLM application development. Humanloop also publicly retains GitHub, X, LinkedIn, Trust Center and Media Kit portals, indicating that it once operated as a dual channel for developers and enterprises.
Market Boundaries: Humanloop officially does not publicly disclose verifiable total active customer ARR, renewal rate, user volume, or adoption scale broken down by industry. Customer logos, cases and investor information can only indicate that they have received attention from enterprises and capital markets, and cannot be inferred to determine business scale.
Cost advantage
C-side/Individual Developers: Humanloop’s historical products are not typical personal consumption tools. The document emphasizes two types of entrances for engineers and product managers. API, Prompt file Datasets, Evaluators and Logs are more suitable for the development process of existing LLM applications. After the platform sunsets, individuals or small teams can no longer purchase and use it as ordinary SaaS. The historical price and free quota are subject to the official archive and contract.
Developer/API Team: The cost advantage mainly comes from reducing the switching loss of "prompts are stored in documents, evaluations are stored in tables, and online logs are stored in the monitoring system". Prompts can be called through API, Logs record the input, output, version and metadata of each Function File execution, and Datasets can be precipitated from Logs into evaluation cases; this combination reduces the hidden costs of positioning regression, recurrence experiments and cross-role communication.
Enterprise/Governance Scenario: Humanloop’s public page once highlighted compliance and security, Trust Center, SOC 2 and HIPAA logo entry, but the enterprise contract price SLA, data retention, privatization and security terms are not fully disclosed on the current public page. After the platform sunsets, procurement actions should be transferred to migration plans and alternative platform evaluations. Refunds, data exports and contract arrangements for historical customers shall be subject to official migration instructions communicated with customers.
Main functions
- Evaluation: Humanloop's Evaluator is a function that judges the Log generated by LLM, and can return results such as boolean, number, select, multi-select or text. It supports offline evaluation in the development phase and online monitoring in production scenarios.
- Prompt Management: Prompt files save templates, models, parameters and available tools; changes to templates, model temperature, max tokens, top_p or tools will generate new versions. This structure is suitable for teams to turn prompts from chat records into versionable, callable, and measurable engineering assets.
- Observability and Logs: Logs captures the inputs, outputs, used versions and metadata of each execution of Function File; external production systems can also report logs to Humanloop for monitoring and evaluation.
- Datasets: Datasets are composed of Datapoints, which can contain inputs, messages and targets. The Dataset version is immutable, and Evaluation will be bound to a specific Dataset Version, making it easy to track which set of test cases were used in a certain evaluation.
- Evaluators source: Evaluators support three types of judgment sources: Code, AI, and Human. Code is suitable for certain rules such as cost token, latency and regex; AI is suitable for semantic quality judgment; Human is suitable for gold standard feedback of high-value or high-risk tasks.
- Agents and Tools: May 2025 changelog shows Humanloop supports nested Agents, linking Files to Agents by environment, and calling linked tools in Prompt templates for building more complex RAGs and multi-step agentic workflows.
- Model and supplier integration: Updated in May 2025, Claude 4 Sonnet/Opus supports AWS Bedrock cross-Region inference and DeepSeek R1 support on Bedrock, indicating that the platform once had multi-model and multi-vendor access as important capabilities.
Model and version evolution
Mainline nodes of Humanloop
Platform Sunset (2025-09-08): The official document homepage and the August 2025 changelog clearly state that the Humanloop platform sunsets on 2025-09-08. After this node, the platform and data are permanently inaccessible, so the latest status is not "new feature version", but "service termination and migration completion node".
Humanloop joins Anthropic (2025-08-13): The official homepage of Humanloop announces that the team has joined Anthropic and thanks customers, investors and supporters. The changelog also states that billing has stopped as of 2025-07-30, and prorated refunds for annual subscriptions will be handled by the team contacting customers.
May 2025 product updates (2025-05): The May changelog is one of the last verifiable set of intensive product capability updates, including RAG linked template tools, Claude 4 support, AWS Bedrock cross-Region inference, Prompts in code, more intuitive observability, trace logs in review tab, nested Agents, and DeepSeek R1 via Bedrock.
Documentation version of Humanloop
v5.0 default document: The official llms.txt shows that Humanloop Docs has two document versions, v5.0 and v4.0, and v5.0 is default. The v5 document covers Prompt, Dataset, Evaluator, Log, Agent, Tool and other objects, which is suitable for understanding Humanloop’s product abstraction before sunset.
v4.0 Historical Documentation: The v4.0 document retains the concepts and API paths of the old version, which is suitable for verifying the old project structure when migrating historical customers. Since the current platform status has been terminated, version research should focus on migration and compatibility judgment rather than new feature integration.
Technical advantages
Unification of evaluation objects: Humanloop integrates Prompt, Agent, Tool, Evaluator, and Flow into the Function File execution system, and Logs captures the input, output, and version of each execution. The advantage of the mechanism is that evaluation, monitoring and playback all revolve around the same type of execution evidence; the applicable scenario is enterprise AI applications with multiple prompts and multiple agents.
Online monitoring and offline evaluation share the same Evaluator: The same type of Evaluator can be used for online monitoring of production Logs, and can also be used with Dataset for offline evaluation. Mechanically, the team does not have to maintain two completely different sets of judgment logic for pre-launch testing and post-launch monitoring; the effect is that quality standards are easier to continuously reuse.
Version link between Prompt and dataset: Changes in Prompt templates, models, parameters and tools will generate new versions, and Dataset Version is uniquely determined by the content of Datapoints. Mechanically, each Evaluation can be traced back to the specific Prompt version and Dataset version; the effect is that the causal relationship between the model replacement Prompt modification and the test set adjustment is clearer.
UI-first and code-first coexist: Official documentation explains that Humanloop serves both developers and subject matter experts. Mechanically, engineers can access through API/SDK/CLI and local file workflow, and product managers or domain experts can create prompts, run reviews and participate in feedback in the UI; it is suitable for teams that need non-technical roles to participate in AI quality acceptance.
How to use
| Usage path | Official entrance/object | Typical steps | Current status |
|---|---|---|---|
| Prompt Management | Prompt File / Prompt Editor / /v5/prompts/call |
Create Prompt -> Configure templates, models and parameters -> Call through UI or API -> Record Logs -> Build Dataset based on Logs | The platform has sunset and can only be used for historical reference |
| Evaluation workflow | Evaluators + Datasets + Evaluation Runs | Create Dataset -> Define Code/AI/Human Evaluator -> Run Evaluation on different Prompt/Agent versions -> Aggregate judgment | The platform has sunset and can only be used for migration and comparison reference |
| Production Observation | Logs + Monitoring Evaluators | Report Logs when running externally -> Deploy monitoring Evaluators -> Continuously check for drift, latency, cost or quality degradation | Platform has sunset |
| Migration Export | Migration Guide | Export data before sunset; the platform and data will be inaccessible after 2025-09-08 | The migration window has passed as of now |
Implementation Boundary: If you are studying Humanloop's product methods, the minimum path is to understand its feedback context from the four objects of Prompt, Log, Dataset, and Evaluator; if it is a historical customer migration, the export file API compatible Prompt template Dataset test case and Evaluator rule should be the core assets; if it is a new project selection, you should choose a similar LLMOps/Evals platform that is still in operation.
Product Pricing
| Cost Hierarchy | Public Status | Description |
|---|---|---|
| Individual/Small Team | Undisclosed available new purchase prices | There are no public packages available for new purchases on the current page, and the platform has closed on 2025-09-08 sunset |
| Developer/API | Historical prices are not fully retained on the current public page | API/SDK capabilities can be verified in the documentation, but billing has stopped since 2025-07-30 |
| Enterprise/Compliance | Contracts and refunds are subject to official customer communication | Customers with annual subscriptions beyond the sunset date will be contacted by the Humanloop team for prorated refunds |
Price Judgment: Humanloop's current business status is not "SaaS on a monthly subscription", but "termination of original platform after joining Anthropic". Therefore, the historical pricing page should not be used as a purchasable price list. New projects need to shift cost analysis to alternative platform subscriptions, model calling fees, data migration fees, Evaluator rewriting Prompt version migration and monitoring reconstruction costs.
Application scenarios
- Prompt version management: suitable for teams that need to compare different combinations of models, templates, parameters and tools. Humanloop's Prompt File separates configuration from query-time data, allowing different versions of the same task to enter the evaluation process.
- LLM Quality Assessment: Suitable for teams that use Dataset as a library of test cases for AI applications. The inputs, messages, and targets of Datapoints can accept real failure samples, boundary samples, and standard answers to support regression testing.
- Production Observability and Monitoring: Suitable for teams that have already launched AI functions. Logs, Monitoring Evaluators, and trace structures help teams continuously observe quality drift, latency, cost, tool calls, and output compliance.
- Cross-role collaborative development: suitable for teams where product managers, engineers and domain experts jointly define quality standards. Human Evaluators and UI-first workflows allow non-technical experts to participate in evaluation, while the code-first path preserves engineering automation.
- Agent/RAG workflow experiment: May 2025 update instructions for linked template tools and nested Agents. Humanloop can support the combined orchestration of Prompt, Tool, and Agent, which is suitable for product design of complex agentic workflows in historical research.
Applicable people
- AI Product Team: Prompt iteration, evaluation and online monitoring need to be placed in the same feedback chain. The prerequisite is that there are real user tasks, failure samples and definable quality standards.
- LLMOps/Platform Engineering Team: Need to uniformly manage Logs, Datasets, Evaluators and API access for multiple business lines. This group is more concerned with version tracking, monitoring consistency, and migration costs.
- Domain Experts and Product Managers: Need to participate in Prompt quality acceptance, manual evaluation and test set maintenance, but do not want to directly modify the code. Humanloop's UI-first design has served this type of collaboration scenario.
- Historical Customer and Migration Leader: Need to understand Humanloop's File, Version, Log, Dataset, and Evaluator abstractions to migrate old assets to platforms or internal systems that are still in operation.
Unsuitable Boundary: Humanloop is currently not suitable for direct adoption in new production projects; teams that only do one-time Prompt demos, no test sets, and no online monitoring requirements will find it difficult to obtain sufficient benefits from its complete methodology. Organizations involved in compliance, auditing, and long-term data retention should prioritize alternatives that still offer contractual SLAs and support.
Summary and Outlook
The historical value of Humanloop lies in its early advancement of LLM application development from "Prompt debugging" to "evaluation-driven development + collaborative development + production observability". Its object model is clear: Prompt manages configuration, Log manages execution evidence, Dataset manages test cases, Evaluator manages quality judgment, and Evaluation/Monitoring manages quality control before and after release. This structure is still worth learning from similar LLMOps and AI governance platforms.
The current limitations and uncertainties are also very clear: the platform has sunset on 2025-09-08, billing will stop after 2025-07-30, and the platform and data will be inaccessible after 2025-09-08; there is no executable new purchase price on the public page; historical customer data, refunds, and migration details depend on the official migration window and customer communication; after the team joins Anthropic, whether the original Humanloop product line will enter Anthropic in a new form The product system has not been made public.
The implementation suggestion is to treat Humanloop as a methodology and migration object, rather than a new procurement object. Historical customers should take inventory of Prompt files, Dataset versions, Evaluator rules, Logs, export API call paths and permission models; new selection teams can use Humanloop’s closed structure as an acceptance framework to compare whether candidate platforms support versioned Prompts, reproducible evaluations, online monitoring, manual feedback API/SDK access and enterprise security terms.
Version Info
- Humanloop Platform Sunset ：Official documentation and the August 2025 changelog show that the Humanloop platform sunset on 2025-09-08 after the team joined Anthropic; the platform and data will be permanently inaccessible after this date.
- Humanloop joins Anthropic transition ：Humanloop’s official announcement announced that the team has joined Anthropic and stated that the platform has entered the migration and sunset stages; billing has stopped since 2025-07-30.
- May 2025 product updates ：May 2025 changelog announces updates to RAG linked template tools, Claude 4 support, AWS Bedrock cross-Region inference, Prompts in code, nested Agents, and observability.
- Humanloop v5 Docs ：The official document index shows that v5.0 is the default document version, and there is no official precise release date; the v5 document covers Evaluation, Prompt Management, Observability, Agents, Tools, Datasets and API.
User Reviews