promptlayer

---
name: Promptlayer
description: Use when managing prompt templates, running evaluations, building workflows, logging LLM requests, or analyzing AI application performance. Reach for this skill when working with prompt versioning, release management, observability, tables/datasets, or multi-step AI workflows.
metadata:
    mintlify-proj: promptlayer
    version: "1.0"
---

# PromptLayer Skill

## Product summary

PromptLayer is a platform for managing, versioning, and evaluating LLM prompts outside your codebase. It provides a Prompt Registry for storing templates with version control, a Tables system for datasets and evaluations, Workflows for multi-step AI pipelines, and Observability tools for logging and tracing LLM requests. Use the Python SDK (`pip install promptlayer`) or JavaScript SDK (`npm install promptlayer`) to fetch prompts, run them against any LLM provider, log requests, and execute workflows. Authenticate with an API key via the `X-API-Key` header or by setting `PROMPTLAYER_API_KEY` environment variable. Primary docs: https://docs.promptlayer.com

## When to use

Use PromptLayer when:
- **Managing prompts**: Creating, versioning, and deploying prompt templates without code changes
- **Evaluating changes**: Building test datasets and scoring prompt outputs before production
- **Logging LLM calls**: Capturing request/response data, metadata, tags, and scores for observability
- **Building workflows**: Chaining multiple LLM calls, tools, and conditional logic into multi-step pipelines
- **A/B testing**: Using release labels to split traffic between prompt versions
- **Tracing execution**: Following the parent-child hierarchy of LLM calls, tools, and application spans
- **Analyzing performance**: Searching logs, running analytics, and comparing prompt versions

Do not use PromptLayer for: managing your LLM provider credentials (store those as environment variables), building UI dashboards (use the web dashboard instead), or replacing your application's core business logic.

## Quick reference

### SDK initialization

```python
from promptlayer import PromptLayer

client = PromptLayer(api_key="pl_your_key")
```

```javascript
import { PromptLayer } from "promptlayer";

const client = new PromptLayer({ apiKey: "pl_your_key" });
```

### Core methods

| Task | Python | JavaScript |
|------|--------|-----------|
| Run a prompt | `client.run(prompt_name="...", input_variables={...})` | `await client.run({promptName: "...", inputVariables: {...}})` |
| Run a workflow | `client.run_workflow(workflow_id_or_name="...", input_variables={...})` | `await client.run_workflow({workflowIdOrName: "...", inputVariables: {...}})` |
| Get a template | `client.templates.get("name", {"label": "prod"})` | `await client.templates.get("name", {label: "prod"})` |
| Log a request | `client.log_request(provider="openai", model="gpt-4", input={...}, output={...})` | `await client.log_request({provider: "openai", model: "gpt-4", input: {...}, output: {...}})` |
| Track metadata | `client.track.metadata(request_id, {"key": "value"})` | `await client.track.metadata(requestId, {key: "value"})` |
| Track score | `client.track.score(request_id, 0.95)` | `await client.track.score(requestId, 0.95)` |

### Environment variables

Set these before running your code:

```bash
export PROMPTLAYER_API_KEY="pl_..."
export OPENAI_API_KEY="sk_..."  # or ANTHROPIC_API_KEY, GOOGLE_API_KEY, etc.
```

### REST API authentication

All REST requests require the `X-API-Key` header:

```bash
curl -H "X-API-Key: pl_your_key" https://api.promptlayer.com/...
```

### Key file paths and concepts

| Concept | Purpose |
|---------|---------|
| **Prompt Template** | Reusable prompt definition with messages, variables, model settings, and version history |
| **Release Label** | Pointer to a specific prompt version (e.g., "prod", "staging") for deployment without code changes |
| **Table** | Workspace for datasets, evaluations, backtesting, and batch workflows with rows, columns, and scoring |
| **Sheet** | Tab within a Table with its own rows, columns, score config, history, and analytics |
| **Workflow** | Multi-step pipeline with prompt, code, tool, and conditional nodes; can be run via SDK or API |
| **Request Log** | Single LLM call record with input, output, model, timing, tokens, cost, metadata, and prompt association |
| **Trace** | End-to-end execution hierarchy linking multiple request logs, tools, agents, and application spans |

## Decision guidance

### When to use `run()` vs. `log_request()`

| Scenario | Use `run()` | Use `log_request()` |
|----------|-----------|-------------------|
| Fetching and executing a prompt from the registry | ✓ | |
| Using your own LLM client (e.g., OpenAI SDK directly) | | ✓ |
| Custom provider or background processing | | ✓ |
| Need to override model or provider at runtime | ✓ | |
| Logging unsupported LLM providers | | ✓ |

### When to use Prompt Registry vs. Workflows

| Need | Use Prompt Registry | Use Workflows |
|------|-------------------|---------------|
| Single LLM call with versioning and release labels | ✓ | |
| Multi-step pipeline with conditional logic | | ✓ |
| A/B testing between prompt versions | ✓ | |
| Chaining multiple LLM calls or tools | | ✓ |
| Simple prompt iteration and deployment | ✓ | |

### When to use Tables vs. legacy Evaluations

| Use case | Tables | Legacy Evaluations |
|----------|--------|-------------------|
| New evaluation or dataset workflow | ✓ | |
| Existing legacy evaluation pipeline | | ✓ |
| Backtesting with request history | ✓ | |
| Scoring with custom code or LLM-as-judge | ✓ | |

### When to use SDK evals vs. dashboard evaluations

| Scenario | SDK evals | Dashboard |
|----------|-----------|-----------|
| Agent runs on your infrastructure with internal data | ✓ | |
| CI/CD integration (run in pipeline) | ✓ | |
| Simple prompt evaluation without custom harness | | ✓ |
| Comparing multiple prompt versions | | ✓ |

## Workflow

### Running a prompt from the registry

1. **Create the prompt** in the PromptLayer dashboard: click **New** → **Prompt**, add messages with `{{variable}}` syntax, set model and parameters, click **Save Template**.
2. **Test in Playground**: define input variables, click **Run**, verify output.
3. **Create a release label**: hover over the version, click **Add Release Label**, enter "prod" or "staging".
4. **Fetch and run in code**: use `client.run(prompt_name="...", prompt_release_label="prod", input_variables={...})`.
5. **Monitor logs**: open the PromptLayer dashboard, search for the prompt name, review request logs and analytics.

### Building and running an evaluation

1. **Create a Table**: click **New** → **Table**, choose **Create from a prompt** or **Start blank**.
2. **Add rows**: import CSV, request history, or add manually.
3. **Add text columns**: for inputs, expected outputs, metadata, or labels.
4. **Add computed columns**: select type (Prompt Template, LLM Assertion, Code, Comparison, etc.), configure source columns.
5. **Run the sheet**: click **Run** to execute all cells, or select specific rows/columns.
6. **Configure scoring**: click **Score**, choose scoring mode (auto, boolean, numeric, custom code, or scorecard).
7. **Review results**: check cell status, view score history, compare versions, export results.

### Building a workflow

1. **Create a Workflow**: click **New** → **Workflow**, drag nodes onto the canvas.
2. **Add nodes**: Prompt Template (registry or inline), Code Execution, For Loop, While Loop, Math Operator, Callback Endpoint, etc.
3. **Connect nodes**: draw edges between nodes, optionally add conditional logic (==, !=, <, >, <=, >=).
4. **Define input variables**: specify required inputs and their types.
5. **Mark output nodes**: designate which node(s) return the final result.
6. **Test in the editor**: click **Run**, provide input variables, inspect execution trace.
7. **Add release labels**: attach "prod" or "staging" to a version.
8. **Run from code**: use `client.run_workflow(workflow_id_or_name="...", input_variables={...})`.

### Logging custom LLM calls

1. **Make your LLM call** using your own client (OpenAI, Anthropic, etc.).
2. **Capture timing**: record `request_start_time` and `request_end_time`.
3. **Call `log_request()`**: pass provider, model, input (messages), output (response), timing, and optional tags/metadata.
4. **View in dashboard**: open **Logs**, search by tag or metadata, inspect the request.

### Setting up observability with auto-instrumentation

1. **Install the SDK** with tracing extra: `pip install promptlayer[openai]` (or `[anthropic]`, `[google]`, `[bedrock]`).
2. **Import and instrument**: call `instrument_openai()` (or equivalent) before making LLM calls.
3. **Set environment variables**: `PROMPTLAYER_API_KEY` and your provider key.
4. **Make LLM calls normally**: the SDK automatically captures and logs them.
5. **View traces**: open the PromptLayer dashboard, click **Traces**, inspect the execution hierarchy.

## Common gotchas

- **API key not set**: Always set `PROMPTLAYER_API_KEY` as an environment variable or pass it to the client constructor. The SDK will not work without it.
- **Provider credentials not set**: PromptLayer never sends your LLM provider keys to its servers. You must set `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc. as environment variables before calling `run()`.
- **Forgetting to save a prompt template**: After editing a prompt in the Playground, click **Save Template** to persist changes. Unsaved edits are lost.
- **Using wrong variable syntax**: Use `{{variable_name}}` (f-string) or `{{ variable_name }}` (Jinja2) in prompts. Mismatched syntax will not interpolate.
- **Release label not found**: If `run()` fails with "label not found", verify the label exists on the prompt version. Use `client.templates.get(name, {"label": "prod"})` to check.
- **Stale cells in Tables**: After changing a column's source or config, cells may be marked stale. Click **Run** to recalculate them.
- **Workflow node dependencies**: Ensure all nodes a node depends on are connected via edges. Unconnected dependencies will fail at runtime.
- **Streaming response structure**: When using `stream=True`, each chunk includes `raw_response` and progressively built `prompt_blueprint`. The `request_id` is only in the final chunk.
- **Cache invalidation**: If using `cache_ttl_seconds`, remember that cache is in-memory and process-local. Requests with `metadata_filters` or `model_parameter_overrides` bypass cache.
- **Deprecated endpoints**: Datasets and Evaluations APIs are deprecated. Use Tables API for new workflows.
- **Async client**: Use `AsyncPromptLayer` for non-blocking operations in async contexts. Sync and async clients cannot be mixed in the same thread.

## Verification checklist

Before submitting work with PromptLayer:

- [ ] Prompt template is saved and has a release label (e.g., "prod") if used in production code
- [ ] All input variables in the prompt are provided when calling `run()`
- [ ] Environment variables (`PROMPTLAYER_API_KEY`, provider keys) are set
- [ ] Request logs appear in the PromptLayer dashboard after running code
- [ ] Tags and metadata are attached to requests for easier filtering
- [ ] Evaluation Table has rows, computed columns, and scoring configured
- [ ] Workflow nodes are connected with edges and conditional logic is correct
- [ ] Release labels are attached to the correct versions for deployment
- [ ] Traces show the full execution hierarchy (if using auto-instrumentation)
- [ ] No hardcoded API keys in code; all credentials use environment variables

## Resources

- **Comprehensive navigation**: https://docs.promptlayer.com/llms.txt
- **Python SDK reference**: https://docs.promptlayer.com/sdks/python
- **REST API reference**: https://docs.promptlayer.com/reference/introduction
- **Prompt Registry overview**: https://docs.promptlayer.com/features/prompt-registry/overview
- **Tables and evaluations**: https://docs.promptlayer.com/features/tables/overview
- **Workflows**: https://docs.promptlayer.com/why-promptlayer/workflows
- **Observability**: https://docs.promptlayer.com/features/observability/overview
- **SDK evals**: https://docs.promptlayer.com/sdks/evals/overview

---

> For additional documentation and navigation, see: https://docs.promptlayer.com/llms.txt