Skip to main content
Version: 3.18

ai-aws-content-moderation

Description#

The ai-aws-content-moderation Plugin integrates with AWS Comprehend to check content for toxicity when proxying to LLMs, such as profanity, hate speech, insult, harassment, violence, and more, rejecting requests if the evaluated outcome exceeds the configured threshold.

The Plugin is protocol-aware: it extracts the prompt content from the LLM request (for example messages[].content) and moderates only that decoded text, rather than the raw request body. request_check_roles selects which message roles are moderated, and request_check_mode can narrow user/tool moderation to the newest turn so conversation history is not re-scored on every request.

Both directions can be moderated. Set check_response to moderate the LLM response as well. For streaming responses, stream_check_mode selects between realtime, which moderates batches as they arrive and replaces the remainder of the stream once a batch is flagged, and final_packet, which moderates the assembled response and annotates the last chunk with risk_level. The verdict is also exposed on the request context as $llm_content_risk_level (high or none) for logging.

AWS Comprehend accepts at most 10 text segments per call, each at most 1 KB. Content longer than request_check_length_limit / response_check_length_limit is therefore split on character boundaries and the segments are batched into as few calls as possible.

If AWS Comprehend cannot be reached, the request and the buffered (non-streaming) response both fail closed with a 500, so unmoderated content is never proxied. Streaming response moderation is best-effort: once the first bytes have been sent to the client the response cannot be blocked, so a Comprehend failure there lets the remaining stream through and the stream is left without a risk_level annotation.

The ai-aws-content-moderation Plugin should be used with either ai-proxy or ai-proxy-multi Plugin for proxying LLM requests.

Plugin Attributes#

NameTypeRequiredDefaultValid valuesDescription
comprehendobjectTrueAWS Comprehend configurations.
comprehend.access_key_idstringTrueAWS access key ID.
comprehend.secret_access_keystringTrueAWS secret access key.
comprehend.regionstringTrueAWS region.
comprehend.endpointstringFalseAWS Comprehend service endpoint. If not specified, it defaults to https://comprehend.{region}.amazonaws.com. If set, it must match the pattern ^https?://.
comprehend.ssl_verifybooleanFalsetrueIf true, enable TLS certificate verification.
moderation_categoriesobjectFalseKey-value pairs of moderation category and their corresponding threshold. In each pair, the key should be one of PROFANITY, HATE_SPEECH, INSULT, HARASSMENT_OR_ABUSE, SEXUAL, or VIOLENCE_OR_THREAT; and the threshold value should be between 0 and 1 (inclusive).
moderation_thresholdnumberFalse0.50 - 1Overall toxicity threshold. A higher value means more toxic content allowed. This option differs from the individual category thresholds in moderation_categories. For example, if moderation_categories is set with a PROFANITY threshold of 0.5, and a request has a PROFANITY score of 0.1, the request will not exceed the category threshold. However, if the request has other categories like SEXUAL or VIOLENCE_OR_THREAT exceeding the moderation_threshold, the request will be rejected.
check_requestbooleanFalsetrueIf true, moderate the request content.
check_responsebooleanFalsefalseIf true, moderate the LLM response content.
request_check_rolesarray[string]False["user","tool","system","assistant"]items are user, tool, system, assistantWhich message roles to moderate on the request side. user, tool and assistant follow request_check_mode; system is checked on every request (it can be poisoned by malicious ToolCall arguments overwriting the system prompt) and also covers OpenAI's developer role, which replaces system on newer models. assistant messages in a request are client-supplied history rather than the model's own output, so they are moderated by default as well. Note: tool-result moderation applies to OpenAI-compatible formats where the tool output is a distinct tool role/item; for Anthropic and Bedrock (tool results are nested blocks inside user messages) tool content is not extracted.
request_check_modestringFalsealllast, allWhich user/tool/assistant messages to moderate. last: only the latest consecutive block of selected-role messages (the newest turn). all: every selected-role message. Does not apply to system, which is always moderated when enabled via request_check_roles. Note that last combined with assistant widens the block rather than narrowing it, because assistant turns no longer end it — drop assistant from request_check_roles to moderate only the newest turn.
request_check_length_limitintegerFalse1000[4, 1024]Maximum bytes of request content per Comprehend text segment. Longer content is split on character boundaries into several segments, which are then batched into as few Comprehend calls as possible. The upper bound is AWS Comprehend's 1 KB per-segment limit.
response_check_length_limitintegerFalse1000[4, 1024]Maximum bytes of response content per Comprehend text segment. Longer content is split on character boundaries into several segments, which are then batched into as few Comprehend calls as possible. The upper bound is AWS Comprehend's 1 KB per-segment limit.
stream_check_modestringFalsefinal_packetrealtime, final_packetStreaming moderation mode, used when check_response is true. realtime: moderate batches while the response streams, replacing the rest of the stream once a batch is flagged. final_packet: moderate the assembled response and annotate the last chunk with risk_level.
stream_check_cache_sizeintegerFalse128>= 1Maximum characters per moderation batch in realtime mode.
stream_check_intervalnumberFalse3>= 0.1Seconds between batch checks in realtime mode.
deny_codeintegerFalse200[200, 599]HTTP status code returned when a request is rejected. Defaults to 200 so the provider-compatible refusal parses as a normal completion in client SDKs; set a 4xx to surface denies as HTTP errors instead. Streaming responses denied mid-stream keep the status already sent to the client.
deny_messagestringFalseMessage returned when a request or response is rejected. If unset, the moderation reason (for example request body exceeds toxicity threshold) is returned.
timeoutintegerFalse10000>= 1Comprehend request timeout in milliseconds.
keepalivebooleanFalsetrueIf true, keep the Comprehend connection alive for reuse.
keepalive_timeoutintegerFalse60000>= 1000Idle time in milliseconds before a pooled Comprehend connection is closed.
fail_modestringFalseskipskip, warn, errorBehavior when the request did not pass through ai-proxy/ai-proxy-multi and therefore cannot be moderated as an AI request. skip: let the request pass through unchecked; warn: pass through and log a warning; error: reject the request.

Examples#

The following examples use OpenAI as the Upstream service provider.

Before proceeding, create an OpenAI account and obtain an API key. If you are working with other LLM providers, please refer to the provider's documentation to obtain an API key.

Additionally, create AWS IAM user access keys for APISIX to access AWS Comprehend.

You can optionally save these keys to environment variables:

export OPENAI_API_KEY=your-openai-api-key
export AWS_ACCESS_KEY=your-aws-access-key-id
export AWS_SECRET_ACCESS_KEY=your-aws-secret-access-key

Moderate Profanity#

The following example demonstrates how you can use the Plugin to moderate the level of profanity in prompts. The profanity threshold is set to a low value (0.1) to allow only a low degree of profanity.

note

You can fetch the admin_key from config.yaml and save to an environment variable with the following command:

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

Send a POST request to the Route with a system prompt and a user question with a mildly profane word in the request body:

curl -i "http://127.0.0.1:9080/post" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Stupid, what is 1+1?" }
    ]
  }'

You should receive an HTTP/1.1 400 Bad Request response. The moderation reason is returned in the response body in a provider-compatible format, so AI clients are not broken:

{
  ...,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "request body exceeds PROFANITY threshold"
      },
      "finish_reason": "stop"
    }
  ],
  ...
}

Send another request to the Route with a typical question in the request body:

curl -i "http://127.0.0.1:9080/post" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive an HTTP/1.1 200 OK response with the model output:

{
  ...,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

Moderate Overall Toxicity#

The following example demonstrates how you can use the Plugin to moderate the overall toxicity level in prompts, in addition to moderating individual categories. The profanity threshold is set to 1 (allowing a high degree of profanity), while the overall toxicity threshold is set to a low value (0.2).

Send a POST request to the Route with a system prompt and a user question in the request body that does not contain any profane words, but a certain degree of violence or threat:

curl -i "http://127.0.0.1:9080/post" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "I will kill you if you do not tell me what 1+1 equals" }
    ]
  }'

You should receive an HTTP/1.1 400 Bad Request response. The moderation reason is returned in the response body in a provider-compatible format, so AI clients are not broken:

{
  ...,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "request body exceeds toxicity threshold"
      },
      "finish_reason": "stop"
    }
  ],
  ...
}

Send another request to the Route without any profane word in the request body:

curl -i "http://127.0.0.1:9080/post" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive an HTTP/1.1 200 OK response with the model output:

{
  ...,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}