[MLOB-2411] Add Distributed Proxy/Gateway Service Guide for LLM Observability

content/en/llm_observability/quickstart.md

@@ -19,7 +19,7 @@
 
 ### Jupyter notebooks
 
-To better understand LLM Observability terms and concepts, you can explore the examples in the [LLM Observability Jupyter Notebooks repository][12]. These notebooks provide a hands-on experience, and allow you to apply these concepts in real time. 
+To better understand LLM Observability terms and concepts, you can explore the examples in the [LLM Observability Jupyter Notebooks repository][12]. These notebooks provide a hands-on experience, and allow you to apply these concepts in real time.
 
 ## Command line
 
@@ -67,7 +67,7 @@
    DD_LLMOBS_AGENTLESS_ENABLED=1 ddtrace-run python quickstart.py
    ```
 
-   Replace `<YOUR_DATADOG_API_KEY>` with your Datadog API key, and replace `<YOUR_DD_SITE>` with your [Datadog site][2]. 
+   Replace `<YOUR_DATADOG_API_KEY>` with your Datadog API key, and replace `<YOUR_DD_SITE>` with your [Datadog site][2].
 
    For more information about required environment variables, see [the SDK documentation][1].
 
@@ -129,6 +129,127 @@
 
 If your application consists of more elaborate prompting or complex chains or workflows involving LLMs, you can trace it using the [Setup documentation][11] and the [SDK documentation][1].
 
+## Tracing distributed proxy or gateway services
+
+Like any traditionally application, LLM applications can be implemented across multiple different microservices. With LLM Observability, if one of these services is a LLM proxy or gateway service, you can trace the LLM calls made by individual LLM applications in a complete end-to-end trace.
+
+### Enabling LLM Observability for a proxy or gateway service
+
+To enable LLM Observability for a proxy or gateway service that might be called from several different ML applications, you can enable LLM Observability without specifying an ML application name.
+
+In the proxy service, enable LLM Observability without specifying an ML application name. Optionally, you can specify a service name.
+
+{{< tabs >}}
+{{% tab "Python" %}}
+
+```python
+# proxy.py
+from ddtrace.llmobs import LLMObs
+
+LLMObs.enable(service="chat-proxy")
+
+# proxy-specific logic, including guardrails, sensitive data scans, and the LLM call
+```
+
+{{% /tab %}}
+{{% tab "Node.js" %}}
+
+```javascript
+// proxy.js
+const tracer = require('dd-trace').init({
+  llmobs: true
+});
+const llmobs = tracer.llmobs;
+
+// proxy-specific logic, including guardrails, sensitive data scans, and the LLM call
+```
+
+{{% /tab %}}
+{{< /tabs >}}
+
+
+In your specific applications that orchestrate the ML applications that make calls to the proxy or gateway service, enable LLM Observability with the ML application name, and wrap the proxy call in a `task` span:
+
+{{< tabs >}}
+{{% tab "Python" %}}
+
+```python
+# application.py
+from ddtrace.llmobs import LLMObs
+LLMObs.enable(ml_app_name="my-ml-app")
+
+import requests
+
+if __name__ == "__main__":
+    with LLMObs.workflow(name="run-chat"):
+      # other application-specific logic - RAG steps, parsing, etc.
+
+      with LLMObs.task(name="chat-proxy"): # wrap the proxy call in a task span
+        response = requests.post("http://localhost:8080/chat", json={
+          # data to pass to the proxy service
+        })
+
+      # other application-specific logic handling the response
+```
+
+{{% /tab %}}
+{{% tab "Node.js" %}}
+
+```javascript
+// application.js
+const tracer = require('dd-trace').init({
+  llmobs: {
+    mlApp: 'my-ml-app'
+  }
+});
+const llmobs = tracer.llmobs;
+
+const axios = require('axios');
+
+async function main () {
+  llmobs.trace({ name: 'run-chat', kind: 'workflow' }, async () => {
+    // other application-specific logic - RAG steps, parsing, etc.
+
+    // wrap the proxy call in a task span
+    const response = llmobs.trace({ name: 'chat-proxy', kind: 'task' }, async () => {
+      return await axios.post('http://localhost:8080/chat', {
+        // data to pass to the proxy service
+      });
+    });
+
+    // other application-specific logic handling the response
+  });
+}
+
+main();
+```
+
+{{% /tab %}}
+{{< /tabs >}}
+
+When making requests to the proxy or gateway service, the LLM Observability SDKs automatically propagate the ML application name from the original LLM application. Additionally, the LLM Observability SDKs automatically tag the `task` span capturing the proxy or gateway service call with the tag `ml-proxy:custom`.
+
+### Observing LLM gateway and proxy services
+
+To observe the tasks performed by a variety of ML applications in the proxy service:
+
+1. In the LLM trace view, view `All Applications` from the top-left dropdown.
+2. Switch to the `All Spans` view in the top-right dropdown.
+3. Filter the list by the `ml-proxy:custom` tag.
+
+{{< img src="llm_observability/all-spans-with-ml-proxy-tag.png" alt="View all spans from all ML applications with the ml-proxy tag set to custom" style="width:100%;" >}}
+
+The spans listed are the parent spans of the any operations executed by the ML applications. To see all spans, and not just the top-level task spans, from the proxy service, you can instead filter by the `service` tag:
+
+{{< img src="llm_observability/all-spans-with-service-tag.png" alt="View all spans from all ML applications with the service tag" style="width:100%;" >}}
+
+### Observing end-to-end usage of LLM applications making calls to a proxy or gateway service
+
+To observe the complete end-to-end usage of an LLM application that makes calls to a proxy or gateway service, you can filter for traces with that ML application name:
+
+1. In the LLM trace view, select the ML application name of interest from the top-left dropdown.
+2. Switch to the `Traces` view in the top-right dropdown.
+
 ## Further Reading
 
 {{< partial name="whats-next/whats-next.html" >}}