Fix CloudPrem wording and add processing page. (#29140)

* Fix CloudPrem wording and add processing page.

* Apply suggestions from code review

Sounds good, thanks @estherk15 !

Co-authored-by: Esther Kim <esther.kim@datadoghq.com>

---------

Co-authored-by: Esther Kim <esther.kim@datadoghq.com>

Author: fmassot

content/en/cloudprem/_index.md

@@ -1,6 +1,6 @@
 ---
 title: CloudPrem
-description: Learn how to deploy and manage Datadog CloudPrem, a self-hosted log solution for cost-effective log ingestion, indexing, and search capabilities
+description: Learn how to deploy and manage Datadog CloudPrem, a self-hosted log management solution for cost-effective log ingestion, processing, indexing, and search capabilities
 private: true # This removes this page from search and limits the availability of this doc to only those that have the link
 further_reading:
 - link: "/cloudprem/installation/"
@@ -22,40 +22,20 @@ further_reading:
 
 ## Overview
 
-Datadog CloudPrem is a self-hosted log solution which provides cost-effective log ingestion, indexing, and search capabilities in your own infrastructure. Designed to address data residency, security or high-volume requirements, CloudPrem seamlessly integrates with the Datadog platform to offer powerful log analysis, visualization, and alerting while ensuring that your sensitive log data remains within your own infrastructure.
+Datadog CloudPrem is a self-hosted log management solution that enables cost-effective log ingestion, processing, indexing, and search capabilities within your own infrastructure. Built to meet data residency, stringent security, and high-volume requirements, CloudPrem integrates with the Datadog platform to provide log analysis, visualization, and alerting - all while keeping your log data at rest within your infrastructure boundaries.
 
 {{< img src="/cloudprem/cloudprem_overview_diagram.png" alt="CloudPrem product overview diagram" style="width:100%;" >}}
 
-### Core Capabilities
-<!-- This sections was populated with Cursor, we can delete if it's not relevant -->
-
-CloudPrem enhances your log management strategy through several fundamental capabilities:
-- **Data Sovereignty**<br>
-  Process and store logs within your own infrastructure while maintaining full integration with Datadog's analysis tools. This gives you complete oversight of your data's location and handling.
-
-- **Infrastructure Efficiency**<br>
-  Scale log processing and storage according to your needs by leveraging your existing infrastructure. This provides flexibility in resource allocation and management as your requirements evolve.
-
-- **Deployment Flexibility**<br>
-  Adapt the deployment to match your infrastructure requirements and security controls while preserving seamless integration with Datadog's platform features and functionality.
-
 ## Architecture
 
-CloudPrem uses a modular architecture that separates processing tasks from data storage:
-
-- Processing tasks like indexing and searching run independently
-- Log data is stored separately in object storage (like S3)
-- Each component can be scaled separately to match your needs
-- This separation allows you to optimize resources based on your specific workload
-
-<!-- {{< img src="path/to/your/image-name-here.png" alt="TBD CloudPrem architecture and component diagram" style="width:100%;" >}} -->
+CloudPrem uses a decoupled architecture which separates the compute (indexing and searching), and data on an object storage. This allows for independent scaling and optimization of different cluster components based on workload demands.
 
 ### Components
 
 The CloudPrem cluster, typically deployed on Kubernetes (EKS), consists of several components:
 
 **Indexers**
-: Responsible for receiving logs from Datadog Agents. Indexers process, index, and store logs in index files called splits to the object storage (such as Amazon S3).
+: Responsible for receiving logs from Datadog Agents. Indexers process, index, and store logs in index files called splits to the object storage (for example, Amazon S3).
 
 **Searchers**
 : Handle search queries from the Datadog UI, reading metadata from Metastore and index data from the object storage.
@@ -67,41 +47,32 @@ The CloudPrem cluster, typically deployed on Kubernetes (EKS), consists of sever
 : Responsible for tasks like indexing tasks scheduling and delete tasks.
 
 
-## Prerequisites for getting started
+## Get started
+### Prerequisites
 
 Before getting started with CloudPrem, ensure you have:
 
 - AWS account with necessary permissions
 - Kubernetes cluster (EKS recommended)
 - S3 bucket for log storage
 - PostgreSQL database (RDS recommended)
-- Datadog agent installed
-- Required tools: `kubectl`, `helm`
-
-For detailed instructions, see the [Installation][2] documentation.
+- Datadog agent
+- `kubectl`
+- `helm`
 
-## Additional considerations
+### Installation
 
-### Log processing capabilities
+1. [Install CloudPrem][2]
+2. [Send logs to CloudPrem](2)
+3. [Configure logs processing](3)(optional)
+4. [Configure your Datadog account to connect the Log Explorer to CloudPrem](2)
 
-CloudPrem includes basic log processing capabilities out-of-the-box. For more advanced use cases such as dual shipping, sensitive data redaction, or log volume control, Datadog recommends using [Observability Pipelines][3] in conjunction with CloudPrem.
-
-### Billing and usage
-
-Logs sent to CloudPrem components are counted toward your Datadog usage, you will be billed for CloudPrem's internal telemetry.
-
-### Network and cost
-
-CloudPrem sends query results outside your network for display in the Datadog UI. These query results are compressed, resulting in negligible egress costs for most deployments.
-
-### Deployment options
-
-You cannot deploy multiple CloudPrem clusters.
+For detailed instructions, see the [Installation][2] documentation.
 
 ## Further reading
 
 {{< partial name="whats-next/whats-next.html" >}}
 
 [1]: https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/ 
 [2]: /cloudprem/installation/
-[3]: /observability_pipelines/
+[3]: /cloudprem/processing/

content/en/cloudprem/advanced.md

@@ -1,6 +1,6 @@
 ---
-title: Advanced Configuration
-description: Learn about advanced deployment scenarios and customization options for CloudPrem
+title: AWS Configuration
+description: Learn how to configure AWS for CloudPrem
 further_reading:
 - link: "/cloudprem/"
   tag: "Documentation"
@@ -21,13 +21,15 @@ further_reading:
 
 ## Overview
 
-This guide covers advanced configuration options and deployment scenarios for CloudPrem, including multiple cluster deployments, advanced processing features, and integration with external tools. For ingress configuration, refer to the [Ingress Configuration guide](/cloudprem/ingress/).
+This guide covers how to configure your AWS account for CloudPrem. For ingress configuration, refer to the [Ingress Configuration guide](/cloudprem/ingress/).
 
-## AWS setup
+Setting up a CloudPrem cluster on AWS requires the configuration of three elements:
+- AWS credentials
+- AWS region
+- IAM permissions for S3
 
-Setting up a CloudPrem cluster on AWS requires the configuration of the following elements:
+## AWS credentials
 
-{{% collapse-content title="AWS credentials" level="h4" expanded=false %}}
 When starting a node, CloudPrem attempts to find AWS credentials using the credential provider chain implemented by [rusoto\_core::ChainProvider](https://docs.rs/rusoto_credential/latest/rusoto_credential/struct.ChainProvider.html) and looks for credentials in this order:
 
 1. Environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, or `AWS_SESSION_TOKEN` (optional).  
@@ -36,18 +38,18 @@ When starting a node, CloudPrem attempts to find AWS credentials using the crede
 4. Instance profile credentials, used on Amazon EC2 instances, and delivered through the Amazon EC2 metadata service.
 
 An error is returned if no credentials are found in the chain.
-{{% /collapse-content %}}
 
-{{% collapse-content title="AWS region" level="h4" expanded=false %}}
+## AWS Region
+
 CloudPrem attempts to find an AWS region in multiple locations and with the following order of precedence:
 
 1. Environment variables (`AWS_REGION` then `AWS_DEFAULT_REGION`)  
 2. Config file, typically located at `~/.aws/config` or otherwise specified by the `AWS_CONFIG_FILE` environment variable if set and not empty.  
 3. Amazon EC2 instance metadata service indicating the region of the currently running Amazon EC2 instance.  
 4. Default value: `us-east-1`
-{{% /collapse-content %}}
 
-{{% collapse-content title="IAM permissions for Amazon S3" level="h4" expanded=false %}}
+## IAM permissions for S3
+
 Required authorized actions:
 
 * `ListBucket` (on the bucket directly)  
@@ -89,8 +91,6 @@ Here is an example of a bucket policy:
  ]
 }
 ```
-{{% /collapse-content %}}
-
 
 ## Further reading
 

content/en/cloudprem/cluster.md

@@ -21,9 +21,7 @@ further_reading:
 
 ## Overview
 
-This document offers comprehensive guidance on dimensioning and managing CloudPrem cluster components, covering indexers, searchers, and auxiliary services.
-
-Find specific resource requirements (CPU, RAM, storage) for each component, along with practical examples for capacity planning. Use these guidelines to properly size your initial deployment and scale components as your needs grow. The recommendations provided help you maintain optimal performance while efficiently utilizing your infrastructure resources.
+This document gives recommendations on dimensioning your CloudPrem cluster components, particularly indexers and searchers.
 
 <div class="alert alert-info">
 These are starting recommendations. Monitor your cluster's performance and resource utilization closely and adjust sizing as needed.
@@ -36,7 +34,7 @@ These are starting recommendations. Monitor your cluster's performance and resou
   - 2 vCPUs and 4GB of RAM
   - 4 vCPUs and 8GB of RAM
   - 8 vCPUs and 16GB of RAM
-- **Storage:** An indexer stores temporary data and requires persistent storage (e.g., AWS EBS).
+- **Storage:** Indexers require persistent storage (preferably SSDs, but local HDDs or remote EBS volumes can also be used) to store temporary data while constructing the index files.
   - Minimum: 100GB per pod
   - Recommendation (for pods > 4 vCPUs): 200GB per pod
 - **Example Calculation:** To index 1 TB per day (~11.6 MB/s):

content/en/cloudprem/ingress.md

@@ -12,19 +12,13 @@ further_reading:
 
 ## Overview
 
-Ingress configuration is a critical component of your CloudPrem deployment that manages how external traffic reaches your services. A properly configured ingress controller ensures secure, efficient, and reliable access to your CloudPrem environment. It provides:
-
-- **Traffic management**: Routes external requests to the appropriate CloudPrem services
-- **Load balancing**: Distributes incoming traffic across multiple instances for better performance
-- **TLS termination**: Handles HTTPS encryption and certificate management
-- **Access control**: Enables you to define rules for who can access your CloudPrem deployment
+Ingress is a critical component of your CloudPrem deployment, CloudPrem has one public ingress and one private one.
 
 ## Public ingress
 
 The public ingress is essential for enabling Datadog's control plane and query service to manage and query CloudPrem clusters over the public internet. It provides secure access to the CloudPrem gRPC API through the following mechanisms:
-
 - Creates an internet-facing AWS Application Load Balancer (ALB) that accepts traffic from Datadog services
-- Implements TLS encryption with SSL termination at the load balancer level
+- Implements TLS encryption with termination at the load balancer level
 - Uses HTTP/2 (gRPC) for communication between the ALB and CloudPrem cluster
 - Requires mutual TLS (mTLS) authentication where Datadog services must present valid client certificates
 - Configures the ALB in TLS passthrough mode to forward client certificates to CloudPrem pods via the `X-Amzn-Mtls-Clientcert` header
@@ -34,6 +28,17 @@ This setup ensures that only authenticated Datadog services can access the Cloud
 
 <!-- {{< img src="path/to/your/image-name-here.png" alt="TBD Public ingress diagram" style="width:100%;" >}} -->
 
+<div class="alert alert-warning">Only the CloudPrem gRPC API endpoints (paths starting with `/cloudprem`) perform mutual TLS authentication. Exposing any other endpoints through the public ingress introduces a security risk, as those endpoints would be accessible over the internet without authentication. Always restrict non-gRPC endpoints to the internal ingress. </div>
+
+### IP Ranges
+The Datadog control plane and query services connect to CloudPrem clusters using a set of fixed IP ranges, which can be retrieved for each Datadog site from the Datadog IP Ranges API, specifically under the "webhooks" section. For example, to fetch the IP ranges for the datadoghq.eu site, you can run:
+```
+curl -X GET "https://ip-ranges.datadoghq.eu/" \
+      -H "Accept: application/json" |
+      jq '.webhooks'
+```
+
+
 ## Internal ingress
 
 The internal ingress enables log ingestion from Datadog Agents and other log collectors within your environment through HTTP.
@@ -96,50 +101,6 @@ rules:
 
 <!-- {{< img src="path/to/your/image-name-here.png" alt="TBD Internal ingress using NGINX ingress controller" style="width:100%;" >}} -->
 
-## Supported ingress controllers
-
-<!-- This is generated by Cursor, if this is incorrect, I'll delete this entire section -->
-
-CloudPrem supports various ingress controllers to accommodate different infrastructure requirements and preferences:
-
-{{% collapse-content title="NGINX Configuration" level="h4" expanded=false %}}
-```yaml
-ingress:
-  internal:
-    create: false
-  nginx:
-    enabled: true
-    annotations:
-      kubernetes.io/ingress.class: nginx
-      nginx.ingress.kubernetes.io/ssl-redirect: "true"
-```
-{{% /collapse-content %}}
-
-{{% collapse-content title="HAProxy Configuration" level="h4" expanded=false %}}
-```yaml
-ingress:
-  internal:
-    create: false
-  haproxy:
-    enabled: true
-    annotations:
-      kubernetes.io/ingress.class: haproxy
-```
-{{% /collapse-content %}}
-
-{{% collapse-content title="Traefik Configuration" level="h4" expanded=false %}}
-```yaml
-ingress:
-  internal:
-    create: false
-  traefik:
-    enabled: true
-    annotations:
-      kubernetes.io/ingress.class: traefik
-```
-{{% /collapse-content %}}
-
-
 ## Further reading
 
 {{< partial name="whats-next/whats-next.html" >}}