Item metadata return (#13)

* Docs updates

* Update to python 3.12 and latest power tools

* Add support for Personalize item metadata return

Author: james-jory

README.md

@@ -40,7 +40,7 @@ See the [Caching](./docs/caching.md) documentation for details.
 
 ## Inference item metadata
 
-Machine learning recommenders like [Amazon Personalize](https://aws.amazon.com/personalize/) often provide only a list of recommended item IDs in their inference responses. However, client applications typically need rich metadata for each item (e.g., name, description, price, category, genre, image URL, media URL, detail URL, etc) to be able to render recommended items in their UIs. This solution provides the option to store item metadata either in a local database right where recommendations are retrieved or in [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) (very large item catalogs). Both options provide very low-latency retrieval of item metadata that is automatically injected into recommender responses before returning them to your applications. This capability allows you to focus more on displaying recommendations in your client applications rather than building undifferentiating middleware to merge item catalog information into your recommendation responses.
+Machine learning recommenders like [Amazon Personalize](https://aws.amazon.com/personalize/) often provide only a list of recommended item IDs in their inference responses. However, client applications typically need rich metadata for each item (e.g., name, description, price, category, genre, image URL, media URL, detail URL, etc) to be able to render recommended items in their UIs. Amazon Personalize does support returning item metadata with responses for [campaigns](https://docs.aws.amazon.com/personalize/latest/dg/campaigns.html#create-campaign-return-metadata) and [recommenders](https://docs.aws.amazon.com/personalize/latest/dg/creating-recommenders.html#create-recommender-return-metadata). However, this capability is limited to returning up to 10 columns per item and up to a maximum of 50 items per response (as opposed to 500 items without item metadata). There is also an additional cost for enabling item metadata in Amazon Personalize responses. If these limits are not enough to suppport your use case, the Personalization APIs project provides an item metadata sidecar implementation that stores item metadata either in a local database right where recommendations are retrieved in the API layer (for small and modest sized item catalogs) or in [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) (for very large item catalogs). Both options provide very low-latency retrieval of item metadata that is automatically injected into recommender responses before returning them to your applications. This capability allows you to focus more on displaying recommendations in your client applications rather than building undifferentiating middleware to merge item catalog information into your recommendation responses.
 
 See the [Inference item Metadata](./docs/item_metadata.md) documentation for details.
 
@@ -162,6 +162,8 @@ The solution retrieves its configuration details from [AWS AppConfig](https://aw
 
 ### Preparing and uploading your item metadata to S3
 
+If you choose to use one of the [item metadata sidecar implementations](./docs/item_metadata.md), you will need to prepare and upload your item metadata to an S3 bucket. This step is not required if you're using item metadata returned by Amazon Personalize.
+
 Once your recommenders are configured in AppConfig, it's time to upload your item metadata to the S3 bucket created by the solution. The uploaded item metadata file contains the attributes of each item that are needed to render them in your client applications (e.g., item name, description, category, genre, image URL, price, media URL, etc). Your item metadata file needs to be contained in a single JSON file in the [JSON Lines](https://jsonlines.org/) format. The solution will automatically detect when you upload your metadata file to the proper folder in the S3 bucket and load your metadata into the datastore setup in your configuration. Be sure to update and deploy your configuration with the appropriate item metadata configuration before uploading your item metadata. See the [item metadata documentation](./docs/item_metadata.md) for details.
 
 ### Deploy OAuth2 edge authentication resources (**OAuth2-Cognito only!**)
@@ -222,7 +224,7 @@ If you used a stack name other than `personalization-apis`, substitute your stac
 
 ## FAQs
 
-***Q: I uploaded item metadata to the staging bucket but I do not see the item metadata in my API responses. What's wrong?***
+***Q: I'm using one of the item metadata sidecar implementations and I uploaded item metadata to the staging bucket but I do not see the item metadata in my API responses. What's wrong?***
 
 ***A:*** Some things to check:
 - Make sure that you added `inferenceItemMetadata` configuration(s) to the namespaces in the solution's configuration in AppConfig **and** that you deployed these changes. **This must be done before you upload your item metadata file to the staging bucket.** Deploying a configuration change in AppConfig requires creating a new hosted configuration version, starting a deployment of that version, and waiting for it to be deployed to the solution. You can verify that the updated configuration is active by inspecting the `X-Personalization-Config-Version` response header from the API. This header's value is the value of the `version` field in your configuration.

docs/item_metadata.md

@@ -1,8 +1,73 @@
 # Personalization APIs Inference Item Metadata
 
-Inference item metadata is used to decorate responses from your origin recommenders with the information needed to fully render recommendations in client applications. You upload your inference item metadata to the S3 staging bucket created by the Personalization APIs deployment. The name of this bucket can be found in the CloudFormation output parameters (look for the `StagingBucket` parameter). When you upload your inference item metadata (described in detail below) to the appropriate folder in the staging bucket (based on the namespace key), an AWS Lambda function is invoked that updates the appropriate datastore(s) based on the configuration described below. **Therefore, you must update your configuration with inference item metadata configuration before uploading your item metadata to the staging bucket.**
+Having access to item metadata (item name, price, description, category, brand, genre, etc) directly in recommender responses allows applications to more easily render recommendations in their UI. However, many primitive recommendation systems, such as Amazon Personalize, provide only item IDs and not item metadata in their responses. Although Amazon Personalize campaigns and recommenders can be configured to return item metadata, there are limits on the number of columns (10) and items (50) that can be returned when item metadata for responses is enabled. There is also an additional charge to have item metadata returned in Personalize responses (see the [pricing page](https://aws.amazon.com/personalize/pricing/) for details).
 
-## Local DBM datastore
+This project provides the ability to leverage the native capability of underlying recommenders from Amazon Personalize to provide item metadata as well as an alternative sidecar item metadata storage and retrievel mechanism that injects item metadata in responses before they are returned by the API. See the detailed instructions below on each mechanism.
+
+** Keep in mind that item metadata in responses can be controlled by the `decorateItems` query string parameter to API requests. By default item metadata is enabled (if configured as described below) but it can be disabled at the request level by setting `decorateItems` to 0/false/no in your requests.
+
+## Personalize item metadata
+
+If you're using Amazon Personalize campaigns and/or recommenders with a deployment of this project and your use case is within the 10 metadata returned column and 50 item limit, then using the native Amazon Personalize item metadata return feature may be the best option. Some configuration is still required to use this approach.
+
+First, you will have to enable metadata to be returned from your Amazon Personalize campaigns and/or recommenders. This can be done in the Amazon Personalize console or API when creating your campaigns or recommenders (for the API/SDK, see the `campaignConfig.enableMetadataWithRecommendations` parameter for the [CreateCampaign](https://docs.aws.amazon.com/personalize/latest/dg/API_CreateCampaign.html) and [UpdateCampaign](https://docs.aws.amazon.com/personalize/latest/dg/API_UpdateCampaign.html) APIs for campaigns and the `recommenderConfig.enableMetadataWithRecommendations` parameter for the [CreateRecommender](https://docs.aws.amazon.com/personalize/latest/dg/API_CreateRecommender.html) and [UpdateRecommender](https://docs.aws.amazon.com/personalize/latest/dg/API_UpdateRecommender.html) APIs for recommenders).
+
+Once metadata has been enabled for your Amazon Personalize campaigns/recommenders, you can then configure the Personalization APIs to request item metadata when making inference calls. This is done with the `inferenceMetadata` section in the Personalization APIs configuration. Below is an example of using item metadata provided by Amazon Personalize (`type` of `personalize`) and specifying that the columns `NAME`, `DESCRIPTION`, `PRICE`, and `CATEGORY` should be requested from Personalize and returned by the API response. The column names must match columns in your Amazon Personalize items dataset schema.
+
+```json
+{
+    "namespaces": {
+        "my-app-1": {
+            "inferenceItemMetadata": {
+                "type": "personalize",
+                "itemColumns": [
+                    "NAME",
+                    "DESCRIPTION",
+                    "PRICE",
+                    "CATEGORY"
+                ]
+            },
+            "recommenders": {
+            }
+        }
+    }
+}
+```
+
+- `namespaces.{NAMESPACE_KEY}.inferenceItemMetadata.type`: Must be `"personalize"` (required to use Amazon Personalize provided item metadata).
+- `namespaces.{NAMESPACE_KEY}.inferenceItemMetadata.itemColumns`: Array of column names to request Personalize to return in responses (required).
+
+Although the example above shows the `inferenceItemMetadata` being specified at the namespace level, it can also be specified at the recommender or variation level instead. This allows you override item metadata configurations at different levels of the configuration.
+
+An API response that includes item metadata from Amazon Personalize would look something like this where the `metadata` dictionary in the response includes the item metadata columns that were configured in `inferenceItemMetadata.itemColumns`:
+
+
+```json
+{
+  "itemList": [
+    {
+      "itemId": "f6231107-7050-44ea-ac6a-dcb09f4a0b33",
+      "score": 0.298052,
+      "metadata": {
+        "name": "Camping Lamp",
+        "category": "outdoors",
+        "description": "Be sure to bring this camping lamp with you to the outdoors",
+        "price": 19.99
+      }
+    }
+  ]
+}
+```
+
+Note that Amazon Personalize will convert snake case column names to camel case in the response. For example, the schema column `BRAND_NAME` will be converted to `brandName` in the response. In addition, categorical field values will be returned as formatted when they were ingested (i.e., `ONE|TWO|THREE`) into Amazon Personalize rather than being returned as an array of values (i.e., `["ONE","TWO","THREE"]`)
+
+## Sidecar item metadata storage, retrieval, and injection
+
+As mentioned above, the Personalization APIs project also supports an item metadata sidecar feature whereby item metadata is injected into recommender responses before they are returned from the API layer. There are currently two sidecar implementations supported by the project.
+
+To take advantage of the sidecar item metadata capability, you upload your inference item metadata to the S3 staging bucket created by the Personalization APIs deployment. The name of this bucket can be found in the CloudFormation output parameters (look for the `StagingBucket` output parameter). When you upload your inference item metadata (described in detail below) to the appropriate folder in the staging bucket (the folder name is based on the namespace key), an AWS Lambda function is invoked that automatically updates the appropriate sidecar datastore(s) based on the configuration described below. **Therefore, it's vital that you update your configuration with inference item metadata configuration before uploading your item metadata to the staging bucket.**
+
+### Local DBM datastore
 
 Declares that item metadata should be managed in a local DBM datastore for a namespace that is automatically downloaded from S3 and stored on the local Lambda volume of the API origin function. This option provides the lowest possible latency for item metadata decoration (~1-3ms) but is not suitable for very large item catalogs for when a large number of namespaces are served by the same Personalization APIs deployment.
 
@@ -24,7 +89,7 @@ Declares that item metadata should be managed in a local DBM datastore for a nam
 - `namespaces.{NAMESPACE_KEY}.inferenceItemMetadata.type`: Must be `"localdb"` (required).
 - `namespaces.{NAMESPACE_KEY}.inferenceItemMetadata.syncInterval`: How often to sync DBM files from the S3 staging bucket in seconds (optional, default is 300 seconds).
 
-## Amazon DynamoDB tables
+### Amazon DynamoDB tables
 
 Declares that a Amazon DynamoDB table should be used to query for item metadata for a particular namespace. The table can optionally be automatically provisioned by the Personalization APIs solution (when the configuration changes) or you can create the table directly (see `autoProvision` field). The table name is derived based on a concatenation of `PersonalizationApiItemMetadata_` and the namespace key. So for the example configuration fragment below, the table name would be `PersonalizationApiItemMetadata_my-app1`. Therefore, if you create the DynamoDB table yourself, you must use the apppropriate table name.
 
@@ -54,7 +119,7 @@ Declares that a Amazon DynamoDB table should be used to query for item metadata
 - `namespaces.{NAMESPACE_KEY}.inferenceItemMetadata.provisionedThroughput.readCapacityUnits`: Read capacity units (required if `billingMode` is `"PROVISIONED"`).
 - `namespaces.{NAMESPACE_KEY}.inferenceItemMetadata.provisionedThroughput.writeCapacityUnits`: Write capacity units (required if `billingMode` is `"PROVISIONED"`).
 
-## Preparing and Uploading Inference Item Metadata to S3
+### Preparing and Uploading Inference Item Metadata to S3
 
 When the Personalization APIs solution is deployed, an S3 bucket is created that is used as a staging area for uploading item metadata. The bucket name can be determined from the CloudFormation output parameter named `StagingBucket`. To provide inference item metadata for your items to the Personalization APIs solution, create a [JSON Lines](https://jsonlines.org/) file for each namespace where each file contains metadata for every item that could be recommended by recommenders for that namespace. All of the fields for an item's ID will be used to decorate the response for the item. For example, the following JSONL fragment includes metadata for 6 products for a sample e-commerce item catalog.
 

samconfig.toml

@@ -25,5 +25,7 @@ stack_name = "p13n-apis-noauth"
 region = "us-east-1"
 confirm_changeset = true
 capabilities = "CAPABILITY_IAM"
-parameter_overrides = "ApplicationName=\"P13n-APIs-noauth\" EnvironmentName=\"prod\" TimeZone=\"UTC\" AuthenticationScheme=\"None\" CreateCognitoResources=\"No\" ApiEntryPointType=\"API-Gateway-HTTP\" CacheScheme=\"CloudFront\" GenerateConfigDatasetGroupNames=\"retaildemostore-products,personalize-poc-movielens-20m\""
+parameter_overrides = "ApplicationName=\"P13n-APIs-noauth\" EnvironmentName=\"prod\" TimeZone=\"UTC\" AuthenticationScheme=\"None\" CreateCognitoResources=\"No\" ApiEntryPointType=\"API-Gateway-HTTP\" CacheScheme=\"CloudFront\" GenerateConfigDatasetGroupNames=\"retaildemostore-products,personalize-poc-movielens-20m\" CreateSwaggerUI=\"Yes\""
 image_repositories = []
+resolve_s3 = true
+s3_prefix = "p13n-apis-noauth"

src/config_validator_function/openapi_template.json

@@ -155,7 +155,7 @@
             },
             "decorateItems": {
                 "name": "decorateItems",
-                "description": "Controls whether item IDs returned from the origin recommender are decorated with [item metadata](./item_metadata.md) in the response. The values `1`, `yes`, and `true` (case-insensitive) will be interpreted as `true`. Any other value will be interpreted as `false`. The default value is `true`. Since this parameter is included in the cache key for response caching, it is important to use consistent values across your client applications (to maximize cache hit rate).",
+                "description": "Controls whether item IDs returned from the origin recommender are decorated with item metadata in the response. The values `1`, `yes`, and `true` (case-insensitive) will be interpreted as `true`. Any other value will be interpreted as `false`. The default value is `true`. Since this parameter is included in the cache key for response caching, it is important to use consistent values across your client applications (to maximize cache hit rate).",
                 "in": "query",
                 "required": false,
                 "schema": {

src/config_validator_function/personalization_apis_config_schema.json

@@ -176,7 +176,7 @@
                                 "type": {
                                     "type": "string",
                                     "enum": [ "personalize-campaign", "personalize-recommender" ],
-                                    "description": ""
+                                    "description": "Recommender type: campaign or domain recommender"
                                 },
                                 "arn": {
                                     "type": "string",
@@ -509,6 +509,25 @@
         "inferenceItemMetadata": {
             "description": "Inference item metadata datastore configuration. Defines how the Personalization APIs store and cache item metadata used to decorate responses.",
             "oneOf": [
+                {
+                    "type": "object",
+                    "description": "Personalize-provided item metadata in the Personalize response. IMPORTANT: the Personalize campaign/recommender must have item metadata return enabled for this to work (see enableMetadataWithRecommendations parameter).",
+                    "required": [ "type" ],
+                    "properties": {
+                        "type": {
+                            "type": "string",
+                            "enum": [ "personalize" ]
+                        },
+                        "itemColumns": {
+                            "type": "array",
+                            "description": "List of column names to request from Personalize campaign/recommender to return in its responses.",
+                            "items": {
+                                "type": "string"
+                            }
+                        }
+                    },
+                    "additionalProperties": false
+                },
                 {
                     "type": "object",
                     "description": "Local DBM database that is built and downloaded to the local Lambda volume from the S3 staging bucket for each API function instance.",