Skip to content

gmurphy-finops-toolkit-xcloud-gcp-pipeline #1425

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: features/xcloud
Choose a base branch
from
Open

gmurphy-finops-toolkit-xcloud-gcp-pipeline #1425

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 118 additions & 0 deletions src/templates/finops-hub/modules/dataFactory.bicep
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ param tagsByResource object = {}
@description('Optional. Enable public access.')
param enablePublicAccess bool

@description('The wildcard folder path for the GCP billing data.')
param gcpBillingWildcardFolderPath string

//------------------------------------------------------------------------------
// Variables
//------------------------------------------------------------------------------
Expand Down Expand Up @@ -4743,6 +4746,121 @@ resource pipeline_ExecuteIngestionETL 'Microsoft.DataFactory/factories/pipelines
}
}

//------------------------------------------------------------------------------
// import_gcp_billing_data pipeline
//------------------------------------------------------------------------------
resource importGCPBillingDataPipeline 'Microsoft.DataFactory/factories/pipelines@2018-06-01' = {
name: 'import_gcp_billing_data'
parent: dataFactory
properties: {
activities: [
{
name: 'convert gcp csv'
type: 'Copy'
dependsOn: [
{
activity: 'Delete Target'
dependencyConditions: [
'Completed'
]
}
]
policy: {
timeout: '0.12:00:00'
retry: 0
retryIntervalInSeconds: 30
secureOutput: false
secureInput: false
}
typeProperties: {
source: {
type: 'DelimitedTextSource'
storeSettings: {
type: 'GoogleCloudStorageReadSettings'
recursive: true
wildcardFolderPath: gcpBillingWildcardFolderPath // Parameterized value
wildcardFileName: '*'
enablePartitionDiscovery: false
}
formatSettings: {
type: 'DelimitedTextReadSettings'
}
}
sink: {
type: 'ParquetSink'
storeSettings: {
type: 'AzureBlobFSWriteSettings'
}
formatSettings: {
type: 'ParquetWriteSettings'
}
}
enableStaging: false
translator: {
type: 'TabularTranslator'
mappings: [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mappings should be stored in the config/schema folder. We also need to be able to detect and load the correct schema version to account for future updates.

{
source: {
name: 'BillingAccountId'
type: 'String'
physicalType: 'String'
}
sink: {
name: 'BillingAccountId'
type: 'String'
physicalType: 'String'
}
}
// Add all other mappings here as per your JSON
]
typeConversion: true
typeConversionSettings: {
allowDataTruncation: true
treatBooleanAsNumber: false
}
}
}
inputs: [
{
referenceName: 'GCSbillingexportBucket'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing the definition for the dataset and linked service.

We should store the secret in KeyVault.
There's some logic to only deploy KeyVault in the remote hub scenario so that also needs to be updated - likely by adding a checkbox to the UI to optionally deploy KeyVault when deploying the toolkit.

type: 'DatasetReference'
}
]
outputs: [
{
referenceName: 'gcp_ingestion'
type: 'DatasetReference'
}
]
}
{
name: 'Delete Target'
type: 'Delete'
dependsOn: []
policy: {
timeout: '0.12:00:00'
retry: 0
retryIntervalInSeconds: 30
secureOutput: false
secureInput: false
}
typeProperties: {
dataset: {
referenceName: 'gcp_ingestion'
type: 'DatasetReference'
}
enableLogging: false
storeSettings: {
type: 'AzureBlobFSReadSettings'
recursive: true
enablePartitionDiscovery: false
}
}
}
]
}
}

//------------------------------------------------------------------------------
// Start all triggers
//------------------------------------------------------------------------------
Expand Down