Overview
The Deployment Engine provides a streamlined way to deploy the OSDU® Data Platform instances into a Google Cloud project. This guide walks you through the complete deployment process, from initial setup to accessing running the OSDU® Data Platform environment.
The instruction has the following:
- How to navigate the Deployment Engine interface;
- Step-by-step deployment process;
- Configuration options and requirements;
- How to access and manage deployed instances.
User Management
Adding Users to an organization
To share access to deployments with team members, there is available an option to manage users through the Users Management page. This allows multiple people in your organization to view and manage deployments associated with your Google Cloud procurement ID.
Understanding Procurement Accounts
A "Users Management Procurement account" in this system refers to an account that manages a user's access to services based on their purchased plans. This system supports two main types of procurement:
- Marketplace (MKPL) Procurement: These accounts are integrated with an external cloud commerce procurement service (like Google Cloud's Procurement Service). Users typically get these accounts by subscribing to a product through a marketplace. The system then checks with the external service to confirm access rights and manage the account's status.
- Non-Marketplace (Non-MKPL) Procurement: These are internal or self-managed procurement accounts. Users can get these by signing up directly within the application. The system stores and manages these accounts in its own database.
How to get a procurement account:
- For Marketplace Procurement: Users usually subscribe to a product through a marketplace (e.g., Google Cloud Marketplace). After subscribing, an approval process starts within the system to activate the account and assign a service plan.
- For Non-Marketplace Procurement: Users can sign up directly within the application. This "Self Sign-up Approval" process activates the account internally and assigns a basic plan (e.g., "base").
Adding a Procurement User
What is it for? The "Add Procurement User" feature allows an existing procurement account holder or an administrator to link new users to an already established procurement account. This is especially useful for:
- Team Collaboration: Allowing multiple team members within an organization to access and use the services covered by a single subscription.
- Centralized Billing: Keeping all billing under one procurement account while giving access to several individuals.
- Simplified Onboarding: Quickly granting access to new team members without them needing to go through a separate sign-up process.
What permissions will these users have? When a user is added to an existing procurement account, they will generally get the same permissions and access rights that come with the plan of that procurement account. The system stores the new user's email and connects them to the existing procurement_account_id and procurement_type. This means:
- Shared Plan Access: The added users will have access to the features and services defined by the procurement account's plan. For example, if the procurement account has a "premium" plan, all linked users will get premium features.
- No Granular Role-Based Access: The system does not currently support detailed role-based access control within the procurement account itself (e.g., some users being "admins" and others "viewers" of the procurement). Access is shared based on the overall procurement plan. Any more specific role-based access would likely be managed at the application level, outside of this direct linking.
- Visibility of other users: Users within the same procurement account can typically see other users associated with that same account.
In summary, adding a procurement user extends the benefits of an existing service subscription to more people, giving them access according to the subscribed plan.
Important Rules for Adding Users:
- Marketplace Only: Only users with a
Google Marketplace(MKPL) procurement type can add new users. If your account isNON_MKPL, you will receive a403 Forbiddenerror, meaning this feature is not available for your account type. - Matching Email Domains: The email domain of the user you are adding must match the email domain of the user making the request. This prevents adding users from different organizations or personal email addresses to a corporate account. If the domains don't match, you'll get a
400 Bad Requesterror. - Linking Process: If all checks pass, the system links the new user's email to the existing procurement account. It also checks if the new user's email is already linked to another procurement account; if so, an error will occur. A new record is added to the database, connecting the new user's email with the procurement account's ID, type, and plan.
Permissions of Added Users:
Users added through this function will:
- Inherit the plan: They get access to the services and features defined by the
planof the procurement account they are linked to. - No administrative rights: They do not automatically get administrative rights over the procurement account itself (e.g., they cannot add or remove other users unless they also have an active Marketplace procurement and meet the domain criteria). Their main role is to use the services.
- Visibility: They will be visible in the list of users associated with that procurement account.
"Forget Me" Functionality
The "Forget me" functionality is designed to handle requests for user data deletion, often to comply with privacy rules. This process involves a two-step approval to ensure data is deleted intentionally.
-
Ask to be Forgotten(User Request):- What it does: When a user starts this request, the system records the time of their request in their user account record. This shows the user's intent to be forgotten but doesn't immediately delete any data.
- Effect: The user's data remains, but their account record is updated to show that a "forget me" request is pending.
-
Cancel Forget Me(User Revocation):- What it does: A user can cancel their "forget me" request at any time after starting it. This removes the request timestamp from their user account record.
- Effect: The pending "forget me" request is canceled, and the user's account returns to its normal state, with no data deleted or hidden.
-
Approve Forget Me Request(System/Admin Approval):- What it does: This is the crucial step where data deletion or hiding actually happens. It's usually an action taken by an administrator or an automated process that confirms the user's request after a waiting period.
- Conditions:
- The user must have an active procurement account.
- The "ask to be forgotten" request must have been made.
- A minimum time (currently 24 hours) must have passed since the "ask to be forgotten" request. This grace period prevents immediate deletion. If this time hasn't passed, an error will occur.
- Effect: This function takes different actions based on the type of procurement account:
Impact on Deployments, Data, and Other Effects
The results of approving a "Forget me" request differ significantly based on whether the user has a Non-Marketplace (NON_MKPL) or Google Marketplace (GOOGLE_MKPL) procurement account.
For Non-Marketplace (NON_MKPL) Users:
If the user has a NON_MKPL procurement account, the system performs a more complete deletion of associated data:
- Deployments: All deployments connected to the user's procurement account are deleted.
- Procurement Account: The
NON_MKPLprocurement record itself is deleted from the database. - EULA Acceptance: Any records of the user accepting the End User License Agreement (EULA) are deleted.
- Service Accounts: Any service accounts linked to the user's procurement account are deleted.
- User Procurement Record: The user's
UserProcurementrecord (which links their Google ID/email to the procurement account) is deleted. - Alerts: An alert is sent to administrators to notify them that a Non-Marketplace user has approved forgetting themselves, with a recommendation to double-check for active service accounts.
For Google Marketplace (GOOGLE_MKPL) Users:
If the user has a GOOGLE_MKPL procurement account, the system takes a less destructive approach. It mainly hides personal information (PII) rather than deleting it entirely, especially for resources managed by the external marketplace:
- Deployments: Instead of deletion, the user's email address associated with deployments (e.g.,
admin_user_email,created_by_email) is hidden. The original email is replaced with a obscured version (e.g.,t***[email protected]). This means the deployments remain, but the direct link to the user's personal information is removed. - Service Accounts: If the user's email is recorded as the creator for a service account, that email is also hidden. The service account itself is not deleted.
- User Procurement Record: Similar to Non-Marketplace users, the user's
UserProcurementrecord is deleted. - External Marketplace: The system does not directly interact with the Google Cloud Marketplace to "forget" the user there. It's assumed that the marketplace has its own data retention and deletion policies. The actions taken by this system focus on the data it directly manages.
- No specific alerts: Unlike Non-Marketplace users, there isn't a specific alert sent for Google Marketplace users being forgotten. This is likely because the data is hidden rather than fully deleted, and external systems manage the core procurement.
Other Effects:
- Email Obscuration: Email addresses are transformed (e.g.,
[email protected]becomest***[email protected]). This is a key way to anonymize personal information while keeping some structural details. - Efficient Operations: Many deletion and hiding tasks are performed at the same time, showing that these operations can be complex and are designed to run efficiently.
In summary, the "Forget me" functionality is a strong system for data privacy. For Non-Marketplace users, it leads to a complete deletion of associated data and deployments. For Google Marketplace users, it focuses on hiding personal information within the application's managed data, respecting the external nature of the core procurement relationship.
Getting Started
Main Dashboard
The main page of the Deployment Engine displays all environments and deployments statuses associated with available organizations via Google Cloud procurement IDs or other organizations. There are following available actions:
- View existing deployments with their current statuses;
- Create new OSDU instances using the "Create Deployment" button;
- Monitor deployment progress and access to the deployed services.
Image DE.01 – Main dashboard overview
Navigation Options:
- Click on a Project ID to view detailed deployment status and service endpoints;
- Click "Create Deployment" to start a new OSDU® Data Platform deployment.
Creating a New Deployment
Step 1: Choose a version of OSDU® Data platform
The current version of Deployment Engine supports M23, M24, M25 OSDU® Data platform versions.
[!NOTE] All features, additional services, optional services, etc available for this product are actual to the LATEST available version of OSDU® Data platform.
Step 2: Choose Implementation Type
When creating a new deployment, you'll first select the type of the OSDU® Data Platform implementation that best fits your needs.
Available Options:
- Google Cloud Implementation: Enterprise-grade deployment with full Google Cloud integration
- Community Implementation: Open-source based deployment with Community Implementation features
Image DE.02 – Selecting the OSDU® Data Platform implementation type
Step 3: Select Deployment Tier
Choose the infrastructure tier that matches your requirements. Each tier is pre-configured with specific resource allocations and capabilities.
Image DE.03 – Available deployment tiers and configurations
Tier Selection Considerations:
- Development/Testing: Choose lower tiers for cost efficiency
- Production: Select higher tiers for performance and reliability
- Features: Review included services and capabilities for each tier
Step 4: Configure Regional and Service Options
Complete your deployment configuration by selecting:
- Region: Choose the Google Cloud region closest to your users
- Additional Services: Enable optional services like Admin UI
- Advanced Configuration: Set up custom networking or domain options
Image DE.04 – Regional and service configuration options
Project Setup and Validation
Step 5: Project Configuration
Provide your Google Cloud project details and administrative information:
- Google Cloud Project ID: The target project for a OSDU® Data Platform deployment
- Admin Email: Administrative contact for certificates and notifications
Image DE.05 – Project ID and admin email configuration
Step 6: Service Account Permissions
The Deployment Engine automatically generates a unique service account for your project. This service account requires specific IAM roles to deploy and manage the OSDU® Data Platform resources.
Image DE.06 – Required IAM roles for deployment
Setting Up Permissions:
You can assign these roles through:
- Google Cloud Console IAM interface (manual assignment)
- Google Cloud Shell commands (automated assignment)
- gcloud CLI scripts (bulk assignment)
The exact roles and setup commands are provided in the interface based on your selected configuration.
Step 7: Additional Service Configuration
If you selected additional services like Admin UI, you'll need to provide OAuth configuration details:
Image DE.07 – Admin UI OAuth configuration
Step 8: Permission Verification
The system automatically verifies that all required permissions are properly configured before proceeding with deployment:
Image DE.08 – Automated permission verification
Deployment Review and Launch
Step 9: Configuration Overview
Review all your deployment settings before launching. This summary shows:
- Implementation type and tier
- Regional configuration
- Enabled services and features
- Project and administrative details
Image DE.09 – Final configuration review
Once you confirm the configuration, click "Deploy" to begin the automated deployment process.
Monitoring Your Deployment
Deployment Progress
During deployment, you can monitor progress and view estimated completion time. Typical deployment duration is approximately 90 minutes for Dev tier configurations.
Image DE.10 – Active deployment monitoring
What Happens During Deployment:
- Infrastructure provisioning
- the OSDU® Data Platform services installation
- Security configuration
- SSL certificate generation
- Service endpoint setup
Deployment Completion
When deployment completes successfully, the status changes to "Deployed" and all service endpoints become available:
Image DE.11 – Successful deployment with available endpoints
Available Information:
- Service Endpoints: Direct URLs to access the OSDU® Data Platform APIs and tools
- Administrative Access: Credentials and connection details
- Monitoring Tools: Links to logs and system monitoring
- Documentation: Service-specific usage guides
Managing Your Deployment
Removing a Deployment
To remove an OSDU® Data Platform deployment while preserving data, use the "Remove Deployment" option:
Image DE.12 – Deployment removal process
What Gets Removed:
- OSDU® Data Platform services
- Compute infrastructure
- Network configurations
What's Preserved:
- Data storage buckets
- Persistent data volumes
Deployment Removal Completion
The removal process typically completes within 15-30 minutes:
Image DE.13 – Completed deployment removal
Complete Project Cleanup
For complete resource removal including all data, use the "Clean-up" option. This action:
- Removes ALL resources from the Google Cloud project
- Cannot be undone - all data will be permanently deleted
- Includes a 10-minute delay to prevent accidental execution
Image DE.14 – Complete project cleanup warning
Post-Cleanup Status
After cleanup completion, the Google Cloud project will be empty of all OSDU® Data Platform related resources:
Image DE.15 – Project state after complete cleanup
Next Steps
After successful deployment:
- Access Services: Use the provided endpoints to connect to the OSDU® Data Platform APIs
- Configure Users: Set up authentication and user access through admin tools
- Upload Data: Begin ingesting your energy data through file and storage services
- Create Workflows: Develop data processing pipelines using the workflow service
- Monitor Performance: Use built-in monitoring tools to track system health
For detailed API usage and development guidance, refer to the OSDU® Data Platform API Documentation.