Config Management

Section 3.3: Configuration and Secrets Management Strategy (Securing Recipes & Special Ingredients)**

• 3.3.1 Why Robust Config & Secrets Management is Crucial in MLOps

  • Security (protecting API keys, database credentials).

  • Reproducibility (tracking exact configurations used for runs).

  • Environment Management (different settings for Dev, Staging, Prod).

  • Collaboration (sharing non-sensitive configs safely).

• 3.3.2 Types of Configurations in an ML Project

  • Data Sources: Paths to S3 buckets, database connection strings (excluding credentials).

  • Feature Engineering Parameters: Binning strategies, embedding dimensions, list of features to use.

  • Model Training Hyperparameters: Learning rate, batch size, number of epochs, model architecture details.

  • Pipeline Parameters: Instance types for jobs, resource allocations, trigger schedules.

  • Infrastructure Settings: VPC IDs, subnet IDs, security group IDs (managed by IaC but might be referenced).

  • API Endpoints: URLs for external services (e.g., LLM provider).

  • Secrets: Database passwords, API keys (LLM, Cloud provider services), private certificates.

• 3.3.3 Common Approaches to Configuration Management

  • Configuration Files (e.g., YAML, JSON, TOML, INI):

    • Pros: Human-readable, easy to edit, commonly supported by libraries, good for version control (Git).

    • Cons: Can become unwieldy for complex projects, managing environment-specific overrides needs a strategy.

    • Strategy: Use base config files and environment-specific override files (e.g., config_base.yaml, config_staging.yaml, config_prod.yaml). Load base then merge environment-specific.

  • Environment Variables:

    • Pros: Standard way to pass configs in containerized environments (Docker, Kubernetes) and CI/CD systems. Easy to set dynamically.

    • Cons: Not ideal for complex/nested structures. Managing many variables can be cumbersome. Less auditable directly within the application codebase if not explicitly loaded from a file.

  • Dedicated Config Management Tools (e.g., HashiCorp Consul, AWS AppConfig):

    • Pros: Centralized management, dynamic updates without redeployment, versioning, access control.

    • Cons: Adds another tool to the stack, can be overkill for simpler projects.

• 3.3.4 Best Practices for Managing Secrets

  • NEVER commit secrets directly to Git.

  • Use .env files for LOCAL development ONLY, and ensure .env is in .gitignore.

  • Secrets Management Services (The Secure Ingredient Lockbox):

    • Cloud-Native: AWS Secrets Manager, Google Secret Manager, Azure Key Vault.

    • Third-Party: HashiCorp Vault.

    • How they work: Store secrets encrypted. Applications/Pipelines fetch secrets at runtime using IAM roles/service accounts with appropriate permissions.

  • Injecting Secrets into Pipelines/Applications:

    • CI/CD systems (e.g., GitHub Actions Secrets) can securely inject secrets as environment variables into build/deployment steps.

    • Orchestrators (e.g., Airflow Connections, Kubernetes Secrets) can manage secrets for pipeline tasks.

    • Applications (e.g., FastAPI service) fetch from secrets manager at startup or per request (with caching).

• 3.3.5 Configuration and Secrets Strategy for “Trending Now”

  • Non-Sensitive Configurations:

    • Use YAML files stored in the mlops/config/ directory.

    • Example: config_base.yaml for common settings.

    • config_dev.yaml, config_staging.yaml, config_prod.yaml for environment-specific overrides (e.g., S3 bucket names, Airflow connection IDs, LLM model choice).

    • These will be version-controlled with Git.

    • Pipelines and applications will load the appropriate config based on an environment variable (e.g., APP_ENV=staging).

  • Secrets Management:

    • Local Development (Dev): Use a .env file (added to .gitignore) to store API keys (LLM provider, AWS keys for local DVC/S3 interaction if needed). The application/scripts will load from this .env file if APP_ENV=dev.

    • Staging & Production:

      • LLM API Key: Store in AWS Secrets Manager.

      • (If needed) Database credentials for Airflow metadata DB (if self-hosted on EC2): Store in AWS Secrets Manager.

      • AWS Service credentials (for S3, App Runner, etc.): Handled via IAM Roles attached to the EC2 instances (for Airflow workers/scheduler) or App Runner service. This is the preferred method for AWS service-to-service communication.

      • GitHub Actions Secrets: Used to store AWS credentials needed for Terraform to deploy infrastructure and for Airflow/App Runner to pull from ECR if using private images.

  • Loading Configs in the Application/Pipelines:

    • Python scripts (in Airflow tasks, FastAPI) will use a helper function to:

      1. Load config_base.yaml.

      2. Identify current environment (from APP_ENV environment variable).

      3. Load and merge the corresponding config_<env>.yaml.

      4. If APP_ENV=dev, load secrets from .env.

      5. For other environments, fetch necessary secrets from AWS Secrets Manager using boto3 and the IAM role associated with the execution context.

---