[{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/cloud/","section":"Tags","summary":"","title":"Cloud","type":"tags"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/series/crossplane/","section":"Series","summary":"","title":"Crossplane","type":"series"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/crossplane/","section":"Tags","summary":"","title":"Crossplane","type":"tags"},{"content":"Based on the first post in this series: Crossplane Is Not Terraform in Kubernetes.\nCrossplane Composition Is Not Templating # After the first post, you might think the story is simple. Managed resources replace Terraform providers. Composition replaces Terraform modules. Same declarative model, different YAML structure.\nI thought that too when I first read about Crossplane v2 composition. I had spent years writing Terragrunt modules. A composition looked familiar: you define variables, reference them in YAML templates, and get cloud resources out the other end. Modular HCL in YAML clothing.\nI was wrong. Composition is not a template engine. It is an API framework. The templates are implementation details. The product is a custom Kubernetes resource that lives next to Deployment and Service.\nWhy not just give teams managed resources and let them figure it out?\nYou can. Crossplane managed resources work fine on their own. The provider controller handles reconciliation, state management, and drift correction. Your team creates a Bucket, an SQS Queue, and a QueuePolicy, and everything works.\nThe problem is not capability. It is cognitive load. Every application team that needs an event-driven S3 pipeline must understand three managed resources, their interdependencies, the IAM policy format, and the cross-resource references. That is knowledge your platform team already has. Composition lets you package it into a single XNotification resource that the application team creates in three lines of YAML.1\nThe argument for composition is not about what Crossplane can provision. It is about who holds the complexity.\nSo composition is like a Terraform module? You template some YAML and call it with parameters?\nThis is the question that took me longest to answer honestly. Yes, both let you reuse infrastructure patterns. Yes, both accept inputs and produce resources.\nThe difference is architectural. A Terraform module is a block of configuration you call from another block of configuration. It has no existence outside the run that invokes it. A Crossplane composition is a first-class API endpoint. When you apply an XRD, your custom resource type registers with the Kubernetes API server. Run kubectl api-resources and it appears next to Deployment and Service. Access control is RBAC. Consumers create instances with kubectl apply, the same way they create Pods. There is no init, plan, or apply phase. There is only desired state and reconciliation.2\nA composition has three pieces, and keeping them separate in your head changes how you design:\nCompositeResourceDefinition (XRD) defines the schema of your API. What fields does it accept? Which are required? Composition defines the implementation. What managed resources should Crossplane create when someone uses the API? Composed resources are what the pipeline actually creates. flowchart LR subgraph D[\"API contract\"] XRD[\"CompositeResourceDefinitionXRD\"] end subgraph P[\"Pipeline\"] C[\"Compositionfunction pipeline\"] end subgraph R[\"Managed resources\"] MR1[\"S3 Bucket\"] MR2[\"SQS Queue\"] end XRD --\u003e C C --\u003e MR1 C --\u003e MR2 The XRD is your API contract. The Composition is your implementation. Separate them and you can change how resources are created without breaking consumers. You can version the contract independently of the implementation.2\nShow me one working.\nSay you want a pattern where app teams create an event notification channel: when files land in an S3 bucket, the system pushes a notification to a queue. No SQS policy hand-editing, no IAM setup. The team writes this:\napiVersion: storage.example.org/v1alpha1 kind: XNotification metadata: name: image-uploads spec: region: us-east-2 bucketName: my-app-image-uploads Under the hood, Crossplane provisions an S3 bucket, an SQS queue, and a queue policy that lets S3 send messages to the queue.\nThe XRD # The XRD defines the schema. region and bucketName are required. Everything else can have defaults:\napiVersion: apiextensions.crossplane.io/v2 kind: CompositeResourceDefinition metadata: name: xnotifications.storage.example.org spec: group: storage.example.org names: kind: XNotification plural: xnotifications scope: Namespaced versions: - name: v1alpha1 served: true referenceable: true schema: openAPIV3Schema: type: object properties: spec: type: object required: - region - bucketName properties: region: type: string description: \u0026#34;AWS region\u0026#34; bucketName: type: string description: \u0026#34;Name of the S3 bucket\u0026#34; Every v2 XRD must use apiextensions.crossplane.io/v2, set scope: Namespaced or Cluster, and include at least one version. Only one version can be referenceable: true (the internal storage version). The old claimNames and connectionSecretKeys fields are v1-only and rejected by v2.3\nThe Composition # The composition tells Crossplane what to do when someone creates an XNotification. It uses a pipeline of composition functions. The standard pipeline has two steps: render the managed resources with function-go-templating, then let function-auto-ready mark the XR as ready when all resources are healthy.4\napiVersion: apiextensions.crossplane.io/v1 kind: Composition metadata: name: xnotifications.storage.example.org-v1alpha1 spec: compositeTypeRef: apiVersion: storage.example.org/v1alpha1 kind: XNotification mode: Pipeline pipeline: - step: render functionRef: name: function-go-templating input: apiVersion: gotemplating.fn.crossplane.io/v1beta1 kind: GoTemplate source: Inline inline: template: | --- apiVersion: s3.aws.m.upbound.io/v1beta1 kind: Bucket metadata: annotations: {{ setResourceNameAnnotation \u0026#34;bucket\u0026#34; }} spec: forProvider: region: {{ .observed.composite.resource.spec.region }} providerConfigRef: name: default kind: ClusterProviderConfig --- apiVersion: sqs.aws.m.upbound.io/v1beta1 kind: Queue metadata: annotations: {{ setResourceNameAnnotation \u0026#34;queue\u0026#34; }} spec: forProvider: region: {{ .observed.composite.resource.spec.region }} writeConnectionSecretToRef: name: {{ .observed.composite.resource.metadata.name }}-queue-conn providerConfigRef: name: default kind: ClusterProviderConfig --- {{ $obs := default (dict) $.observed.resources }} {{ $data := dig \u0026#34;queue\u0026#34; (dict) $obs }} {{ $arn := dig \u0026#34;resource\u0026#34; \u0026#34;status\u0026#34; \u0026#34;atProvider\u0026#34; \u0026#34;arn\u0026#34; \u0026#34;\u0026#34; $data }} apiVersion: sqs.aws.m.upbound.io/v1beta1 kind: QueuePolicy metadata: annotations: {{ setResourceNameAnnotation \u0026#34;queue-policy\u0026#34; }} spec: forProvider: region: {{ .observed.composite.resource.spec.region }} queueUrlRef: name: {{ .observed.composite.resource.metadata.name }}-queue policy: | { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Principal\u0026#34;: { \u0026#34;Service\u0026#34;: \u0026#34;s3.amazonaws.com\u0026#34; }, \u0026#34;Action\u0026#34;: \u0026#34;sqs:SendMessage\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;{{ $arn }}\u0026#34; } ] } providerConfigRef: name: default kind: ClusterProviderConfig - step: ready functionRef: name: function-auto-ready Three things happen here:\nBucket: created with the region from the XR. No dependencies. Queue: created with writeConnectionSecretToRef so connection details are automatically published to a Kubernetes Secret. QueuePolicy: references the queue via queueUrlRef, and reads the queue\u0026rsquo;s ARN from Crossplane\u0026rsquo;s observed state to build the IAM policy document. The queueUrlRef creates an ordering dependency: the queue must exist before the policy. The ARN read from $.observed.resources uses a nil-guard pattern (default (dict) ... dig ...) because on the first reconcile pass, observed.resources is empty and dig handles missing keys safely.\nWhere did this model bite you?\nThe pipeline looks clean on paper. Then you apply your first composition and spend an hour decoding an error message that points at the wrong location. Here are the ones that got me.\nThe trim-collapse bug. Go templates use {{- (left trim) to eat whitespace before a directive. If you put {{- range }} on its own line between a YAML key and its list value, it eats the newline separating them. The YAML decoder sees queue: - id: foo instead of queue: on one line and - id: foo on the next. The error message points at the parent key, not your template directive. The fix: drop the left trim. Use {{ range }} (no dash) on its own line. The blank line between the key and the first item is valid YAML.5\nobserved.resources is nil on first reconcile. The first time Crossplane evaluates your composition, no resources exist yet. $.observed.resources is an empty dict, or nil for older function versions. Accessing a key with Go\u0026rsquo;s index panics. Use dig from Sprig instead:\n{{ $obs := default (dict) $.observed.resources }} {{ $data := dig \u0026#34;queue\u0026#34; (dict) $obs }} {{ $arn := dig \u0026#34;resource\u0026#34; \u0026#34;status\u0026#34; \u0026#34;atProvider\u0026#34; \u0026#34;arn\u0026#34; \u0026#34;\u0026#34; $data }} Each line is safe at every level. If the queue has not been created yet, $arn is an empty string and the policy document still renders. It just will not include a real ARN until the next reconcile pass.\nsetResourceNameAnnotation is required. Every composed resource needs {{ setResourceNameAnnotation \u0026quot;key\u0026quot; }} in its metadata annotations. Without it, Crossplane treats the resource as a status update on the XR and never creates it. Also, never set crossplane.io/external-name in your template. The provider manages that automatically. Setting it causes a perpetual diff on every reconcile, and your generation counter climbs until the circuit breaker trips.\nThese are not bugs. They are the shape of a system where the template is not the product. The pipeline has phases, and referencing resource state across phases requires explicit guards.\nTip: Crossplane provides a crossplane render command that runs your composition pipeline locally against a mock XR, no cluster needed:\ncrossplane render xr.yaml composition.yaml functions.yaml \\ --crossplane-version=v2.3.1 The version pin matters: without it, the default CLI rejects v2 XRD schemas. This workflow lets you iterate on template syntax, test nil-guard patterns, and verify output before committing.\nWhat if Go templates are not enough?\nThe example uses function-go-templating because Go templates are familiar to anyone who has written a Helm chart. But Crossplane v2 supports half a dozen composition functions. They all package as OCI images, install the same way, and plug into the same pipeline.6\nPatch and Transform (function-patch-and-transform) ported the v1.x mode to a function. You define YAML resource templates with static field patches from the XR. No conditionals, no loops. Good for small, stable compositions where you just need to copy a few fields.\nGo Templating (function-go-templating) uses Go templates with the Sprig function library. Full control flow with if, range, with. Best when you need conditionals, iteration over arrays, or the nil-guard patterns above. The tradeoff is verbose syntax and cryptic error messages.\nPython and KCL (function-python, function-kcl) handle complex data transformation and conditional resource creation. If you would reach for a for loop or a locals block in Terraform, reach for a programmatic function here.\nYAML + CEL (function-kro) takes a different approach: define resources in YAML, wire them with CEL expressions, and let the function resolve dependencies automatically. No templates, no code files. CEL strikes a balance between YAML simplicity and the expressiveness of a full language.\nUtilities (function-auto-ready, function-extra-resources, function-environment-configs, function-cidr) handle single jobs well: checking readiness, fetching cluster-scoped resources, injecting environment overrides, computing subnet ranges.\nCustom functions through the Go SDK. If none of the published options fit, write your own. Package it as an OCI image and install it with kubectl apply, same as any other function.\nThe pipeline architecture makes this powerful. Because each step passes its output to the next, you can mix functions: render with Go templating in step one, inject environment configs in step two, then check readiness with function-auto-ready in step three. Each function does one job, and the pipeline composes them.\nWhat happens when my API changes?\nThe XRD defines a contract. Contracts change. You will add a field, rename a property, or realize that spec.engine should be an enum. Crossplane supports multiple API versions in a single XRD, so you can make breaking changes without disrupting existing resources.\nSchema evolution rules. Additive changes (new optional fields, new enum values) are safe within a version. Removals, renames, type changes, and new required fields need a new version.\nAdding a version # Extend the versions array with the new schema, keeping the old version served:\nversions: - name: v1beta1 served: true referenceable: true schema: openAPIV3Schema: type: object properties: spec: type: object required: [region, bucketName] properties: region: type: string bucketName: type: string notificationEvents: type: array items: type: string - name: v1alpha1 served: true referenceable: false schema: # unchanged from before This adds v1beta1 with an optional notificationEvents field. v1alpha1 stays served but is no longer the storage version. Crossplane converts all stored resources to v1beta1 internally.3\nComposition revisions # Every Composition change creates a CompositionRevision, an immutable snapshot with an incrementing revision number. By default, XRs auto-update to the latest revision. For canary rollouts or rollback, set compositionUpdatePolicy: Manual to pin an XR to a specific revision. When you are ready to roll forward, update the compositionRevisionRef name. No redeploy needed.7\nDeprecation flow # Once all consumers have migrated, set served: false on the old version. Clients must update their apiVersion. The deprecationWarning appears in API responses for anyone still using the old version. Once no resources reference it, remove it from the versions array entirely.\nCrossplane\u0026rsquo;s conversion webhook handles internal storage between versions automatically. Resources written as v1alpha1 are stored as v1beta1 and served back in whichever version the client requests. Field renames between versions are lossy: if you renamed bucketName to name, clients reading the old schema would not find their field. Plan schema changes carefully.\nDesign principles I learned the hard way # When I designed my first XRD, I made every field required. I regretted it. A few things I wish I had known:\nRequired fields sparingly. Once a field is required in a published version, you cannot remove it. Prefer optional with a sensible default. region should default to your primary region, not require it from every team.\nPrefer enums over booleans. spec.engine: \u0026quot;postgresql\u0026quot; | \u0026quot;mysql\u0026quot; lets you add values later without a new API version. spec.enableBackups: true locks you in. Booleans are fine for debug toggles. Avoid them for anything that might grow.\nPrefer arrays over scalars. If spec.widget: \u0026quot;foo\u0026quot; might become spec.widgets: [\u0026quot;foo\u0026quot;, \u0026quot;bar\u0026quot;], start with the array. A single-element array is fine. A breaking version bump is expensive.\nLeave room for variants. If you might support MySQL and PostgreSQL under one XR, put engine-specific fields in nested objects toggled by a top-level spec.engine enum. A flat schema forces you to add top-level fields per engine, which gets messy fast.\nTL;DR # Crossplane v2 composition is not a template engine. It is an API framework:\nCompositeResourceDefinition (XRD) defines the schema of your custom API. It registers as a first-class Kubernetes resource. Composition uses a function pipeline to decide what managed resources to create when someone uses that API. Function ecosystem covers six categories: declarative YAML, Go templates, Python, KCL, YAML+CEL, and custom functions through the SDK. All plug into the same pipeline. Key gotchas: the trim-collapse bug (drop {{- on its own line), nil observed.resources on first reconcile (use dig), and required setResourceNameAnnotation. Test locally with crossplane render --crossplane-version=v2.3.1 before committing. Design for evolution: required fields sparingly, enums over booleans, arrays over scalars. API versioning and composition revisions let you make breaking changes without disrupting consumers. Next Up # The first post showed you why Crossplane is more than Terraform in Kubernetes. This post showed you how composition turns managed resources into custom APIs. The third post in this series takes the next step: Backstage as the developer portal, Crossplane as the provisioning engine, and ArgoCD as the GitOps delivery controller. Three tools, one platform.\nHave a good day!\nWeb - Crossplane documentation - \u0026ldquo;Composition\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Get Started With Composition\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Composite Resource Definitions\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Compositions\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - function-go-templating GitHub repository\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Functions\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Composition Revisions\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"12 July 2026","externalUrl":null,"permalink":"/posts/crossplane-composition/","section":"Posts","summary":"","title":"Crossplane Composition Is Not Templating","type":"posts"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/devops/","section":"Tags","summary":"","title":"DevOps","type":"tags"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/","section":"ditwrd","summary":"","title":"ditwrd","type":"page"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/iac/","section":"Tags","summary":"","title":"IaC","type":"tags"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/kubernetes/","section":"Tags","summary":"","title":"Kubernetes","type":"tags"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/learn/","section":"Tags","summary":"","title":"Learn","type":"tags"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/platform-engineering/","section":"Tags","summary":"","title":"Platform Engineering","type":"tags"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/reinventing-the-wheel/","section":"Tags","summary":"","title":"Reinventing the Wheel","type":"tags"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"},{"content":"","date":"12 July 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"You look at Crossplane and think: this is just Terraform in Kubernetes. YAML instead of HCL, kubectl apply instead of terraform apply. Same declarative config, same cloud resources at the end. I thought the same thing when I first saw it.\nThen I created an S3 bucket through Crossplane and deleted it from the AWS console just to see what would happen. Crossplane recreated it within seconds.\nThat moment changed how I think about infrastructure. The two tools share the same input format, but they operate in completely different categories. One is a compiler. The other is a runtime.\nWhat\u0026rsquo;s actually wrong with the run-and-forget model?\nTerraform does exactly what it was designed to do. It has an enormous provider ecosystem, mature state management, and it\u0026rsquo;s battle-tested at scale. The issue isn\u0026rsquo;t Terraform, it\u0026rsquo;s what happens after the process exits. Resources get modified out of band (tags via the console, policy changes, accidental deletions), and that drift accumulates silently until your next plan.1\nCrossplane takes the opposite approach. Built on the Kubernetes reconciliation loop, it constantly watches resources and corrects drift back to your desired state.1\nI came to Crossplane from Terragrunt. Hundreds of resources across environments, modular HCL, remote state locking. It worked for years. The question that cracked it open wasn\u0026rsquo;t whether Terraform can provision resources. It can. The question was what happens between runs.\nWhat does \u0026ldquo;constantly watches resources\u0026rdquo; actually mean?\nIt means you move from an execution model to a control plane model. A control plane is software that manages other software: you declare desired state, it makes it happen, and it keeps it that way. AWS runs on this same pattern. When you call the EC2 API to create an instance, AWS\u0026rsquo;s control plane provisions the VM and reports back. If the instance terminates, the control plane detects the drift and acts.\nControl planes are a core cloud native pattern.2 Crossplane generalizes it. You build a control plane for anything: cloud resources, SaaS APIs, DNS records, even other Kubernetes clusters.\nThe shift I had to internalize: IaC tools are compilers. They translate config into API calls, then stop. Crossplane is a runtime. It runs continuously, watching and correcting. Same declarative input, completely different behavior class.\nSo Crossplane runs inside Kubernetes?\nExactly. Crossplane is a set of controllers that extend Kubernetes with new custom resources. The neat part: you don\u0026rsquo;t write Go controller code. Crossplane provides the controllers; you describe what you want with YAML (or Python, or Go functions) and it handles the rest.1\nHere\u0026rsquo;s what that looks like architecturally:\nflowchart TB subgraph K8S[\"Kubernetes Cluster\"] KAPI[\"Kubernetes API Server\"] --\u003e XCORE[\"Crossplane Core\"] XCORE --\u003e PROV[\"Provider Controller(e.g. AWS)\"] end USER[\"kubectl apply\"] --\u003e KAPI PROV --\u003e CLOUD[\"Cloud Provider API\"] CLOUD --\u003e RES[\"S3 Bucket\"] For me, this clicked when I provisioned an S3 bucket I\u0026rsquo;d managed through Terragrunt for years. Same resource. But then I deleted it from the AWS console just to test, and Crossplane recreated it within seconds. That was the moment I stopped thinking about provisioning and started thinking about governing.\nHow do I create an S3 bucket with Crossplane?\nYou install a Crossplane provider, a package that adds support for a set of managed resources. The AWS S3 provider installs support for all AWS S3 resources, including Bucket.3\nHere\u0026rsquo;s what a managed resource looks like:\napiVersion: s3.aws.m.upbound.io/v1beta1 kind: Bucket metadata: namespace: default generateName: crossplane-bucket- spec: forProvider: region: us-east-2 Create it with kubectl apply. The provider controller talks to the AWS API, provisions the bucket, and reports the status through the resource\u0026rsquo;s status field. When READY and SYNCED both show True, your bucket is live. Delete the custom resource and the bucket goes with it.3\nNote: Before using providers, you need credentials. Crossplane providers authenticate using Kubernetes Secrets, IRSA (for EKS), or Workload Identity (for GKE). The setup varies per provider, but the pattern is consistent: create a Secret, reference it in a ProviderConfig.[^3] That sounds like Terraform with a kubectl frontend. What\u0026rsquo;s actually different?\nIt\u0026rsquo;s the question I started with, and it deserves a real answer.\nBoth tools are declarative. The difference isn\u0026rsquo;t the syntax, it\u0026rsquo;s the execution model:\nTerraform runs as a CLI or CI pipeline. It resolves config, plans changes, and applies them. Until the next run, it doesn\u0026rsquo;t know if a resource drifted. Crossplane runs as a control plane. It continuously watches every managed resource. Delete a bucket through the console and Crossplane recreates it on the next reconciliation loop. Drift a tag and it\u0026rsquo;s corrected. This makes Crossplane GitOps-native by design. You commit desired state to Git, ArgoCD or Flux syncs it to the cluster, and Crossplane reconciles from there. No CI pipeline required for provisioning changes. No credential management across multiple runners. No state file to store or lock.4\nflowchart LR subgraph T[\"Terraform / Terragrunt\"] TF_PLAN[\"terraform plan \u0026 apply\"] --\u003e TF_RES[\"Resourcescreated\"] TF_RES -.-\u003e|\"drift detectednext run\"| TF_PLAN end subgraph C[\"Crossplane\"] CP_APPLY[\"kubectl apply\"] --\u003e CP_WATCH[\"Providerreconciles\"] CP_WATCH -.-\u003e|\"corrects driftcontinuously\"| CP_WATCH end The distribution model is another difference worth understanding. With Terraform, you publish modules to a registry and pin versions through Git tags. Teams that forget to bump the ref stay on old versions. Crossplane takes a different approach: XRDs and Compositions are regular Kubernetes resources, installed through the same GitOps pipeline as your applications. The API server IS your registry. Run kubectl api-resources and XRDs list alongside built-in types like Deployments. Access control is RBAC. No separate registry credentials to manage.4\nWhat if my team doesn\u0026rsquo;t want to write S3 bucket manifests? They want a higher-level \u0026ldquo;Storage\u0026rdquo; API.\nThis is where Crossplane\u0026rsquo;s composition engine comes in. Not in the hype sense. It literally changes what kind of tool you\u0026rsquo;re operating. You stop provisioning resources and start defining APIs that provision resources.\nComposition lets you build custom APIs, Composite Resources (XR), backed by a pipeline of Composition Functions. You define the schema with a CompositeResourceDefinition (XRD) and wire up the logic in YAML, Python, KCL, or Go.5\nThis was the second click for me. Managed resources replace Terraform providers. Composition replaces Terraform modules. But composition does something modules can\u0026rsquo;t: your custom resource becomes a first-class API endpoint. A Database XR shows up in kubectl api-resources alongside Deployment and Service. The platform team defines the schema. App teams just create instances.\nHere\u0026rsquo;s an App that provisions a Deployment, a Service, and an RDS instance from three lines of YAML:\napiVersion: example.crossplane.io/v1 kind: App metadata: name: my-app spec: image: nginx Behind the scenes, Crossplane renders the full stack, all without writing a Kubernetes controller.5\nflowchart TB subgraph A[\"App team creates\"] APP[\"kind: Appimage: nginx\"] end subgraph P[\"Platform team owns\"] C[\"Composition functionpipeline\"] end subgraph R[\"Provider provisions\"] S3[\"S3 Bucket\"] RDS[\"RDS Instance\"] end APP --\u003e C C --\u003e S3 C --\u003e RDS This separation defines platform engineering with Crossplane:6\nPlatform team owns the XRDs, Compositions, and provider configurations. They define what \u0026ldquo;a database\u0026rdquo; means for the organization. Application team creates custom resources without caring whether the underlying resource is RDS, Cloud SQL, or on-prem PostgreSQL. Beyond managed resources and composition, is there more?\nCrossplane v2 introduced a third component: Operations. These are function pipelines that run to completion. Certificate monitoring, rolling upgrades, scheduled maintenance. Three flavors: Operation (run once), CronOperation (scheduled), and WatchOperation (triggered by resource changes).7\nThere\u0026rsquo;s also the Package Manager, which lets you install providers, functions, and entire control plane configurations across multiple clusters or regions.8\nTL;DR # Crossplane is a control plane framework built on Kubernetes that lets you:\nProvision cloud resources (S3 buckets, RDS instances, VPCs) using kubectl, with continuous drift correction instead of run-and-forget. Build custom APIs using Composition: define your own XRDs and wire them to managed resources using Python, YAML, KCL, or Go functions. Run operational tasks (certificate monitoring, upgrades) through function pipelines. Package and distribute control plane configurations across multiple environments. The key difference from traditional IaC tools: Crossplane never stops reconciling. Your desired state is the source of truth, and the control plane corrects any drift automatically.\nNext Up # This was the conceptual overview. The next post explores Crossplane v2 composition: building custom APIs with XRDs, function pipelines, and the patterns that make platform engineering work.\nThe post after that closes the series with the full platform picture: Backstage as the developer portal, Crossplane as the provisioning engine, and ArgoCD as the GitOps delivery controller.\nHave a good day!\nWeb - Crossplane documentation - \u0026ldquo;What\u0026rsquo;s Crossplane?\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Get Started\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Get Started With Managed Resources\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Configuring Crossplane with Argo CD\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Get Started With Composition\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Composition\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Get Started With Operations\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Crossplane documentation - \u0026ldquo;Packages\u0026rdquo;\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"11 July 2026","externalUrl":null,"permalink":"/posts/crossplane/","section":"Posts","summary":"","title":"Crossplane Is Not Terraform in Kubernetes","type":"posts"},{"content":"","date":"19 August 2024","externalUrl":null,"permalink":"/tags/machine-learning/","section":"Tags","summary":"","title":"Machine Learning","type":"tags"},{"content":"","date":"19 August 2024","externalUrl":null,"permalink":"/tags/python/","section":"Tags","summary":"","title":"Python","type":"tags"},{"content":" Read this article at ditwrd.dev/en/posts/dvc-workshop Prerequisites # Tools # Laptop/PC Internet Step by step # Github account (Register here) Fork the repository for the demo here Login to gitpod here Create a new workspace Choose the forked repository Ensure you are using in-browser vscode We\u0026rsquo;ll get back to it later\nMachine Learning Crash Course # If you want to know more about Python, you can check my old \u0026ldquo;Introduction to Python\u0026rdquo; article (free and publicly accessible) that I haven\u0026rsquo;t import to this website, but for now it can be accessed here What is machine learning? # An algorithm that iteratively learn by identifying patterns in data and making sense of it over time using the magic of math 🔢✨\nHow can machine \u0026ldquo;learn\u0026rdquo;? # I think we need to start with\nHow can Human learn?\n🧠🧠🧠\nWe have a brain that contains billions of neurons\nSo, do machines have brain?\nYes\nKinda\nNot really\nIt is inspired by how human brain works\nHow do we build this machine \u0026ldquo;brain\u0026rdquo;?\nTo build a ML model, we need to know what it is made up of. It is made up of Neuron, a simple place holder for a number.\nWhen we have bunch of neuron that \u0026ldquo;talk\u0026rdquo; to each other, they create a \u0026ldquo;Neural Network\u0026rdquo;\nA neural network consist of layers that simply called:\nInput Hidden Output Where the number of layers and neuron per layer can be tuned to fit a problem. The increase of layer count create a \u0026ldquo;deeper\u0026rdquo; model, this is where the term \u0026ldquo;Deep Learning\u0026rdquo; came from\nHow Neural Network actually learn?\nUsing clever math, it all started in 1943 on the Bulletin of Mathematical Biophysics\u0026rsquo;s journal with the article titled \u0026ldquo;A Logical Calculus of Ideas Immanent in Nervous Activity\u0026rdquo; by W. Mcculloch and W. Pitts.\nPerceptron introduced a concept called \u0026ldquo;Forward Propagation\u0026rdquo;, a mathematical model based on neuron as shown below\n$$ \\hat{y} = f(\\sum_{n=1}^{N}w_n.x_n + b) $$where:\n\\(\\hat{y}\\) = output \\(f\\) = non linear function \\(w\\) = weight \\(x\\) = input \\(b\\) = bias (or sometime \\(w_0\\)) The \u0026ldquo;learning\u0026rdquo; part of the model back then was done either manually or using a simple weight update like the function below\n$$ w_{n+1} = w_n + \\alpha . (y-\\hat{y})x $$where:\n\\(y\\) = desired output \\(\\hat{y}\\) = model output \\(\\alpha\\) = learning rate \\(w\\) = weight \\(x\\) = input \\(b\\) = bias (or sometime \\(w_0\\)) This approach of updating the weight base on the error or in this case the difference between the model and desired output help the model learn\nResearcher took a long time going down the rabbit hole and finally on 1970 (S. Linnainmaa on his Master Thesis about reverse mode of automatic differentiation) the modern method of \u0026ldquo;learning\u0026rdquo; was introduce where it help scale the \u0026ldquo;learning\u0026rdquo; portion\nI\u0026rsquo;ll save you from reading too much math, because it\u0026rsquo;s a lot of math to dumb down, I recommend these playlist for you to get both the math and intuition behind it\nPlaylist Reference 3blue1brown Andrew Ng - DeepLearningAI For now, I\u0026rsquo;ll just give you the intuition behind the magic of the method called \u0026ldquo;Backpropagation\u0026rdquo;\nIn essence, it\u0026rsquo;s calculating the gradient between loss (or error) and weight (and bias). The gradient is then used to do \u0026ldquo;gradient descent\u0026rdquo; where we update the weight and bias based on the gradient. From the image above you can see that the gradient help guide the parameters to the lowest loss. Yes, this is an oversimplification, because the search of the lowest loss doesn\u0026rsquo;t exactly happened in 2 dimension as we seen in the image. The number of dimension the model search is proportional to its parameter, so if you have a million parameter, the model search space is a million dimension in size\nMachine Learning Lifecycle # This is going to be a short one, this is a breakdown of a simple machine learning model lifecycle. In production environment for a larger company the technique used for each of these steps would differ and some steps can even be broken down into more steps, but at its core the step and goal remain the same.\nData Fetching # There are multiple way to get data, we can do our own data collection/scraping or maybe use secondary data from data repository such as HuggingFace or Kaggle\nData Analysis # It\u0026rsquo;s going to be helpful if we do data analysis to at least know the characteristic of our data. This will help us prepare for the next step\nData Preparation # Once we know the characteristic of our data, we can try to use technique to sanitize our data (Undersampling, Augmentation, Noise Reduction, etc) ready for model training. The data can be split into 3, training, testing and validation. Usually the definition of testing and validation data can be mixed up so don\u0026rsquo;t get confused by it. Training data is the data used by the model to learn. Testing data is the data that is used during training for us to know whether the model can generalize its knowledge on data they never see, this is where we tune our model. Validation data is the last data that the model never see and we never use to tune our model, this data serve for the final model to really test its capability.\nModel Training # This is the core of a machine learning lifecycle, 99% of your time will be spent on this, this is where you\u0026rsquo;ll fine tune the model, changing parameter and even going back a few steps to ensure you are using the best version of your data. The training process will use the training and testing part of your data\nModel Validation # This is where the model is tested with validation data\nDeployment # If everything is good, we can deploy the machine learning model to server our purpose. For research purpose, we can create a GUI or CLI that can help us use the model.\nHands On: ML Project with DVC # In this hands on project, we are going to use the famous MNIST dataset, a dataset that consists of hand written digit and its label. This is a cool first project because in the end of the project we can see the model we built actually learn to label data\nInitial Setup # Dependency Manager - uv # We start by installing uv, a pretty new python dependency manager based in Rust that is definitely blazingly fast.\nDependency manager is a tool that we can use to manage our dependencies or to put it simply, manage our libraries on our project. There are a lot of dependency manager in the market right now, and Python have their own built in dependency manager called pip but we are using uv as it is way faster.\nTo install we can run this command in our gitpod terminal\ncurl -LsSf https://astral.sh/uv/install.sh | sh Continued with initializing uv in our project\nuv init We then can start installing our libraries for our project, in this case it\u0026rsquo;ll be dvc, dvc-gdrive, tensorflow, numpy and ipykernel\nuv add dvc dvc-gdrive dvclive \\ ruamel.yaml python-box \\ tensorflow numpy \\ ipykernel DVC - ML Pipeline Tools # DVC (Data Version Control) is a tool to manage the version of our data. Not only data management, it can be use to create a machine learning pipeline that interface with multiple python scripts and data where it\u0026rsquo;ll automatically run specific part of the pipeline based on the current changes in our code, minimizing error such as forgetting to run specific script\nTo use dvc in our project we must first initialize it\nuv run dvc init We can then setup dvc to use a Google Drive folder to save our data. Create a google drive folder and save the UNIQUE_ID written in the folder URL\nuv run dvc remote add dvc-demo gdrive://UNIQUE_ID There is currently an issue with dvc access being blocked when using personal gmail account to connect\nThe issue arise just two weeks before the writing of this post\nTrack the issue here here Due to the issue above, the next best thing to do is to use a service account, to do that you need to have a GCP (Google Cloud Provider) account with a project, then you can create a Google service account through this guide here and once done, create a service account key using this guide here\nWe can then save the json to accountService.json in our project and adding accountService.json inside .gitignore to prevent it being commited to the repository. Ensure that you\u0026rsquo;ve shared the Google Drive folder with the service account email as an Editor\nOnce finished, we setup dvc to use the service account and let it know where the json file is\nuv run dvc remote modify dvc-demo gdrive_use_service_account true uv run dvc remote modify dvc-demo gdrive_service_account_json_file_path accountService.json uv run dvc remote default dvc-demo Data Fetching # We are going to use the MNIST dataset that we can easily download from Google. But first, we need to create the folder where we save our data. Create a folder called data and create 3 subfolder inside it named raw, interim and processed with an additional .gitkeep inside each subfolder. Raw is where we placed our raw data, interim is where we create a cleaned version of our data and proccessed is where the data that\u0026rsquo;ll be used for experiment will be.\nGo to data/raw and download the data manually first from an external source\ncd data/raw wget https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz mnist.npz will be right inside our data/raw folder. Now we can use dvc to save the data to our Google Drive folder.\nuv run dvc add mnist.npz uv run dvc commit uv run dvc push It\u0026rsquo;ll create a mnist.npz.dvc file that is used by dvc to track our data. This file is the only file that is pushed to Github, so there is no need to push a lot of data to Github and it\u0026rsquo;ll definitely be a hassle to manage when we have multiple GB of file to work with.\nData Exploration # This exploration will be simple just to keep this hands on short. To do this we can use Jupyter notebook to play around with our data. For now the goal is to simply check how to open mnist.npz.\nCreate a notebooks folder and create 1 Data Exploration.ipynb inside. Open the file and we can start coding\nimport numpy as np path = \u0026#34;data/raw/mnist.npz\u0026#34; with np.load(path) as f: x_train, y_train = f[\u0026#34;x_train\u0026#34;], f[\u0026#34;y_train\u0026#34;] x_test, y_test = f[\u0026#34;x_test\u0026#34;], f[\u0026#34;y_test\u0026#34;] print(f\u0026#34;{x_train.shape=}\u0026#34;) print(f\u0026#34;{x_test.shape=}\u0026#34;) print(f\u0026#34;{y_train.shape=}\u0026#34;) print(f\u0026#34;{y_test.shape=}\u0026#34;) print(f\u0026#34;{x_train[0]}\u0026#34;) This code is going to load the file using np.load and from there we can check the amount of data + the data given.\nData Preparation # From our previous step we can see the amount of data and we can see the data used. Now, we are going to create a data pipeline that\u0026rsquo;ll normalize the data value range from 0-255 to 0-1 and saved it into a new file in processed. To keep this hands on short we\u0026rsquo;ll not use interim data, but if your data is scattered around, this\u0026rsquo;ll definitely help.\nNow, we\u0026rsquo;ll create the script that\u0026rsquo;ll do the things mentioned above, but first, we need to restructured our folder. uv init has created src/dvc_demo with an __init__.py inside of it. We need to first extract dvc_demo outside of src, contiuned to deleting the src folder. This is done to reduce folder depth that may cause headache in the future once the code grows bigger\nLet\u0026rsquo;s start our script at dvc_demo/preprocessing/normalize.py. The code for the script is as shown below\nimport numpy as np from box import ConfigBox from ruamel.yaml import YAML INPUT_PATH = \u0026#34;data/raw/mnist.npz\u0026#34; OUTPUT_PATH = \u0026#34;data/processed/mnist_clean.npz\u0026#34; yaml = YAML(typ=\u0026#34;safe\u0026#34;) def open_mnist(): with np.load(INPUT_PATH) as f: x_train, y_train = f[\u0026#34;x_train\u0026#34;], f[\u0026#34;y_train\u0026#34;] x_test, y_test = f[\u0026#34;x_test\u0026#34;], f[\u0026#34;y_test\u0026#34;] return x_train,y_train, x_test, y_test def normalize(arr,num): return arr/num def save_data(data_dict): np.savez_compressed(OUTPUT_PATH,**data_dict) if __name__ == \u0026#34;__main__\u0026#34;: params = ConfigBox(yaml.load(open(\u0026#34;params.yaml\u0026#34;, encoding=\u0026#34;utf-8\u0026#34;))) x_train,y_train, x_test, y_test = open_mnist() x_train_norm = normalize(x_train, params.preprocess.normalize.num) x_test_norm = normalize(x_test, params.preprocess.normalize.num) data_dict = { \u0026#34;x_train\u0026#34;: x_train_norm, \u0026#34;y_train\u0026#34;: y_train, \u0026#34;x_test\u0026#34;: x_test_norm, \u0026#34;y_test\u0026#34;: y_test, } save_data(data_dict) and an additional params.yaml\npreprocess: normalize: num: 255 This script has the raw data as the input and the clean data is output to data/processed/mnist_clean.npz. To run the script, go back to the root folder and use uv run\ncd ../.. uv run dvc_demo/preprocessing/normalize.py This will generate the normalized mnist data\nThis approach is okay, but running this everytime is not great, thats why we use dvc. Now we are going to create a stage, or in some sense a start of a pipeline where we tell dvc what every input and output dependency. To do this for the script above we can do this\nuv run dvc stage add \\ -n preprocess-normalize \\ -p preprocess \\ -d dvc_demo/preprocessing/normalize.py \\ -d data/raw/mnist.npz \\ -o data/processed/mnist_clean.npz \\ uv run dvc_demo/preprocessing/normalize.py The -n flag correspond to the stage name, -p is the parameter that we can use for experimenting later, -d is the dependency for the process this can be any file and scripts needed for the process to work and -o is the output of the process.\nNow you can run these command to see the connection between process and also run all of the stage\nuv run dvc dag uv run dvc repro Model Training # For the last two step of an ML lifecycle, we are going to go with the same approach as before:\nCreating scripts in Notebook to experiment Putting it into a script Creating stage with dependencies and parameters This is the code\nfrom dvclive import Live from dvclive.keras import DVCLiveCallback import tensorflow as tf from ruamel.yaml import YAML from box import ConfigBox import numpy as np yaml = YAML(typ=\u0026#34;safe\u0026#34;) INPUT_PATH = \u0026#34;data/processed/mnist_clean.npz\u0026#34; def train(): params = ConfigBox(yaml.load(open(\u0026#34;params.yaml\u0026#34;, encoding=\u0026#34;utf-8\u0026#34;))) print(params) # Open data with np.load(INPUT_PATH) as f: x_train, y_train = f[\u0026#34;x_train\u0026#34;], f[\u0026#34;y_train\u0026#34;] x_test, y_test = f[\u0026#34;x_test\u0026#34;], f[\u0026#34;y_test\u0026#34;] layer = [tf.keras.layers.Flatten(input_shape=(28, 28))] for node_num in params.train.node: layer.append(tf.keras.layers.Dense(node_num, activation=\u0026#34;relu\u0026#34;)) layer.append(tf.keras.layers.Dropout(0.2)) layer.append(tf.keras.layers.Dense(10)) model = tf.keras.models.Sequential(layer) loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True) model.compile(optimizer=\u0026#34;adam\u0026#34;, loss=loss_fn, metrics=[\u0026#34;accuracy\u0026#34;]) with Live(\u0026#34;results/train\u0026#34;) as live: model.fit( x_train, y_train, validation_data=(x_test, y_test), callbacks=[DVCLiveCallback(live=live)], epochs=params.train.epochs, ) model.save(\u0026#34;models/model.keras\u0026#34;) live.log_artifact(\u0026#34;models/model.keras\u0026#34;, type=\u0026#34;model\u0026#34;) if __name__ == \u0026#34;__main__\u0026#34;: train() To ensure that dvc have data for each training loop, there is an additional Live from dvclive that help track the training process\nTo ensure everything works, we need to add this to params.yaml\ntrain: epochs: 5 node: - 32 - 8 Now we can add a stage for training like this\nuv run dvc stage add \\ -n train \\ -d data/processed/mnist_clean.npz \\ -d dvc_demo/model/train.py \\ -o models/model.keras \\ uv run dvc_demo/model/train.py Model Evaluation # The model evaluation for this Hands On is a bit redundant because we are just doing the exact validation done by the model during training, this is added for the sake of completeness and can be a boilerplate for numerous amount of evaluation that you can do\nfrom dvclive import Live from dvclive.keras import DVCLiveCallback import tensorflow as tf from ruamel.yaml import YAML from box import ConfigBox import numpy as np yaml = YAML(typ=\u0026#34;safe\u0026#34;) INPUT_PATH = \u0026#34;data/processed/mnist_clean.npz\u0026#34; INPUT_MODEL = \u0026#34;models/model.keras\u0026#34; def eval(): params = ConfigBox(yaml.load(open(\u0026#34;params.yaml\u0026#34;, encoding=\u0026#34;utf-8\u0026#34;))) print(params) # Open data with np.load(INPUT_PATH) as f: x_test, y_test = f[\u0026#34;x_test\u0026#34;], f[\u0026#34;y_test\u0026#34;] model = tf.keras.models.load_model(INPUT_MODEL) with Live(\u0026#34;results/evaluate\u0026#34;) as live: evaluation = model.evaluate( x_test, y_test, ) print(evaluation) live.summary[\u0026#34;evaluation\u0026#34;] = evaluation if __name__ == \u0026#34;__main__\u0026#34;: eval() uv run dvc stage add \\ -n evaluate \\ -d data/processed/mnist_clean.npz \\ -d dvc_demo/model/evaluate.py \\ -d models/model.keras \\ uv run dvc_demo/model/evaluate.py Experimentation # This is where things become fun. We now have created a trackable and idempotent ML pipeline where we can start our tuning.\nThis is the main command that\u0026rsquo;ll you use for tuning\nuv run dvc exp run \\ -n experiment-1 \\ -S \u0026#34;train.node=[128,64,32]\u0026#34; \\ -S \u0026#34;train.epochs=10\u0026#34; -n correspond to the name of the experiment (if we don\u0026rsquo;t use this the experiment name is autogenerated) and -S correspond to the params that we are changing\nUsing vscode we can install an extension extension to help visualize everything, this\u0026rsquo;ll help us see the experiment and also plot everything, but we can also run uv run dvc exp show where we can see everything via the terminal\nNow, for every change you do, either from your program, additional stage or trying to find a better param, your ML pipeline has been settled and you can focus on the things that matter\nFor more in-depth documentation you can check out DVC own website\nWait, what about deployment?\nDeployment is a vast topic, especially in Machine Learning, both simple and super complex, either to scale for millions of user or for research purposes, for now this is left as an exercise for the reader\nThank you! # Thank you for reading this article, the topic that has been taught here just scratch the surface of machine learning with a touch of imperfection and oversimplification here and there, I hope this\u0026rsquo;ll bring you closer to your goal, whatever it is\nFor any inquiry feel free to contact me via email hi@ditwrd.dev or through my Linkedin DMs\nHave a good day!\n","date":"19 August 2024","externalUrl":null,"permalink":"/posts/dvc-workshop/","section":"Posts","summary":"","title":"Simplifying Machine Learning Lifecycle with DVC","type":"posts"},{"content":"","date":"19 August 2024","externalUrl":null,"permalink":"/tags/talk/","section":"Tags","summary":"","title":"Talk","type":"tags"},{"content":"","date":"19 August 2024","externalUrl":null,"permalink":"/tags/workshop/","section":"Tags","summary":"","title":"Workshop","type":"tags"},{"content":"","date":"28 June 2024","externalUrl":null,"permalink":"/tags/etcd/","section":"Tags","summary":"","title":"Etcd","type":"tags"},{"content":"","date":"28 June 2024","externalUrl":null,"permalink":"/series/raft/","section":"Series","summary":"","title":"Raft","type":"series"},{"content":"","date":"28 June 2024","externalUrl":null,"permalink":"/tags/raft/","section":"Tags","summary":"","title":"Raft","type":"tags"},{"content":" Leader Election (And a bit of refreshing) # Base on all the question from the previous post, it seems that Raft single handedly powered a majority of modern cloud technology since its invention.\nSo, lets learn about dig deep down\nQuestion Chain # Who made it?\nRaft algorithm was introduced by Diego Ongaro and John Oustershout around 2013 with an extended paper published in 2014[^6] (that I\u0026rsquo;ll be using as the source of truth for my code later). The focus on Raft is to create an \u0026ldquo;understadable\u0026rdquo; consensus algorithm, pushing others to improve on its building blocks.\nI've embed the whole PDF paper here if you want to see, this chapter is basically may attempt to sums everything up Previous Next \u0026nbsp; \u0026nbsp; / [pdf] View the PDF file here. What is Consensus?\nTo put it simply, consensus is an agreement surrounding a state between fault tolerant distributed systems. Reaching an agreement meant that the decision on a state value is final, creating a cluster of server with a replicated state. To reach that goal these server need a protocol to do so.[^7]\nHow Raft works?\nRaft work with a single leader approach and decompose the consensus problem to multiple subproblem. Each server talk to each other via two Remote Procedure Call (RPC), AppendEntries and RequestVote, we\u0026rsquo;ll talk about it later on.[^6]\nLeader? What\u0026rsquo;s that?\nLeader is one of the three state of a server in Raft. A Leader is responsible of handling these things:\nClient requests (If a client connect to a server with a state other than a Leader they will redirect it to the Leader) Log replication The other two state are Candidate and Follower. The name is self explanatory, a Follower follow what Leader ask it to do, while Candidate ask for vote to be elected to a new Leader. [^6]\nWhat about the multiple subproblem?\nThe subproblem to create a working consensus is listed below:\nLeader election Log replication Safety Cluster membership changes Log compaction How do Raft deal with Leader election?\nWe need to start with what trigger the election. The election is triggered when a Follower failed to receive a \u0026ldquo;heartbeat\u0026rdquo; (AppendEntries Remote Procedure Call (RPC) with no new log data) from the current elected Leader within a period of time called \u0026ldquo;election timeout\u0026rdquo; that is choosen randomly from a fixed interval.\nOnce the timeout has passed, a Follower would change it\u0026rsquo;s state to Candidate and increase it\u0026rsquo;s term count by 1. A term signify an arbitrary division of time in Raft, just like what you see in real world Democracy leader election.\nThe new Candidate then ask for vote to other server by sending a RequestVote RPC and vote for itself. Other server can only vote for 1 server and vote on first-come-first-served basis. The election can end in three way:\nThe Candidate win It receives a majority of vote (Not detailed in Paper, I assume it as (N/2) + 1 where N is the amount of servers) It become Leader and send heartbeat to other server The Candidate lost It receive an AppendEntries RPC from another server claiming to be leader with a term as big as itself (if not it\u0026rsquo;ll reject the RPC and continue to be a Candidate) The Candidate neither win or lose There has been multiple Candidate that split vote to themselves causing no majority wins by the Candidate The Candidate will timeout and rerun the election with an increased term To prevent another tie in the next term, the election timeout is re-randomized To sum it up, it\u0026rsquo;s a bunch of simple \u0026ldquo;if-else\u0026rdquo; logic that if followed correctly would create an simple yet complex behaviour in the system.\nWith the Leader elected, can it start to do it\u0026rsquo;s job?\nYes, in this case we can go to the next subproblem Log Replication, we\u0026rsquo;ll discuss it in the next post.\nTL;DR # Raft Leader Election Overview:\nCreated By: Diego Ongaro and John Ousterhout (2013-2014) to develop an understandable consensus algorithm.1 Consensus: Agreement on a state among distributed systems, requiring a protocol.2 How Raft Works: Utilizes a single leader to handle client requests and log replication, with servers operating as Leaders, Followers, or Candidates.[^6] Leader Election Process: Triggered when a Follower misses a heartbeat from the Leader. The Follower becomes a Candidate, requests votes, and if it secures a majority, it becomes the Leader. If the Candidate fails to win, it either receives a vote from another Leader or faces a tie, prompting a new election with re-randomized timeouts. Outcome: The elected Leader then manages client requests and log replication. Next Up: Details on Log Replication in the following post.\nPaper - In Search of an Understandable Consensus Algorithm (Extended Version) by Diego Ongaro and John Oustershout\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nWeb - Raft official website\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"28 June 2024","externalUrl":null,"permalink":"/posts/raft-leader-election/","section":"Posts","summary":"","title":"Raft Consensus Algorithm: Leader Election","type":"posts"},{"content":" Overview # Why? # This is my first technical post, and when this article is created, I\u0026rsquo;m 2 years in my journey working in cloud. Not too long, yet not too short. I felt that there is still a skill gap or skill issue for me to get to the next level. Hence I\u0026rsquo;m starting Reinventing The Wheel, a journey of me chain-questioning things surrounding cloud and other technology, I hope that I\u0026rsquo;ll learn a thing or two along the way.\nQuestion Chain # How do Kubernetes ensure cluster state is persistent?\nKubernetes use etcd cluster to store all cluster data. 1\nHow can an etcd cluster work together to ensure all data is persisted?\netcd use a consensus algorithm called Raft 2\nWhy Raft? Is there any other consensus algorithm?\n\u0026ldquo;There is only one other alternative3\u0026rdquo;, Paxos4, but Raft is way simpler than its counterpart and has been used in production for multiple cloud technology such as etcd, Kubernetes, Docker Swarm, Cloud Foundry Diego, CockroachDB, TiDB, Project Calico, Flannel, Hyperledger and more.5\nI don\u0026rsquo;t want to read the whole series, any TL;DR?\nTL;DR # It\u0026rsquo;s been 10 years since Kubernetes launch back in 2014 where the trends of \u0026ldquo;scalable\u0026rdquo; and \u0026ldquo;distributed\u0026rdquo; system is still ongoing to this day. This boom doesn\u0026rsquo;t happen without any reason, it started on a paper released by Diego Ongaro and John Ousterout titled \u0026ldquo;In Search of an Understandable Consensus Algorithm\u0026rdquo; where they challenge themselves to find an easier version of the Paxos algorithm, the go-to distributed system algorithm back then. They invented, Raft, the algorithm that power etcd, the backbone of Kubernetes, and many more to come.\nRaft is designed for understandability in mind and consist of three parts. First is leader election, where the cluster select one of the servers as a leader, second, log replication, where the leader receive command from clients, appends them to its log and replicate its logs to other servers, and last is, safety, where only a server with an up-to-date log can become leader\nLeader election start with a timeout in a servers, it\u0026rsquo;ll start the election by turning into a \u0026ldquo;Candidate\u0026rdquo; and ask for vote to other servers. Once it get the majority of votes, it\u0026rsquo;ll act as a leader. The election timeout is always refresh everytime a new Leader communicate to a server, preventing any election to happened. In a split vote situation, it\u0026rsquo;ll just wait out for another server to timeout and start a new round of election.\nOnce a leader is elected, it\u0026rsquo;ll start the log replication part of Raft. There are 2 states of log, appended and committed, where an appended log wait for itself to be shared to the majority of server, and once shared, will be committed, in this case the log is applied to the state machine on each of the servers. These states are use to ensure the consistencies of log can be maintain between outage. The safety of the log replication is also considered during leader election, where candidate with lower committed log index won\u0026rsquo;t get any vote.\nSince Raft is implemented everywhere, notable Golang-based library would be etcd/raft and hashicorp/raft that to this day still being maintained by the community. By the end of this series, my goal would be to create my own Raft implementation in Golang\nWeb - Kubernetes documentation\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nGithub - etcd\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nYoutube - Ben Johnson\u0026rsquo;s \u0026ldquo;Raft - The Understandable Distributed Protocol\u0026rdquo; talk on Strange Loop Conference\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nPaper - Paxos by L.Lamport\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nGithub - Raft implementation used by etcd\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","date":"27 June 2024","externalUrl":null,"permalink":"/posts/raft/","section":"Posts","summary":"","title":"Raft Consensus Algorithm: Overview","type":"posts"},{"content":"","date":"13 June 2024","externalUrl":null,"permalink":"/tags/about/","section":"Tags","summary":"","title":"About","type":"tags"},{"content":" I\u0026rsquo;m Aditya Wardianto or just Adit, a so called YAML Engineer based in Indonesia focusing on cloud and other bleeding edge technology\nResume # Previous Next \u0026nbsp; \u0026nbsp; / [pdf] View the PDF file here. ","date":"13 June 2024","externalUrl":null,"permalink":"/about/","section":"ditwrd","summary":"","title":"About Me","type":"page"},{"content":"","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":"","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"}]