Guide for Emerald teams

Introduction

The Emerald OpenShift cluster is the third iteration of our OpenShift Platform as a Service (PaaS), following the Silver and Gold/GoldDR environments. It’s the most secure of the 3 environments and is considered Zero Trust by default.

Zero Trust on Emerald is a framework that’s implemented via VMWare NSX to secure pods at the infrastructure level. The same technology is also used to secure Virtual Machines (VMs) that are hosted specifically on a VMWare Elastic Sky X integrated (ESXi) cluster also managed by NSX.

Zero Trust means that all valid traffic to/from workloads inside the Software-Defined Network (SDN) network space controlled by NSX must be defined with explicit firewall rules. The technology involved for how these firewall rules are defined will vary depending on the technology involved.

1. Firewall rules are defined on the OpenShift Emerald cluster via the creation of NetworkPolicy objects which will explicitly define all allowed egress and ingress traffic inside each user namespace. If the traffic desired isn’t defined in the NetworkPolicy objects, then the traffic involved will not flow
2. VMWare NSX controls firewall definitions using Distributed FireWall (DFW) rules. By using the NSX Manager, individuals can add firewall rules involving VMs hosted in the NSX SDN. Integration exists between Emerald and NSX so that all NetworkPolicy resources are translated into NSX DFW rules automatically. Due to the nature of Zero Trust, NSX DFW rules also need to be created for regular VM workloads to completely define what traffic is allowed in and out of the VMs. Suitable access is granted to users for the creation of those rules
3. For servers (both physical and VM) that do not reside on the NSX-managed network, traditional firewalls and their corresponding firewall request documents are still a reality to contend with. For containers on the Emerald cluster, we are able to isolate remote firewall access down to the individual namespace level, thus allowing a regular server to be solely reached from pods inside a single Emerald namespace without worrying about any other Emerald namespace also having access. Each Emerald namespace receives for itself its own dedicated CIDR /26 range of IP addresses, and firewall requests created in this fashion would reference that class C range of IP addressees since individual pod IP addresses will change over time, but always be an IP from the range allotted inside its namespace

There are top-level DFW rules in NSX that are called Guard Rail rules. These rules cannot be changed by owners of VMs or containers as they are meant to enforce specific behavior. While allowing/denying access to shared services (VMs and pods inside NSX still need to be able to send emails via apps.smtp.gov.bc.ca for example), these Guard Rail rules also define a concept known as “Security Classification” as a value involving specific tiers (Low, Medium and High) as well as dictate how traffic between workloads of different (or same) Classifications can or cannot interact.

---

Security classifications

For a detailed breakdown of how the different Security Classifications are defined with regards to workloads making use of private data, please go to the official project document specifying more details here: SDN Security Classification Model. Ministry architects must work with their Ministry’s security and privacy teams to choose the appropriate security classification for their workloads based on their organization’s risk.

---

Notable difference between Emerald and Silver/Gold OpenShift clusters

For individuals who have been using existing OpenShift clusters which do not reside in the SDN space, some differences exist that may affect how individuals use and interact with the services offered by Emerald.

API and Web Console URLs are no longer public

Unlike the Silver/Gold/GoldDR clusters where the API and Web console URLs use public IP addresses (meaning accessible from the Internet), these URLs in Emerald are given private IP addresses which are not accessible on the Internet, but are instead accessible within the Data Centres where the other OpenShift clusters reside, as well as Zone C network space, which allows B.C. Government employees to access these URLs either from their office network or via an approved VPN connection.

OpenShift routes no longer use Public IP addresses

By default, any route created in a user’s namespace with no additional annotation will default to a Low Security Classification which only gives a private IP address instead of a public IP address. With specific annotations to the route, the classification can be changed to a Medium or High Security Classification (also a private IP address). If there’s a specific need to grant a route a public IP address, that can be done as well, or the route can be proxied either by a third party server, or a reverse web proxy service as offered by the OCIO in their catalogue. Technical details on how this is done will be discussed later.

Pod IP addresses now route internally within the data centres

For all OpenShift clusters regardless of how they are implemented, pods are assigned an internal IP address. Unlike the regular OpenShift clusters, the Emerald cluster allows routing of these internal pod IP addresses within the data centre, meaning that attempts to access a remote server in the data centre that is protected by regular firewall technologies can allow access from Emerald by providing a network range that is unique on a per-namespace basis, thus allowing remote access to legacy DB servers from one or more specific namespaces on Emerald without allowing any other namespace this access.

NetworkPolicy resources need complete Ingress and Egress definitions

Regular OpenShift clusters normally only enforce the creation of accurate Ingress definitions. For Emerald, accurate Ingress and Egress definitions need to be defined in order for any network interaction involving containers in Emerald and other workloads to work, since Zero Trust is enforced, thus all valid traffic patterns specific to the application need to be created via NetworkPolicy resources.

Direct web access from Emerald to the Internet no longer allowed

Due to the security classification of the Emerald nodes themselves (High), they’re no longer allowed direct access to make HTTP and HTTPS connections to the Internet. For allowing core functionality to persist, the EMERALD cluster is configured for making use of a Forward Web Proxy configuration which has been configured with all of the known URLs required for the basic functionality of the Emerald cluster (ability to reach Red Hat URLs for OpenShift upgrades, reach GitHub and reach common URLs for repositories for common software like python/pip). This means that if a user finds that their operations in Emerald requires the addition of a URL that is not yet in the configuration of the current Forward Web Proxy, a request will need to be made.

Ask on Rocket Chat, #emerald-how-to channel, specifying the URL (wildcards accepted as well as specific URLs) and detailing why it’s needed.

Software managed by operators may or may not work on Emerald as is

Some software as managed by Operators (OpenShift Integration components like Kafka and Zookeeper) may work on OpenShift clusters like Silver without extra work, but not for the Emerald cluster. There needs to be a means of adding additional NetworkPolicy definitions to satisfy the Zero Trust aspect of the Emerald cluster, and also add the required pod annotations to satisfy the requirement that all workloads have an assigned DataClass value (more technical details about this later).

One possible mechanic that will allow such additions for Emerald would be the Custom Resource Definition (CRD) that is responsible for managing the deployment of software on Emerald. For Kafka, that would be the kafka CR. Documentation for all of the features available will vary, so the best method of exploring what can be done with a given CRD would be to use oc explain to be informed what the CRD can do and if that feature allows management of pod and route security classification levels and full NetworkPolicy definitions. It may be possible for the NetworkPolicy definitions to be created manually outside of the control of the operator, but one will want to confirm this first by creating a test deployment of the software and working through the issues and roadblocks experienced while adding new data to the Custom Resource (CR) to over-come connectivity issues experienced in the Emerald cluster. The following is an example of a place in the Kafka CRD where annotation for a pod security classification label needs to be applied.

$ oc explain kafka.spec.entityOperator.template.pod.metadata.labels
KIND:     Kafka
VERSION:  kafka.strimzi.io/v1beta2

DESCRIPTION:
     Labels added to the resource template. Can be applied to different
     resources such as `StatefulSets`, `Deployments`, `Pods`, and `Services`.

In the end, it is best to fully evaluate software deployed by an Operator before committing to its use in a production environment on the Emerald cluster, since there is no universal requirement or standard that an Operator will be 100% compatible with the Zero Trust environment implemented by VMWare NSX. If not sure if an operator has already been evaluated by someone else, ask first on Rocket Chat either in the #nsx-early-adopters channel, or the #devops-howto channel, making sure for the latter channel to clarify it is for the Emerald cluster since otherwise others may offer solutions that work on Silver or Gold, but not on Emerald.

OpenShift versions will likely differ between Silver/Gold/GoldDR and Emerald

With the vendor support matrix for VMWare NSX against OpenShift as well as needing to account for upgrade release dependencies for NSX before considering OpenShift upgrades, the versions of OpenShift on Emerald will be behind Silver.

All Emerald pods need to have a valid Security Classification

Unlike the Silver and Gold OpenShift clusters which do not implement such a concept, for the Emerald cluster, a Security Classification needs to be assigned to all pods. This is accomplished by applying a label called DataClass with a value of Low, Medium, or High. Some types of pods do not allow the organic application of this label, such as the pods created when starting an image build. Automation exists to address this by automatically ensuring such build pods are annotated with a label of DataClass=Low.

Any pods that are started up without a valid DataClass label will not be able to connect to other pods, even pods residing inside the same namespace inside the Emerald cluster.

---

How security classifications work in Emerald

The Security Classifications defined above need to be defined for 2 specific resources in OpenShift. Each pod needs to have a specific label called DataClass assigned to it and all network traffic going to and from these pods will be evaluated with the setting for each pods DataClass label accordingly.

For incoming (ingress) traffic that’s not coming directly to the pod internal IP addresses, Kubernetes routes will also need annotation so that a VIP (virtual IP) is assigned to the route DNS entry of appropriate Security Classification. This annotation also controls if a VIP is allocated that’ll allow the route to be reachable outside of the B.C. Government data centres and on the Internet. At present all routes created without specific annotation are created with a Security Classification of Low.

Emerald technical how-to’s

While not exhaustive, this section provides technical examples of various activities that may be needed for a developer to fully make use of the Emerald OpenShift cluster. If your technical question isn’t listed here, we recommend posting the question on B.C. Government Stack Overflow where the “sdn” tag already exists for questions specific to Emerald.

How to find the internal CIDR /26 assigned to pods for my namespace

Since internal Pod IP addresses are routable within the B.C. Government main 2 data centres, it’s imperative to know how to find the specific /26 CIDR assignment for any given namespace, since such information is then used when opening up access to non-SDN workloads using traditional firewall rules. We cannot offer any finer granularity for this type of access when involving legacy firewalls in the data centres. Finer granularity requires all involved workloads reside inside the SDN network space as managed by NSX.

Using the oc command, the following one-liner will display the assigned CIDR, just replace <namespace> with your actual namespace in question. The login account usingoc` needs to be a member of that namespace/project though.

oc get namespace <namespace> -o jsonpath='{.metadata.annotations.ncp\/subnet-0}{"\n"}'
10.90.68.0/26 (response given)

If only the web console is available to you instead of oc, then use the following URL template to see the full YAML for your namespace, and from there, find the suitable section containing this information. Same as before, replace with the namespace name.

https://console.apps.emerald.devops.gov.bc.ca/k8s/cluster/projects/<namespace>/yaml

metadata:
  annotations:
...
    ncp/subnet-0: 10.90.68.0/26
...

How to set security classifications inside pods

For pods, a Security Classification is assigned using a specific label that is named “DataClass”. This label can have values corresponding to the 3 presently defined values for a Security Classification: “Low”, “Medium” and “High”. Here is an example snippet for a pod with a Security Classification set for “Medium”.

apiVersion: v1
kind: Pod
metadata:
...
  labels:
    DataClass: Medium
...

If the DataClass label is missing, or present but doesn’t have a recognized value assigned to it, then expect network traffic to and from that pod to not work as expected until this is addressed.

Dealing with pod deployments

For pods such as deployment pods which are responsible for creating other pods in turn, the addition of the DataClass takes place not inside the DeploymentConfig metadata section, but instead the section for the pod template.

apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
...
spec:
...
  template:
    metadata:
...
      labels:
        DataClass: Low
...

Dealing with pod builds

Running pod builds is a special case on Emerald, since Unlike DeploymentConfigs and other resources which create pods, resources involving pod builds have no place to inject a valid DataClass settings. To resolve this, we make use of a technology called Kyverno which has the ability to modify API requests before the OpenShift cluster fulfills the body of the request. Thus in Emerald when a build pod is started up, Kyverno injects a DataClass=Low and an additional label of build=true which can be referenced in a NetworkPolicy to ensure the build pods can access specific other resources/pods as needed.

Dealing with custom resources managed by operators

Another special case to consider is when pods are managed by an operator on Emerald that controls how pods are deployed via a Custom Resource (CR). The example used here is the “Kafka” Custom Resource as handled by the AMQ Streams operator. This Kafka CR is a more complex Custom Resource than most, the following snippet shows where one can add the necessary “DataClass” labels to ensure all pods generated due to this CR have the correct DataClass settings.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
...
spec:
  entityOperator:
    template:
      pod:
        metadata:
          labels:
            DataClass: Medium
...
  kafka:
...
    template:
      pod:
        metadata:
          labels:
            DataClass: Medium
  zookeeper:
...
    template:
      pod:
        metadata:
          labels:
            DataClass: Medium

If not sure where one injects this data into a different Custom Resource, look online for vendor documentation detailing the custom resource specification, or explore the specification (known as the Custom Resource Definition or CRD) using the “oc explain” command. The following shows how we know the CRD supports pod labels for the entityOperator section of the CRD:

oc explain kafka.spec.entityOperator.template.pod.metadata.labels
KIND:     Kafka
VERSION:  kafka.strimzi.io/v1beta2

DESCRIPTION:
     Labels added to the resource template. Can be applied to different
     resources such as `StatefulSets`, `Deployments`, `Pods`, and `Services`.

Do recognize that supporting the ability for a Custom Resource to inject the required “DataClass” labels isn’t a certainty in the CRD specification. When looking at a Custom Resource not yet tested by someone else in the Emerald cluster, first evaluate to see if the CRD specification supports the addition of label metadata before consider the use of this Custom Resource and operator for supporting your desired application build in Emerald.

How to allow a public VIP to access a Low DataClass pod

Based on the SDN Security Classification Model, there would normally be no network traffic allowed from the internet to a Low DataClass pod. This can be changed by applying an additional label to the Low DataClass pod called “Internet-Ingress=ALLOW”. Applying this will allow the Public VIPs to connect to Low DataClass pods, but in exchange, the Low DataClass pod can no longer connect to a High DataClass pod.

How to assign security classifications to routes

A Security Classification is also required for routes in the Emerald cluster. Unlike pods, there’s a default Security Classification assigned to all routes and that’s a Security Classification of “Low”. To assign a different data classification, you need to annotate your Route using a different syntax than what’s used for Pods. For routes, you use “aviinfrasetting.ako.vmware.com/name” and assign it a value from one of the following: “dataclass-medium”, “dataclass-high” or “dataclass-public”.

For the route Security Classification, all settings result in a private IP address that cannot be accessed from the Internet. If internet access is required, then add the annotation with a value of “dataclass-public”.

The default “apps.emerald.devops.gov.bc.ca” subdomain cannot be resolved by DNS from the public internet, so public routes will need to use a vanity domain and dedicated TLS certificates to be accessible from the internet.

The following snippet of YAML is for a Route modified so that the assigned VIP will be for a Security Classification of Medium.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  annotations:
...
    aviinfrasetting.ako.vmware.com/name: dataclass-medium

How to use a vanity domain in a route

The AVI load balancer needs to know about the Second Level Domain name of all routes it manages. If you have a new vanity domain you’ll need to contact the Ops team to have it added to the list. Currently in the list is *.bc.ca since most domains will be sub domains of that.

Due to the ability to have different Security Classifications for Route VIPs and AVI sharing the load across several “service engines”, this means there’s no single VIP assigned and known in advance. You need to create a DNS entry to point to your vanity routes. You can either view the Route YAML in the Web Console, or use the following “oc” command to obtain the assigned IP address from the .status.ingress.conditions.message field. Replace them with the actual values for your namespace and route name.

$ oc -n <namespace> get route <routename> -o jsonpath='{.status.ingress[].conditions[].message}{"\n"}'
10.99.10.8

This assigned IP address will persist for the lifespan of the named route. If that route is deleted and created again, it’s quite possible the assigned IP address may be now different than what was present before. Keep this in mind when automating your application’s deployment where accurate DNS records are required for the routes. Go to NNR and create the needed A records for your vanity domain.

How to expose non-HTTP access to a service

On the Silver and Gold OpenShift clusters, this is managed in a more complicated way using the F5 Load Balancer and integration between OpenShift and the F5.

On Emerald, this additional setup is not necessary, but is now done natively with a change of type for the Service resource involved with your workload. Normally this would be set to “ClusterIP”, changing it to “LoadBalancer” will expose the service onto an IP address which can be used to reach the service on its defined non-HTTP port(s). The following .spec snippet will do this for a service involving postgresql.

spec:
  type: LoadBalancer
  ports:
  - name: postgresql
    port: 5432
    protocol: TCP
    targetPort: 5432
  selector:
    name: postgresql

Like routes, the assigned IP address defaults to a low data class, and to change this, you annotate the Service in the same fashion as you would annotate a Route.

To find out the assigned IP, the following one-liner will suffice. Replace it with the appropriate values for your situation.

oc -n <namespace> get service <servicename> -o jsonpath='{.status.loadBalancer.ingress[].ip}{"\n"}'
10.99.10.74

How to connect from my pods to the outside Internet via HTTP or HTTPS

Unlike regular OpenShift clusters like Silver and Gold, the Emerald OpenShift cluster doesn’t have the ability to make direct connections with Internet-hosted (outside of the B.C. Government data centres) sites via HTTP or HTTPS. To fetch content like this, pods will need to make use of an existing Forward Web Proxy configuration. In most cases, pods will inherit the required proxy settings without further intervention since the EMERALD cluster has a cluster-wide proxy configuration.

Here are the details for cases where a pod might need to make use of this Forward Web Proxy, but do so in a fashion where it’s not automatically inheriting the settings from the OpenShift cluster itself.

Proxy URL: http://swpxkam.gov.bc.ca:8080
Proxy credentials: (not necessary, the specific proxy configuration is specific to the source IP addresses including the EMERALD cluster)
Proxy IP addresses: 142.34.229.249, 142.34.94.249

There are 2 IP addresses for the web proxy since the proxy URL is managed by an F5 Global Server Load Balancer (GSLB) configuration and if the native proxy service in Kamloops (where Emerald also resides) is not functioning, then the URL will point to the Calgary service instead. These IP addresses are required in order to create a suitable NetworkPolicy to allow the pod to reach the proxy servers on both IP addresses in case the web proxy fails over. If using the example below, make sure to adjust the values of name, namespace and value for matchLabels as appropriate.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-django-to-web-proxy
  namespace: test-namespace
spec:
  egress:
  - to:
    - ipBlock:
        cidr: 142.34.229.249/32
    - ipBlock:
        cidr: 142.34.94.249/32
    ports:
    - port: 8080
      protocol: TCP
  podSelector:
    matchLabels:
      name: django-psql-persistent
  policyTypes:
  - Egress

If the pod doesn’t natively pick up the proxy configuration but do understand and accept environment variables, then try setting the following and see if that makes the pod work as expected.

HTTP_PROXY=http://swpxkam.gov.bc.ca:8080
HTTPS_PROXY=http://swpxkam.gov.bc.ca:8080
NO_PROXY=.cluster.local,.svc,10.91.0.0/16,172.30.0.0/16,127.0.0.1,localhost,.gov.bc.ca

For situations where the pod is attempting to fetch content that could otherwise be hosted in Artifactory, it may be better to check or ask if the content you need is already in Artifactory, or if not, can be made to be hosted there, thus eliminating a need to find out and track required URLs for the Forward Web Proxy.

At present the URL allow list has been set for allowing URLs involved with maintaining the Emerald cluster along with a few other sites. There’s a process by which requested URLs can be added to the configuration for the Forward Web Proxy, but will require a communication to us (Platform Operations) in the Rocket Chat #devops-operations channel, along the lines of:

Please add, if not already present, the URL <insert URL here, just need the Fully Qualified Domain Name> to the Forward Web Proxy configuration to allow access from the Emerald cluster.

Once the request is received and confirmed to be a new URL not yet configured, Platform Operations will put in the paperwork to have the F5 Proxy services re-configured, which will take a bit of time (up to a week depending on how fast all parties are with the changes). Keep this delay in mind when planning new software deployments.

How to use Vault in Emerald

Vault Server lives in Gold/GoldDR OpenShift clusters, which by default is not accessible for High DataClass pods in Emerald. We have setup a reverse Proxy in Emerald that handles the network traffic outside of the cluster. Application pods will still need the same Vault Injector annotations as in other clusters, the only difference is the additional Egress netpol that allows your app to request for the secret from the Proxy at port 2015. Here are detailed setup your workload in Emerald to use encrypted secrets from Vault:

1. Refer to the Vault technical document for basic configuration
2. Add the Egress NetworkPolicy to enable your pods to talk to the proxy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: init-egress-app-to-vault-injector
spec:
  podSelector:
    matchLabels:
      app: my-app-label
  egress:
  - ports:
    - port: 2015
      protocol: TCP
    to:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: vault-agent-proxy
      namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: openshift-bcgov-vault
  policyTypes:
  - Egress

How to connect from High DataClass pods in Emerald to shared services like SSO

At present, this isn’t allowed under the current implementation of how the SDN network currently works along with the existing Guard Rail rules.

From the perspective of the SDN as controlled by NSX, anything hosted on either the Silver or Gold/GoldDR OpenShift clusters is classified as Low, by virtue of the IP addresses involved with network traffic exiting the involved clusters being classed as Low overall.

For now, if you want to connect from a Medium or Low pod to SSO, use a NetworkPolicy like this:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-from-myapp-to-sso-gold
  namespace: license-dev
spec:
  podSelector:
    matchLabels:
      app: my-app-label
  egress:
    - ports:
        - protocol: TCP
          port: 443
      to:
        - ipBlock:
            cidr: 142.34.229.4/32 # Gold cluster *.apps IP
        - ipBlock:
            cidr: 142.34.64.4/32 # Gold DR cluster *.apps IP in case SSO fails over
  policyTypes:
    - Egress

How to connect to a route from a pod

Routes are fronted by a load balancer outside the SDN. They can’t be addressed by label in a NetworkPolicy. Instead, you need to create an Egress policy to allow the pod to connect to the IP of the Route. Remember also that the load balancer is shared, so the policy will allow access to routes on several other tenants namespaces. Whenever possible, you should use a Service name instead to manage traffic between pods in the cluster.

To get the IP assigned to your Route:

$ oc -n <namespace> get route <routename> -o jsonpath='{.status.ingress[].conditions[].message}{"\n"}'
10.99.10.8

Then create a NetworkPolicy to allow your pod to egress to that IP:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-from-myapp-to-route
  namespace: license-dev
spec:
  podSelector:
    matchLabels:
      app: my-app-label
  egress:
    - ports:
        - protocol: TCP
          port: 443
      to:
        - ipBlock:
            cidr: 10.99.10.8/32
  policyTypes:
    - Egress

How to use Tekton webhooks to trigger CI/CD from GitHub

Since the API of the cluster isn’t available on the public internet, GitHub Actions are unable to trigger deployments on the cluster or create objects. An alternative you can use is OpenShift Pipelines aka Tekton. Review the documents on triggering a pipeline run for how to set up a Trigger, EventListener and Route. When creating the Route however, you’ll need to make use of a vanity domain name that you control. The default *..apps.emerald.devops.gov.bc.ca subdomain cannot be resolved by DNS from the public internet however. Change the route to use a subdomain of your production vanity domain and add a TLS certificate and key to secure the Route. Then you can get the IP assigned to the Route and create the DNS A record in NNR or other DNS provider to allow GitHub to look up webhook URL.

Forward proxy allow list URLs

This is the allow list URLs currently in the forward proxy:

*.apps.emerald.devops.gov.bc.ca
*.apps.klab2.devops.gov.bc.ca
api.loginproxy.gov.bc.ca
ws1.postescanada-canadapost.ca
*.akamaitechnologies.com
*.s3.amazonaws.com
*.s3-us-west-2.amazonaws.com
*.us-east-1.amazonaws.com
galaxy.ansible.com
registry.developers.crunchydata.com
registry-auth.developers.crunchydata.com
production.cloudflare.docker.com
hub.docker.com
github.com
*.githubusercontent.com
*.google.com
storage.googleapis.com
api.openshift.com
*.api.openshift.com
mirror.openshift.com
*.mirror.openshift.com
*.p1.openshiftapps.com
*.access.redhat.com
cdn.redhat.com
cloud.redhat.com
*.connect.redhat.com
console.redhat.com
maven.repository.redhat.com
subscription.rhn.redhat.com
subscription.rhsm.redhat.com
sso.redhat.com
*.service-now.com
bcgov.sharepoint.com
download.sysdig.com
app.sysdigcloud.com
collector-alt.sysdigcloud.com
projects.registry.vmware.com
*.pkg.dev
*. ghcr.io
registry.devfile.io
docker.io
*.docker.io
gcr.io
*.gcr.io
ghcr.io
*.icr.io
releases-docker.jfrog.io
*.k8s.io
registry.k8s.io
quay.io
*.quay.io
registry.redhat.io
*.akamaiedge.net
*.fastly.net
*.alpinelinux.org
repo.maven.apache.org
*.debian.org
repository.jboss.org
repo1.maven.org
*.nuget.org
api.nuget.org
pypi.org
pypi.python.org
files.pythonhosted.org
developers.redhat.com
app.powerbi.com
*.statcan.gc.ca
analysis.windows.net
login.microsoftonline.com
api.powerbi.com
graph.microsoft.com
developer.microsoft.com
bcgov.webhook.office.com
bccourtsca-my.sharepoint.com
*.ca-central-1.amazonaws.com

apiVersion: ako.vmware.com/v1alpha1
kind: HostRule
metadata:
  name: testing-waf-in-emerald
  namespace: license-test
spec:
  virtualhost:
    fqdn: my-vanity-domain.com
    fqdnType: Exact
    wafPolicy: My-Django-Test-Policy

$ oc -n steve-test get hostrule testing-waf-in-klab2 -o jsonpath='{.status.status}{"\n"}'
Accepted
$ oc -n steve-test get hostrule testing-waf-in-klab2
NAME                   HOST                                             STATUS     AGE
testing-waf-in-klab2   waftest-steve-test.apps.klab2.devops.gov.bc.ca   Accepted   21h

$ oc -n steve-test delete hostrule testing-waf-in-klab2
hostrule.ako.vmware.com "testing-waf-in-klab2" deleted