Upon trying to create a NAT-Routed network within vCD, you can be faced with a

“Error creating Edge VM”

If you look within the debug logs, the excerpt of the stack-trace will look something like this:

com.vmware.ssdc.util.LMException: Error creating Shield network appliance.
… blah blah
at java.lang.Thread.run(Thread.java:619)
Caused by: com.vmware.ssdc.util.LMException: Error creating Edge VM.
... 17 more
Caused by: java.lang.NullPointerException
… yadda yadda


Click image to enlarge

The cause is a misconfigured ESX Host that is sitting in the same resource pool as other Hosts connected to the dVS against which vCD is trying to create the network group.

Solution being to either remove the host, or alternatively verify that all hosts are connected EQUALLY to the dVS. This is where Host Profiles come in very handy.

I apologise for the lack of a vCD screenshot, I fixed the problem before realising that it was one; at least there will always be logs. Once you do see this again, feel free to send in a screenie.

Related Local Files

logs/vcloud-container-debug.log

In order to demonstrate to some customers, partners and colleagues that haven’t had a chance to enjoy vCloud Director yet, I’ve recently prepared some videos which demonstrate how you setup a network within an Organization on top of vCD.

They’re designed to be completely self-sufficient, so the ‘intro’ is re-used.

Direct Network

This is by far the most … straight forward way of providing your VM’s with an outside connection.

vCloud Director Networking : Part 1 : Direct External

Routed Network

The elegance of vShield Edge allows you to configure DHCP, Firewall, and NAT IP Mapping through an easy to click-through UI. Be it for your Organization or vApp. Here I’ll show you how to setup a NAT-Routed vShield backed internal network.

vCloud Director Networking : Part 2 : NAT-Routed

Custom’ish vApp Network

There are environments that have already invested a lot in customizing their own DHCP server rules, Dynamic DNS registrations, reverse lookups, firewalls… the list goes on. In essence, here I can show you how you can re-use your current soft-router, and have it used within the vCloud realm.

vCloud Director Networking : Part 3 : Custom Routing Appliance in a vApp

Attempted to bring up vCD and saw the following in vcloud-container-debug.log

| ERROR | Start Level Event Dispatcher | StartupUtils | Error starting application: Could not bind network port: 80 on host address: 172.16.225.131 |
java.net.BindException: Cannot assign requested address
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:365)
…

One of these days I’ll allocate some static IP’s to my whole Fusion-bound vLab. Until then – I will put up with DHCP assigned IP’s.

To fix, first stop vCD [ vmware-vcd stop ], the modify cloud-director/etc/global.properties to reflect the IP changes for:

vcloud.cell.ip.primary
consoleproxy.host.https
vcloud.cell.ips

or anywhere you have the old address[es]. Some love from sed will do the trick here.

Now back to it …

A colleague recently stumbled upon this beautiful error when trying to import an OVA to an ESX and ESXi host:

Unsupported element 'VirtualSystemCollection'

The cause is that the packaged OVA is actually a vApp extract from vCenter, and standalone hosts (not under management of a vCenter) are not able to accomodate vApp.

From VMware

vApps: Ensuring seamless application movement and choice between clouds

It seems a single host isn’t “cloud”-enough.

An OVA that is extracted from vCenter which contains only a single VM is extremely close in its structure to the OVF apart from a tuple called … *drum roll* … <VirtualSystem ovf:id="cake"> in the single, and <VirtualSystemCollection ovf:id="cake"> in one that encompasses multiple machines. So a hosts ability is limited to not being able to parse this collection of machines and discriminate between their properties, that’s why you need to spend $ on vCenter!

–

This is all great, but my colleague is now stuck half-way around the world with a set of machines that need to be deployed on a single host, and all of them are saved within this OVA.

An OVA is just a wrapper for an OVF (think VMX) and some VMDK’s {anyone VM/OVconfused yet?}.


$ tar tf cake.ova
cake.ovf
cake-disk1.vmdk
cake-disk2.vmdk
cake-disk3.vmdk
cake-disk4.vmdk
cake-disk5.vmdk
cake.mf


We can see there’s a collection of VMDK’s, the manifest which inside looks like this:


SHA1(cake.ovf)= 03AD765EC45B90E13BC22D0115088F08021F2AE5
SHA1(cake-disk1.vmdk)= 92BB519FD1926D4F3C170C727C037E2D5D79775B
...


… and more importantly the OVF, which describes the OVA motherload.

My thoughts were at first to just create a VMX for each of the VMDKs and then I can go to bed. Unfortunately one of the machines has two disks, so the first issue is with figuring out how the OVF nests the rasd – Resource Allocation Setting Data (as per the DMTF OVF Specification V1.1).

Thanks to Mr. Pekka for writing xmlindent or you could just use the online version

I was quickly able to see to which VM disks 3+4 belonged to, and create the appropriate VMX skeleton. Now if only there was a parser…

ESX doesn’t have NAT inbuilt, so here’s how to configure it with the help of a VMware appliance called pfSense (an Open Source [free!] firewall/router).

There are three components in this setup:

1. Host
2. router/pfSense
3. NAT Client

Host

A great read for beginners and those refreshing is the VMware Virtual Networking Concepts whitepaper

Now lets create a network that our NAT’ed VMs will be using.

When prompted under Connection Type, select a ‘VM Network’, as this is for the typical traffic within the Virtual Machine (not IO or management of your machines).

Lets create a vSwitch that doesn’t connect to anything, a dud, a blankie. This will be our NAT’ed environment. It’s quite important that you DON’T connect a network card to this vSwitch to prevent any inadvertent DHCP leakage. Make sure you have nothing selected.

Give it a name to differentiate.

Once you’re done, click finish and you will have something two network available to your VMs:

• VM Network
• NAT Network

Time to setup pfSense. Once you’ve downloaded and extracted it. You have the option of either copying it directly to your datastore and then adding directly to inventory, or importing via the Standalone Converter. I find the latter is always faster.

router/pfSense

Incase you’re converting pfSense first (like I did whilst re-doing it for this post), I recommend you disable the network interfaces until you’ve finished setting up the host networks. We’ll enable these in a later step.

Disabled interfaces

Once the conversion is complete, time to configure our virtual router. pfSense is provided with two NICs out of the box. One for the WAN interface (which is your internal LAN), and one for ‘its’ LAN – the one on which it will be servicing DHCP requests.

Mark down the last 4 digits of the MAC address, these will help to validate the following step.

Configuring pfSense NICs

Start the pfSense VM. You will be guided through the mapping of the interfaces, and just to make sure – check to see the MAC addresses matching to the VM Network (in my case 67:3c) and NAT Network (67:46).

Upon following the wizard, and if you’ve followed everything accordingly (or rather I documented the steps properly) you will be shown the interfaces within pfSense, their mapping (WAN vs LAN) and IP addresses.

NAT Client[s]

You are now ready to assign clients on this host to the NAT Network and have them pick up addresses dished out by your shiny new appliance.

The whole setup takes just under 5 minutes from start to finish to complete.

When you look at the mostly fragmented cloud and ancillary services market, who do you pick?

•

Amazon, the incumbent in providing no-frills, pay as you go raw crunch-power has carved out the niche over the last few years with its EC2 offering which has been very popular with corporates dipping their toes in the cloud space. Due to the non-exhaustive list of governance, privacy, flexibility, compliance and classification – organisations are finding it difficult to dive completely in, thus the ‘private cloud’ model is cast.

In order to partake in the race to the bottom within cost of IT operations and service agility so well presented by the cloud pundits, organisations (the more intelligent ones at least) started to re-think internal infrastructure. The simple question was posed – “how do we make what we have now – better ?”. First step with any problem worth solving is to break it down into its elementary components: as such organisations by their very nature will have numerous divisions, departments, sub-organisations and overhead associated with managing not only the division of these business entities, but mainly their computational dependence – this stuff is COMPLEX!

… or is it?

With what Amazon has achieved within the IaaS and Salesforce in S/PaaS in the recent years is a testament to the fact that true multi-tenanted solutions that can scale and be at face value simple to interact-with and build-atop are possible. The only problem is that to date, no one has managed to do this effectively within the confines of an enterprise.

In a recent article Scott Drummonds talks about the necessity in transparency of costs associated with spending between various Lines of Business in the organisation. I wish to see that taken further, and not only must transparency exist in consumption, but also in design and most importantly management of the core infrastructure and the systems that uphold the business.

From Citrix, VMware, Microsoft, Oracle or RedHat – there is still not a single public offering which allows you to manage, automate, delegate and provision datacenter technologies on the fly. The best means to competitively differentiate private cloud vendors will not be done in the hypervisor, but within the middleware. You can wheel as many CloudBursts (am looking at you IBM) or Vblocks (VCE) into an the datacenter and try and call it “cloud”, but when it comes to true agility with the various datacenters something is missing.

In comes Nimbula, the love child between two former EC2 executives and a development team in South Africa. The aim of the unreleased product is to “combines the flexibility, scalability and operational efficiencies of the public cloud with the control, security and trust of today’s most advanced data centers” – sounds great! The Nimbula Director is slated for general availability in Q4 2010. What is most interesting is that one of the investors is … *drum roll please* – VMware. They haven’t even started, and already have an exit. Wow.

According to TechTarget, VMware themselves are coming out with a Service Director product, which aims to do exactly what Nimbula just announced.

Given Nimbulas pedigree they will be going with the Xen hypervisor, and building their management suite on top. Coupled with the fact that VMware is now one of the main stakeholders provides us with some assurance of compatibility and hopefully a solution that is truly hypervisor agnostic.

Let the management games begin!

• adapted from flickr users maisonbisson | dominik99 | zeno77

Exception when adding a network - click to enlarge

Adding a new port (i.e. a vMotion interface) to a vSwitch on vSphere/ESX leads to this lovely error message. If you check your vpxd.log you’ll see something the image verbalised.

[2010-06-04 15:30:52.411 03152 info 'App'] [VpxLRO] — ERROR task-54417 — host-36589 — vim.host.NetworkSystem.updateNetworkConfig: vim.fault.PlatformConfigFault:
(vim.fault.PlatformConfigFault) {
dynamicType = ,
faultCause = (vmodl.MethodFault) null,
text = “SysinfoException: Node (VSI_NODE_net_tcpip_plumb) ; Status(bad0017)= Out of resources; Message= Instance(0): Input(4) if=0 portset=VMkernel macaddr=00:50:56:76:16:67 tsomss=65535 “,
msg = “Error during the configuration of the host: SysinfoException: Node (VSI_NODE_net_tcpip_plumb) ; Status(bad0017)= Out of resources; Message= Instance(0): Input(4) if=0 portset=VMkernel macaddr=00:50:56:76:16:67 tsomss=65535 “,
}

Key here is the Out of resources; message. The reason for this is none other than the default number of ports for the vSwitch on ESX is 24, and if you have VM’s or other interfaces using up ports (such as AppSpeed probes), you will quickly run out. Switch this easily by going to Configuration -> Networking -> Properties [for the vSwitch in question] and up the value up to and including your growth requirements for the future.

NOTE: After you setup a new host, set it to more ports than you require, as you’ll need to restart the host for the ports to be provisioned; best to do this immediately after installation.

I think a better message would be to explicitly say “Out of available ports on vSwitch – [blah]“; instead of the semi-cryptic one presented.