I was off to a good start, but have not posted in a few weeks. Partly due to recovering from end of quarter craziness, (lots of OT), and preparing for 4 certification exams before Nov 15th. Most recently 70-535. I won’t go into the gritty details of preparing for the exam. There are many blog posts on the subject. Here’s my prep. A few months ago I attended two week long Azure classes. A Azure train the trainer class, and an Azure Stack installation class. Fast forward lots of OT, and a looming exam date that can’t be rescheduled. I quickly watched the PluralSight 70-534 course, and used a free Azure trial as a refresher. Going into the exam I felt Ill prepared, but passed the exam in the end. I dislike the new MS exam format. No going back after completing a section. I take tests based on first instinct. Your first choice is often the correct one, don’t overthink it. If the question requires more contemplation, mark it for review, and go back later if time permits. If you learn the service definitions, and apply the 50/50 elimination rule to answers you should be alright. On to the next exam.
EHC upgrade foundation init failure. SyntaxError: String contains control character
When upgrading DellEMC’s EHC foundation modules. The DellEMC EHC foundation module initialization workflow fails with an error: SyntaxError: String contains control character Workflow:CreatePropertyGroupAndCustomPropertiesSiteAffinity / Create Property Group (item1)#35) (Workflow:CreatePropertyGroupAndCustomPropertiesSiteAffinity / Create Property Group (item1)#35)
This failure occurs due to special characters found in property groups property values during a JSON conversion.
A few property value examples causing the failure:


To resolve:
- Backup the property groups using cloudclient, or manually steps 3-6
- Delete the property groups from the vRA UI
- Run Initialize EHC foundation workflow again
- Restore property groups.
Convert vRA property groups visibility from global to tenant.
When migrating from vRA 6.2 to 7.x property groups are copied, but are converted to global visibility. It’s great that everything copies over, but the property group conversion causes an export problem. Cloudclient should be able to read/export global property groups soon. Someone else had the same issue. In my use case I needed to temporarily export the property groups and delete them due to an error in a EHC workflow. I had to open a ticket with VMware vRA Engineering for this process. (Thanks guys)
On with the show, here’s how to convert property group visibility manually. I’ll work on automating the process when time allows.
Requires Linux or MAC machine with access to vRA appliance.
1. Obtain auth token – login as a user with permission to create/manage property groups:
– POST to https://$VA/identity/api/tokens
– curl syntax:
curl –insecure -H “Accept: application/json” -H ‘Content-Type: application/json’ –data ‘{“username”:”user@domain”,”password”:”password”,”tenant”:”tenant_name”}’ https://<vra_url>/identity/api/tokens
Example:
dev-1 $ curl –insecure -H “Accept: application/json” -H ‘Content-Type: application/json’ –data ‘{“username”:”fritz@coke.sqa-horizon.local”,”password”:”Password1!”,”tenant”:”rivet”}’ https://$VA/identity/api/tokens {“expires”:”2018-07-24T05:49:43.000Z”,”id”:”MTUzMjM4MjU4MzY2MjoxNGY4ZjcxMjZhOThhYjViNTI1YTp0ZW5hbnQ6cWV1c2VybmFtZTpmcml0ekBjb2tlLnNxYS1ob3Jpem9uLmxvY2FsZXhwaXJhdGlvbjoxNTMyNDExMzgzMDAwOjViY2JkOTBhYzI4YWRiYjMyMmY5NjUyZmY1NDIyODlhNDhjYzkxYjI0M2QzMThjN2E5MDMyY2JhMzAwZWEwM2I0MzhmZTFmNDM4ZGQ4ZjhmZTM5M2EyZTU3YjgzYTViZDAyYzdhOWJlZWQ0MmRkNWRlNjMxOGY4OTJmODY3″,”tenant”:”rivet”}
2. For convenience, assign auth token value to shell environment variable:
dev-1 $ export TOKEN=”MTUzMjM4MjU4MzY2MjoxNGY4ZjcxMjZhOThhYjViNTI1YTp0ZW5hbnQ6cWV1c2VybmFtZTpmcml0ekBjb2tlLnNxYS1ob3Jpem9uLmxvY2FsZXhwaXJhdGlvbjoxNTMyNDExMzgzMDAwOjViY2JkOTBhYzI4YWRiYjMyMmY5NjUyZmY1NDIyODlhNDhjYzkxYjI0M2QzMThjN2E5MDMyY2JhMzAwZWEwM2I0MzhmZTFmNDM4ZGQ4ZjhmZTM5M2EyZTU3YjgzYTViZDAyYzdhOWJlZWQ0MmRkNWRlNjMxOGY4OTJmODY3″
3. Retrieve JSON definition for desired global property group:
– GET from https://$VA/properties-service/api/propertygroups/<property-group-id>
– curl syntax (note Authorization header with auth token from steps 1 and 2):
curl –insecure -H “Content-Type: application/json” -H “Accept: application/json” -H “Authorization: Bearer $TOKEN” https://$VA/properties-service/api/propertygroups/<property-group-id>
Example:
dev-1 $ curl –insecure -H “Content-Type: application/json” -H “Accept: application/json” -H “Authorization: Bearer $TOKEN” https://$VA/properties-service/api/propertygroups/Test
{“id”:”Test”,”label”:”Test”,”description”:”All tenants”,”properties”:{“one”:{“facets”:{“visibility”:{“type”:”constant”,”value”:{“type”:”boolean”,”value”:false}},”encrypted”:{“type”:”constant”,”value”:{“type”:”boolean”,”value”:false}},”mandatory”:{“type”:”constant”,”value”:{“type”:”boolean”,”value”:false}},”defaultValue”:{“type”:”constant”,”value”:{“type”:”string”,”value”:”two”}}}}},”tenantId“:null,”createdDate”:”2018-07-23T20:55:48.908Z”,”lastUpdated”:”2018-07-23T20:55:48.908Z”,”version”:0}
4. Copy JSON returned in step 3 and edit: set the tenantId:
– Change: “tenantId“:null to “tenantId“:”<desired-tenant>“
Optional: change the description field as/if desired.
5. From the vRA UI, delete the property group for which you obtained the JSON definition in step 3.
Example:
Login to vRA UI and delete property group “Test”.
6. From the command line, create a new property group from the edited JSON at step 4.
– POST to https://$VA/properties-service/api/propertygroups
– curl syntax (note Authorization header with auth token from steps 1 and 2):
curl –insecure -H “Content-Type: application/json” -H “Accept: application/json” -H “Authorization: Bearer $TOKEN” –data ‘<edited-JSON-from-step-4>’ https://$VA/properties-service/api/propertygroups
Example:
dev-1 $ curl –insecure -H “Accept: application/json” -H “Content-Type: application/json” -H “Authorization: Bearer $TOKEN” –data ‘{“id”:”Test”,”label”:”Test”,“description”:”New Test”,”properties”:{“one”:{“facets”:{“visibility”:{“type”:”constant”,”value”:{“type”:”boolean”,”value”:false}},”encrypted”:{“type”:”constant”,”value”:{“type”:”boolean”,”value”:false}},”mandatory”:{“type”:”constant”,”value”:{“type”:”boolean”,”value”:false}},”defaultValue”:{“type”:”constant”,”value”:{“type”:”string”,”value”:”two”}}}}},“tenantId”:”rivet”,”createdDate”:”2018-07-23T20:55:48.908Z”,”lastUpdated”:”2018-07-23T20:55:48.908Z”,”version”:0}’ https://$VA/properties-service/api/propertygroups
7. Repeat steps 3 – 6 until all property groups have been redefined.
vRA 7.3 installation Web Server config fails
Web server config fails with Endpoint {0} registration in virtual appliance failed
Verify TLS 1.0 is enabled on the IAAS web servers by using regedit to change the following entry from 0 to 1 HKey_Local_Machine\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server\enabled
vRA remove requests pending Approval
While trying to remove a few stuck requests pending approval I found the VMware KB 2114385 is wrong. The State needs to be changed.
From https://kb.vmware.com/s/article/2114385
Correct commands below
#SSH to vRA node
#Backup postgres DB
service vcac-server stop
su -m -c “/opt/vmware/vpostgres/current/bin/pg_dumpall -c -f /PATH/vcac.sql” postgres
bzip2 -z /PATH/vcac.sql
service vcac-server start
#Connect to the database by running these commands:
/opt/vmware/vpostgres/current/bin/psql vcac postgres
\connect vcac;
\x
#Check for total count of hung requests.
select count(*) from cat_request where state = ‘PENDING_PRE_APPROVAL’;
#Run these cleanup scripts
BEGIN;
UPDATE comp_bprequest
SET status = ‘FAILED’
WHERE external_request_id::uuid IN
(SELECT id
FROM cat_request
WHERE STATE in (‘PENDING_PRE_APPROVAL’)
AND (DATE_PART(‘day’, now() – lastupdated) * 24 + DATE_PART(‘hour’, now() – lastupdated) > ‘8’));
UPDATE cat_request
SET state = ‘FAILED’
WHERE id::uuid IN
(SELECT id
FROM cat_request
WHERE STATE in (‘PENDING_PRE_APPROVAL’)
AND (DATE_PART(‘day’, now() – lastupdated) * 24 + DATE_PART(‘hour’, now() – lastupdated) > ‘8’));
UPDATE cat_resource
SET
status = ‘DELETED’
WHERE request_id in
( select id from cat_request where state in ( ‘FAILED’ ) AND type = ‘CI’ AND (DATE_PART(‘day’, now() – lastupdated) * 24 + DATE_PART(‘hour’, now() – lastupdated) > ‘8’));
#Verify that updates are successful by running Select queries:
SELECT id,
status
FROM comp_bprequest
WHERE external_request_id::uuid IN
(SELECT id
FROM cat_request
WHERE STATE in (‘PENDING_PRE_APPROVAL’)
AND (DATE_PART(‘day’, now() – lastupdated) * 24 + DATE_PART(‘hour’, now() – lastupdated) > ‘8’));
SELECT count(*)
FROM cat_request
WHERE STATE in (‘PENDING_PRE_APPROVAL’)
AND (DATE_PART(‘day’, now() – lastupdated) * 24 + DATE_PART(‘hour’, now() – lastupdated) > ‘8’);
SELECT count(*) FROM cat_resource WHERE request_id IN (SELECT id FROM cat_request WHERE STATE in (‘PENDING_PRE_APPROVAL’) AND type = ‘CI’);
#If count is 0 for all 3 of the select queries, then COMMIT; else ROLLBACK;
vRA 7.3 LDAP Authentication Fails after adding connector
Ran into a strange bug while installing vRA 7.3 in preparation for vRA 6.2 to 7.3 migration. The install went without issue, and no errors. After adding a secondary connector all LDAP authentication failed. Getting the connector added was a pain. It took forever to add, or would not add. As part of our troubleshooting we removed LDAP and started over several times. The connector add failure was caused by port 4003 not open on vRA nodes, and the hosts file was not created correctly. At the time of this writing there is not a published VMware KB, but one should be forthcoming.
vRA 7.x if unable to add secondary connector, or access denied to EHC tenant after enabling LDAP access.
Verify idP hostname set to vRA-vip FQDN, port 4003 enabled on vRA host via iptables
Port 4003 instructions:
SSh to vRA node and run iptables –L –n if port 4003 not seen for both vRA nodes contact VMware support. You’ll need to replace 3 files to enable ehcache port 40003 in vRA 7.3.1 firewall rules.
The Journey Begins
Well folks. I’ve been told I needed a blog for years. Early on I always viewed a blog as to much of a hassle due to NDA’s and what not. I finally decided to start a blog for the odd technical issues I encounter at work. Maybe my notes will save someone a few hours of searching for a work around. If you’ve wandered here and expected me to spin a yarn, you’ve come to the wrong place. Riveting reading this is not. Short and sweet is what to expect. The entries are often my notes from working through a problem. My notes may confuse folks at times, so as always any questions just ask. Maybe we’ll both learn something. Thanks for joining me!
Why riveting clouds? It’s a cross between my work and personal interests. “Today” I work as a Consultant for DELL EMC focused on Multi-Cloud. “Today” only because I’m from the legacy EMC side, and as we all know Everything Must Change. My title or division seems to change once or twice a year. What I do doesn’t. Pointed down range engaging customers. I spend most of my days fixing problems. Please note these are my musings, and in no way should reflect my employers opinion. The little free time I have away from work and family I like to use for aviation. Be it flying or building aircraft. Who knows you may see a few Thatcher CX5 build log entries here as well.
Good company in a journey makes the way seem shorter. — Izaak Walton

