How-to – VMware vCenter 5.1 Installation: Part 1 (SSO Service)


VMware vSphere 5.1.0A was released on October 25th, and 5.1 GA was released in mid-September. Given all of the new components and architecture of vCenter, I wanted to walk through the entire configuration process from scratch assuming a greenfield environment. The original 5.1 release had a lot of, shall we say, bugs and issues related to the SSO service and SSL problems.

This series covers:

  • vCenter 5.1.0A Installation on Windows Server 2008 R2
  • SSO service, Inventory Service, vCenter Server, vSphere client, vSphere Web client, and VUM
  • Configure SSL certificates for all vSphere services

If you work in a very security conscious organization, you may be using SQL SSL encryption. If so, the instructions below have been updated to provide a method of fully implementing a trusted SSL SQL connection with the JDBC driver for the SSO service. You can check if you are forcing SQL SSL encryption by looking in the SQL Server Configuration Manager and looking at the properties of the Protocols for MSSQLSERVER. As shown below, if Force Encryption is set to Yes you will need to modify the JDBC connection string to use SSL.

On another security note the SQL server MUST be configured to allow both Windows integrated authentication AND SQL authentication. SQL authentication is very weak, which makes the use of SSL for the database connection that much more imperative. Should the SQL server only allow Windows integrated authentication you will likely get the following error:

Error 29115.Cannot authenticate to DB.

Use SQL studio to login to your SQL server, open the server properties then use the less secure option of SQL Server and Windows Authentication mode. Restart the SQL services.


These instructions reflect using vSphere 5.1.0A, which has a different database setup than the 5.1 GA release. I would strongly urge you to use only the 5.1.0A release, and not 5.1 GA.

  • I provisioned a Windows Server 2008 R2 SP1 VM, with 2 HDs, and all of the latest Windows updates. This is just for my modest home lab, so I’m not modifying my 3GB/1vCPU configuration, and this will be an all-in-one server configuration. 6GB, and even 8GB of RAM for an all-in-one server is recommended, otherwise vCenter and SSO will run verrrrrrrry slowly. I also created a service account (SVC-VCTR02-001) in AD, and added that account to the local Administrator’s group on what will become the vCenter 5.1 server. You need to ensure the service account also has the “Act as part of the operating system” user right on the vCenter server. If the Administrators group has the right then you are covered. If not, explicitly add the service account to the user right as shown below. Should you need to add the account, reboot the server after so that it takes effect.


  • Login as the newly created vCenter service account and launch the vSphere 5.1.0A (build 880471) installer from the ISO image and you are presented with the following screen.


  • At this point if you try and install the .NET 3.5 SP1 from this screen it will fail, since you need to add the “feature” to the OS. Open the Server Manager and add the .NET Framework 3.5.1 feature and wait for the install to complete.


  • Now you can continue with the vSphere Installer and either do a “Simple Install” or install each component separately. Since I like to customize the installation location and possibly other settings, I don’t want Simple. Unlike prior vCenter installations, there is a new software installation order. So don’t just jump to the vCenter installer and expect it to work.
  • Click on vCenter Single Sign On, then click on Install. Select the appropriate language and wait for the wizard to open. After clicking through the licensing agreements and carefully reading all of the patents, you are presented with the following screen with several options:


  • One of the cool features about the SSO service is that you can have multiple instances, fronted by a hardware load balancer for HA. Since SSO is required for functional vCenter and other VMware services, having a highly available SSO service is important in production environments. Given the greenfield installation in my lab I left the default option selected since I have no existing SSO instances. On the next screen (shown below) you have two more options. To leave the door open to a future HA configuration I chose the second option.


  • Next the installer will prompt you for the password to the default SSO Administrator account. Yes, this is a local account not tied to AD or the Windows host. After SSO is installed, you can configure it for one or more LDAP/AD server and other identity sources, so don’t fret too much about this application password but DO remember it. The password must have at least eight characters, at least one lowercase character, one uppercase character, one number, and one special character. Maximum password length is unknown, but a reader said 32 characters did not work but 16 did.

    Note: Do NOT use the ^ ; * $ ; ” symbols in the password, as it may cause a “Error 29133.Administrator login error.” further on in the installation process.


  • At this point you are presented with a dialog asking what kind of database you want to use. Unfortunately, if you want to use a full blown SQL server the DB creation process is a bit manual. Since I want to use my SQL Server 2008 R2 enterprise instance in my lab, I selected the second choice.


  • Now you can create a database for the SSO service. Unfortunately there are some hard coded restrictions in the SSO service which limit your ability to use customized names for all of the fields. In particular the DB name must only include letters, numbers, underscore (_), the at symbol (@) and the hash (#). No periods, no hyphens, no spaces.There is a custom database creation script, based on the canned VMware script mentioned in the wizard above. In my case I called the database “D001_VMware_SSO”. Run this script in SQL studio, modified to your liking. Note that you CAN NOT change “RSA_DATA” or “RSA_INDEX” as the SSO service is hard coded to use them and the install WILL fail if they are not present.
         FILENAME='K:\Microsoft SQL Server\MSSQL\Data\D001_VMware_SSO_Data.mdf',
         FILENAME='K:\Microsoft SQL Server\MSSQL\Data\D001_VMware_SSO_Index.mdf',
     LOG ON(
         FILENAME='L:\Microsoft SQL Server\MSSQL\Data\Logs\D001_VMware_SSO_Log.ldf',
         FILEGROWTH=10% )
     EXEC SP_DBOPTION 'D001_VMware_SSO', 'autoshrink', true
     EXEC SP_DBOPTION 'D001_VMware_SSO', 'trunc. log on chkpt.', true
  • Unlike the 5.1 GA version which automatically created two SQL logins to access the database, in 5.1.0A you need to do additional account configuration. In SQL Studio we need to create two logins for the SSO service. These logins need to be created under the main security container in SQL studio, then added to your SSO database.

Create a new user using SQL Server authentication, such as SSO_User, and modify the password options as needed. I would suggest not expiring the password, and certainly can’t select the option to change at next login. The password must be 32 characters or less, and cannot contain semicolon (;), double quotation mark (“), single quotation mark (‘), circumflex (^), and backslash (\).


  • Repeat the process but create a second user, say SSO_DBA.
  • In SQL Studio on your SSO database you now need to add your two newly created users so they have proper permissions in the database. For the SSO_DBA user I granted it db_owner permissions for both the schemas and database. For the SSO_User I only granted db_owner for the database.


  • Now that the database is properly configured, switch back to the vCenter server and enter the database information you used. If you do NOT want to use SQL SSL encryption you can click next and proceed to step 15.


  • If you are concerned about security and want to use an SSL connection to the SQL server you have two implementation options. This assumes your SQL server is configured for SSL. If it is the first, the less secure but easier option is just to amend the JDBC connection string and blindly trust the SQL SSL certificate. Not ideal, but better than clear connectivity to the SQL server. Second, you can configure a Java keystore with the public key of your SQL SSL certificate CA so the JDBC driver can validate the certificate. This is more secure and the preferred option.

To configure blindly trusted SQL SSL follow these steps:

On the vCenter Single Sign On screen above, click on Next, then click on Back. You should now see the JDBC URL populated but greyed out. Check the box next to “I will enter the JDBC URL myself”. At the end of the connection string add the following two options:


For example, my JDBC URL is (single line):


Your window should now look like the screenshot below. Click on Next, and if everything is validated, no errors will appear.


To configure fully trusted SQL SSL follow these steps:

  • First you need to export the public root CA certificate to a DER encoded file so the Java keytool can import it. In my case I use an internal MS CA.You can do this a variety of ways, but I like using the MMC tool to do this. The process assumes your vCenter server has the root CA certificate in its certificate store. 
  • Open a blank MMC, add the Certificates snap-in, and manage the Computer account.
  • Open the Trusted Root Certification Authority then browse through the certificates to find the one matching the Certificate Authority that was used to issue the certificate to your SQL server. Once you find the certificate, right click an Export it. Click through the wizard until you get to the screen below.


  • Make sure DER encoded is selected and export the certificate. I called it root.cer and put it at the root of my C:\ drive for easy access. Note: If you went crazy with your CA and configured it for SHA256 and elliptical curve encryption algorithms then the Java keytool won’t import the certificate.
  • Open an elevated command prompt then change directory to where the SSO installer files extracted themselves and find the jre\bin directory. In my case this was:
  • Create a Java keystore by issuing the following command (using the path of your SSO service account). Enter a keystore password (write it down). When prompted to trust the certificate type Yes.
     keytool -import  -alias root -file c:\root.cer -keystore c:\users\SVC-VCTR02-001\jks.keystore


  • With the keystore now created, it is time to update the JDBC connection string back in the SSO installer. Add the ‘truststore’ and ‘truststorepassword’ connection options with the appropriate path and password. If everything validates you should see the screenshot in step 15.
     jdbc:sqlserver://;;port=1433; databaseName=D001_VMware_SSO;integratedSecurity=true; encrypt=true;truststore=C:\Users\SVC-VCTR02-001\jks.keystore;truststorepassword=testpassword
  • Here, if you were using a hardware load balancer, you would enter the FQDN of the VIP. Since I’m just installing one SSO instance, I’ll stick with the FQDN of the vCenter server.


  • At this point I input the vCenter service account details. Note that if you input the wrong password you will get an error “Could not find the specified user on provided domain.” which is not entirely correct. The user exists but you just fat fingered the password.


  • On the next screen I left the HTTPS port the default, then sent the installer off on its merry way.
  • After the SSO service installs, it generates a number of unique passwords. For safe keeping, I would store these passwords in a secure location. Although you may never need most of them, several passwords are lost forever when you install your own SSL certificate. Should you need to access the default keystores, you are out of luck. Open an elevated prompt and type:

    “c:\Program Files\VMware\Infrastructure\SSOServer\utils\rsautil” manage-secrets -a list