If you have trouble getting past the Database Information Screen during setup, with error connecting to Database server, then this might help.
1. Ensure you are not using dynamic TCPIP ports in SQL Server. I used default port 1433.
2. Do not use Windows Authentication ( I dont know why VMWare provided that option, if they dont support it ). You must have mixed mode authentication enabled in your SQL Server.
3. You should have run the DBscripts (rsaIMSLiteMSSQLSetupTablespaces.sql and rsaIMSLiteMSSQLSetupUsers) provided in VMWare install media. Before running the script, Copy and modify the SQL script to point to right install path and specify the password in the users script. No need to run rsaIMSLiteMSSQLCreateSchema script.
4. Enable “Use Manually created DB users” check box.
5. Specify Database user name : RSA_USER and password you specified in Step3.
6. Specify Database user name : RSA_DBA and password you specified in Step3.
7. Check the “I will enter the JDBC URL myself”. Provide this
jdbc:sqlserver://<serverName>:<portNumber>;instanceName=<InstanceName>;databaseName=RSA
ofcourse, modify it as per your environment. For InstanceName, specify <Servername>\<instancename>
Click Next.. Voila…