This is a newbie question, but I need create a TCP server in C# that our partner's client (built with OpenSSL) can connect to.

I am implementing the SecureTcpListener class and was given a set of der encoded certificate files: root, intermediate, server, and key (which also seemed to be der encoded). I created a certificate store by loading up the root .cer file and then tried associating the private key file but it said it wasn't a valid pvk file. If I try to run it without associating a private key file it throws an error saying a private key is missing.

How do I create a pvk file from the der encoded key file I was given. Also, do I need to combine the intermediate and root certificates in some way?

Are there any incompatibilities between Mentalis.org's SSL implementation and OpenSSL?

By default, OpenSSL and the CryptoAPI use a different file format for storing certificates and keys. You'll have to convert a certificate/key to a format that both can understand. PVK is a Microsoft specific file format.

Your best choice is to use PFX [alias PKCS#12 files] files. OpenSSL can convert your key to PFX format and the CryptoAPI can import those files.

First you'll have to convert your private key to the PEM file format. You can use the following command if you have a compiled version of OpenSSL installed:
openssl pkcs8 -inform DER -in key.der -out key.pem

After that you can convert the PEM encoded certificate and key to a PKCS#12
file. You can use the following command:
openssl pkcs12 -export -in certificate.pem -inkey key.pem -out
certificate.pfx -name "My Certificate"

You can then either load the PFX file directly from the Security Library [CertificateStore.CreateFromPfxFile(...)] or import them manually into a certificate store [by double-clicking the pfx file in explorer for instance] and then load them by opening the certificate store for your code.

> Also, do I need to combine the intermediate and root
> certificates in some way?

all wil be well if you place the server certificate together with the intermediate certificates in the same certificate store, and the root certificate in the ROOT store.

> Are there any incompatibilities between Mentalis.org's
> SSL implementation and OpenSSL?

Not that I'm aware of. Keep in mind that our implementation only supports RSA key exchanges; Diffie-Hellman key exchanges are not supported, but this is usually not a problem.

P.S.: the documentation of the OpenSSL conversion tools can be found here:
[PKCS8 == DER, PKCS12 == PFX]

http://www.mkssoftware.com/docs/man1/openssl_pkcs8.1.asp

and

http://www.mkssoftware.com/docs/man1/openssl_pkcs12.1.asp

would this be the right way to load certificates and add to the root store?

...

CertificateStore store = CertificateStore.CreateFromCerFile("server_cert.cer");
CertificateStore rootStore = new CertificateStore("ROOT");

Certificate cert = store.FindCertificate();
cert.AssociateWithPrivateKey("key.pvk", "pass1", true);

Certificate intermediate = Certificate.CreateFromCerFile("intermediate_cert.cer");
Certificate rootcert = Certificate.CreateFromCerFile("root_cert.cer");
store.AddCertificate(intermediate);
rootStore.AddCertificate(rootcert);

SecurityOptions options = new SecurityOptions(SecureProtocol.Ssl3,cert,ConnectionEnd.Server);

//Create an instance of sslListener to listen for SSL connection.
SecureTcpListener sslListener = new SecureTcpListener(localAddr, tcpPort, options);


...

Theoretically, yes, but installing root certificates programmatically is never done in practice. This is usually done by the Administrator of a computer [manually].
If you run the code you posted, you'll get a big dialog box asking the user if he wants to install a certificate in the root store.
Are you sure this is what you want?

Ah ok. then I only want to load the server certificate programmatically then?

Yes, and potentially the intermediate certificates. You shouldn't bother with the root certificate because most of the time the SSL servers don't send the root CA certificate to the client [the point of certificate authorities is that the client already has the root CA certificate].