The certificate has to be text-based PEM (base64) encoded . Usually the files will have a .pem file extension but it's common to see them with the generic extension of .cer or .crt

The appliance doesn't like the binary format DER which is usually what Windows will create certificates as. These usually have the .der extension but it's common to see it with a .cer or .crt extension too.

Ideally the certificate admins should export these out in a PEM encoded certificate and use the .pem, .cer or .crt file extension . If not, opening the file with Windows certificate store should allow the files to be exported to a base64 format .cer file.

The CA bundle can be a .p7b file extension but has to be base64 encoded. The docs Configuring HTTPS settings  state
"If you are using Microsoft Certificate Services, you must use the Download CA certificate chain option
with Base 64 encoding. This produces a PKCS #7 encoded file (.p7b) that is automatically converted to the correct format during the upload."