The popup title and caption text styles are inherited from your template styles, as they use the H4 heading and paragraph elements respectively. There are two methods you can use to change these styles:

Edit the JCE MediaBox CSS file

Depending on the JCE MediaBox theme selected (the default being 'standard') open the css file in plugins/system/jcemediabox/themes/[theme]/css/style.css and add the necessary style changes to the empty #jcepopup-caption h4 and #jcepopup-caption p classes, eg:

 #jcepopup-caption h4 {
    font-size: 16px;
    font-weight: bold;
}
#jcepopup-caption p {
    font-size: 12px;
}

Edit the Template CSS file

This is essentially the same as the method above, but the css changes are added to your template css file. These changes would therefore apply to all JCE Mediabox popup themes, unless the method above is also used, in which case those styles would take precedence.

In this method, add the classes to your template.css file in templates/[template]/css/, eg:

 #jcepopup-caption h4 {
    font-size: 16px;
    font-weight: bold;
}
#jcepopup-caption p {
    font-size: 12px;
}