The markup created by the Captions tool is a little bit dated - Captions was created before the <figure> element was available - so it might be better to use Figure from the Format list instead. This will wrap the image in a <figure> tag with an accompanying <figcation> tag which you can write or paste the caption into.

Figure tags can be styled using the Captions tools, or you can style them using a css selector, eg:

figure {
    border: 1px solid #ccc;
}
figcaption {
    background-color: #eee
}

The <figure> tag is a block element, so it cannot exist inside a paragraph, but it is usually used with full width images or media. I use it for images in news articles - https://www.joomlacontenteditor.net/news/jce-pro-2-9-82-released

Where you need "inline" captions inside paragraphs, as in the "Christy Beal" image here - https://sas.rutgers.edu/about/news/students/students-news-detail/the-one-credit-elective-thats-helping-rutgers-students-survive-general-biology - then the Captions tools are still the best solution. You can style them with css using the "w_caption" selector, eg:

.wf_caption {
    border: 1px solid #ccc;
}
/* caption text */
.wf_caption span {
    background-color: #eee;
}