Which parameter in JCE Editor controls the upload file size limit?

(e.g., Editor Profiles → Plugin Parameters → File Browser?)

To set the value globally for all dialogs / plugins, use the Upload file size parameter in Editor Profiles → Editor Parameters → Filesystem.

You can then also set it individually for each plugin / dialogin in Editor Profiles → Plugin Parameters.

Does JCE rely solely on PHP settings such as upload_max_filesize and post_max_size,

or does it also apply its own internal limits?

The value is limited by both PHP ini values upload_max_filesize and post_max_size, in other words you cannot have a value higher than these.

If both apply, what are the correct settings to change in:

PHP.ini

.htaccess

Joomla Global Configuration

JCE Editor Profile settings

What is the correct recommended configuration if I want to allow uploads of 20MB or 50MB files?

In php.ini set:

upload_max_filesize = 20M
post_max_size = 20M

or in htaccess you can set:

php_value upload_max_filesize 20M
php_value post_max_size 20M

In Editor Profiles → Editor Parameters → Filesystem → Upload file size, set the value to 20.