There are no specific options for either of these requests, but using some css you can hide the Help button, and adjust the size of the Image Manager window (which will also chnage the Media Manager, File Manager etc. window size).

To hide the Help button

Create a file called plugin.css in /media/jce/css and add the following:

[data-plugin*="manager"] #help {
	display: none;
}

To change the diaog size

Create a file called editor.css in /media/jce/css and add the following (adjust the values as appropriate) :

.mceModal .mce-modal-portrait-full {
    max-width: 1024px;
    max-height: 768px;
}

Please try JCE Pro 2.9.97 RC - https://www.joomlacontenteditor.net/downloads/editor/pro/development

There is now a Help Button option in the parameters for all dialogs that use the File Browser, eg: Image Manager etc. which can be set to No.

For example, for the Image Manager, go to Editor Profiles -> Plugin Parameters -> Image Manager, and set the Help Button option to No.

What about the definition of the height of the file dialog at the front?

That must be done as indicated previously:

To change the diaog size

Create a file called editor.css in /media/jce/css and add the following (adjust the values as appropriate) :

.mceModal .mce-modal-portrait-full {
    max-width: 1024px;
    max-height: 768px;
}

The css rule should be in media/jce/css/editor.css

The css rule above will only affect "Manager" dialogs, eg: Image Manager, Media Manager etc.

For the File Browser, use:

.mceModal .mce-modal-landscape-full {
    max-width: 1024px;
    max-height: 268px;
}

This will not work for File Browser windows opened from a Media Field.

I have a custom field of type JCE Media that a user edits.

This you can set in your template's css file as the modal window is created by the Joomla API, eg:

joomla-dialog dialog {
    width: 80vw;
    height: 80vh;
}

There is a minimum size requirement for the File Browser for it to be practical. I would say height:50vh is about the minimum you should use. 268px is definately too small.

iPhone SE size is just about useable in landscape mode, eg:

joomla-dialog dialog {
    max-width: 667px;
    max-height: 375px;
}

Why are you trying to make the dialog so small?

I would like a smaller height for the file browser, because the user only will have one file there and having a file browser with the height of the entire window is too big.

That is not going to be possible, so you will need to accept the larger dialog window, or find a compromise width and height combination, eg:

joomla-dialog dialog {
    max-width: 640px;
    max-height: 480px;
}