[codec,rfx] return nullptr if input parameters are invalid

rfx_encode_messages maxDataSize must be > 1024 or the function will
fail. Add a check to abort early if this is the case.
while this is a usage error, it is helpful to have proper error handling
in the function as well.

libfreerdp/codec/rfx.c

@@ -1952,6 +1952,9 @@ static inline RFX_MESSAGE* rfx_split_message(RFX_CONTEXT* WINPR_RESTRICT context
 	WINPR_ASSERT(message);
 	WINPR_ASSERT(numMessages);
 
+	if (maxDataSize <= 1024)
+		return nullptr;
+
 	maxDataSize -= 1024; /* reserve enough space for headers */
 	*numMessages = ((message->tilesDataSize + maxDataSize) / maxDataSize) * 4ull;