On non-Windows systems, calling either TA_CheckAndSavePKey() or TA_UseTrial() with TA_SYSTEM requires admin permission the first time you make that call. So, the solution to the "TA_E_PERMISSION" error is to either call it the first time with admin permissions (e.g. with something like an installer) or use TA_USER.
>> "I've tried encoding the string "TA_SYSTEM" as UTF-16LE, but something tells me that's not correct."
No, those flags are unsigned integers.