Yeah this seems like a IPC question. The crux of your question is "what is the best way to communicate between the CLI user interface, to the actual program".
Now before I delve into the possibilities I am obligated to ask if it is possible to simply attach a console window directly to said "Windows service". This seems like the simplest and most performant way to go about things. Obviously, I don't know much about your particular use case aside from the fact that you simply want a separate CLI process.
Would you make the service an ASP.NET Core server and communicate with it over a REST-API or something else?
That is certainly a possibility for IPC, and honestly I rather prefer this approach. Docker utilises a very similar approach with its docker API. The benefits of this is obviously extensibility and flexibility. Namely, users of your app can now create their own "CLIs" so to speak or control your service programatically. The downside is the overhead of both spinning up a HTTP server and the actual HTTP call overhead. Spinning off of this idea, you can simplify things and use a TCP server running on localhost, this shaves off some communication overhead.
Another possibility, which is more performant, is to use named pipes. The upside is that it is a lot more performant, in terms of "in-flight" time for comms. The downside is that it is a lot more tricky to work with, and can potentially backfire if you opt to use blocking IO.
The final, more obscure possibility is to use shared memory. On Windows this is possible via the File mapping API. I honestly do not recommend doing this unless you want to hide your communication from the OS for whatever reason (a lot of malware uses a similar technique to obfuscate their IPC comms). I left it here as a technical possibility.
How would you distinguish between the *.exe being run as a service or a CLI?
Not too sure if I follow, because I'm working off the assumption that these are two distinct binaries. This is probably the approach you want to take anyhow if you are going to proceed with this. Much better to decouple the user interface from the actual service.
Are there other things I need to consider that I might not have in my mind yet?
When it comes to developing, I encourage you to first forget about the technical details and draft up a rigorous "API contract", that is, a set of rules defining how the CLI (client) communicate with the service (server). This will help you choose the correct IPC implementation. For example, if you need to send a lot of large binary data across, then TCP might be superior. If there is a lot of parameters that needs to be passed around, then HTTP might be better suited. Once that is done, figure out how to serialise/deserialise data and go from there.