First of all: there are no “relay pins” on the ESP32, only GPIO: General Purpose Input Output; what you do with the pins is your responsibility.
Microcontroller pins usually can be set as inputs or outputs. They also usually start configured as inputs on reset, and may have a pull-up or pull-down internal resistor that may or may not be configured on reset.
From this moment until your mJS script is executed, the pin will remain in the reset default state, you have to take precautions when designing your hardware.
What you see depends on how you connect your relay and how pins are configured at reset. Once mOS finished booting and your script is executed, the pin will be set as an output and will stay at 0 logic level until you write a different level. But, during boot time, it will stay at its default reset state, and that may be output high or input with pull-up, and I guess you don’t want that. Unfortunately that can not be changed, so if it does bother you, you’ll have to design your hardware to accordingly.
There is no “protocol”, no need for a specific order, you just have to call the right function to set the hardware as you want it to be.
GPIO.setup_output(27, 0) (explained here) is what you need to do to correctly setup the GPIO27 pin as an output at 0V and avoid possible glitches on the output. It is logically equivalent to first writing 0 and the setting as an output.
GPIO.set_mode(27, GPIO.MODE_OUTPUT) will set the pin as output, its initial state will be whatever is carried on from prior operations or reset state. In your case it will just do again what you already did.
GPIO.write(27, 0) will set the pin at logic level 0
I personally would not connect a relay directly to a microcontroller pin.