If there are red flags that you wouldn't know until working, then yeah I would quit day 1.
Personally, I'm also agoraphobic so unfortunately, it is feasible that I bite off more than I can chew. That said, there was a time where I only worked one day at a job. I ended up getting incredibly sick on my second day (like probably should have gone to a hospital sick) and my boss had told me the day before to text her because she had like 7 kids and couldn't answer the phone. I ended up getting one digit wrong in the number I texted and she didn't text me until what would have been my third day, a couple days later. I explained the mistake and sent proof but apparently they had a no tolerance policy and fired me.