When you first start your company, one of the founders will essentially be the chief financial offier (CFO). While they may not have the full financial background of a CFO, someone will be in charge of managing the numbers behind the business and taking care of generating insights and reports for the company to discuss, reports to investors, managing accountants, and handling state and federal financial filings.
In the beginning, I'd recommend outsourcing payroll and bookkeeping to external firms from the beginning, and save your time for managing cash flow and budgets. Check out Bench.co, Pilot and local bookkeepers as options for bookkeeping services. You'll still be in charge of understanding and using the financial reports, but someone else will be responsible for doing the accounting and creating the consolidated financial statements.
Eventually you'll have to hire a CFO, but the answer for "when" isn't crystal clear. Often you will start with a VP of Finance because you won't need the level of seniority and experience of a CFO for a small company. You can also hire a CFO on a part-time basis; I once worked as a part-time CFO for a startup, and there are many people available to work on this basis. Nomad Financial, Propeller, and Decision CFO are three of many companies that specialize in providing outsourced part-time financial services and have noted experience with startups. Later on, you'll have to have a CFO full-time in house as a senior member of the management team. Exactly when can vary by the company and the other roles that you'd envision the CFO taking on and taking from the other senior executives.
CFOs are often hired to solve a problem, to fix a company's critical financial management processes, rather than proactively to help create a company's culture of using numbers and analysis to guide business decisions, but it need not be that way. A strong CFO that understands strategy and your industry can be a great asset in other ways earlier on. Don't wait to bring in a CFO to clean up problems.