If the professor can't attest to stellar academic ability, you might want to ask him what the best recommendation he'd be willing to write might look like.

You obviously can't hide your transcript from schools you're applying to, so if he writes, "rainbowschubert isn't the best at coursework" that won't be a surprise to the admissions committee. If he can write "rainbowschubert isn't the best at coursework with respect to grades, but based upon my interactions with them, I have no concerns about their ability to successfully complete a graduate curriculum" -- that's even better. Combined with something along the lines of "This student shows great curiosity, asks intelligent questions in the lab, shows outstanding lab technique, is dedicated and trustworthy, and works without supervision" -- now we're talking about something that might support a decision for admission.

My recommendation would be to go have an adult conversation with your supervisor asking him if he's comfortable with writing any recommendation, even if it's not glowing, and whether such a recommendation could positively support any aspect of your whole portfolio. A question that sometimes appears in admissions questionnaires for recommenders is "would you hire this student". You might ask if he would answer positively.

To back this up, when I'm asked to write a recommendation for a student I would think of as marginal, I have a meeting with the student and nicely, but clearly, tell them the best type of recommendation I could write for them before agreeing to write it. Sometimes they'll say "thank you -- that would be of help because you're the only person in a position to make this particular point", and other times they move on to the next person on their list.

If you're going to have such a conversation, do your best to not take the responses personally, though that can be difficult.

If the professor can't attest to stellar academic ability, you might want to ask him what the best recommendation he'd be willing to write might look like.

You obviously can't hide your transcript from schools you're applying to, so if he writes, "rainbowschubert isn't the best at coursework" that won't be a surprise to the admissions committee. If he can write "rainbowschubert isn't the best at coursework with respect to grades, but based upon my interactions with them, I have no concerns about their ability to successfully complete a graduate curriculum" -- that's even better. Combined with something along the lines of "This student shows great curiosity, asks intelligent questions in the lab, shows outstanding lab technique, is dedicated and trustworthy, and works without supervision" -- now we're talking about something that might support a decision for admission.

My recommendation would be to go have an adult conversation with your supervisor asking him if he's comfortable with writing any recommendation, even if it's not glowing, and whether such a recommendation could positively support any aspect of your whole portfolio. A question that sometimes appears in admissions questionnaires for recommenders is "would you hire this student". You might ask if he would answer positively.

To back this up, when I'm asked to write a recommendation for a student I would think of as marginal, I have a meeting with the student and nicely, but clearly, tell them the best type of recommendation I could write for them before agreeing to write it. Sometimes they'll say "thank you -- that would be of help because you're the only person in a position to make this particular point", and other times they move on to the next person on their list.

If you're going to have such a conversation, do your best to not take the responses personally, though that can be difficult.

UPDATE: the prof seems to feel that the highest probability plan for you to get admitted to a PhD program is to start in a MS to demonstrate that you can get good grades before applying for the PhD. Lot's of students with less than stellar transcripts plan that exact route. I can't tell you whether they are successful in getting into a PhD program or not.