I can understand your apprehension. I faced the same nervousness when I took my first set of public visual notes. It gets better with time.
I suggest you go for it and if people are curious about what you are doing, you have the opportunity to introduce them to this wonderful thing called visual note-taking. If someone is being critical about what you’re doing, hear them out-learn from their critique and use it to improve you note-taking skills.
When I do notes, I remind myself that my primary purpose of doing this is for ‘me’, so the quality of the drawings/notes really don’t matter, unless you want to go and make them public.
Relax and enjoy the experience.
Best of luck!