I had the same problem with the blue (on while fabric, of course) and will never use the chalk again. While it brushes right off of the fabric before quilting, I find that the thread locks it underneath.

I first soaked and washed in plain cold water, then used a paste made up of oxy-clean spray & powder and gently massaged with a toothbrush to get underneath the stitches and let it sit for several hours. After about 4-5 treatments (and washings) like this, it finally came clean. Do not dry in the dryer between washings, just air dry. Not easy in an RV and quite nerve wracking to say the least as we were on our way to Michigan to deliver the quilt to my best friend of 30+ years. Best of luck!

Thanks Pamar for the suggestion of using the Mr. Clean Magic Eraser. Between that and using a toothbrush it got 98% of the blue line out. Just a couple of small spots left. Luckily this is a quilt for my 6 year old granddaughter so she won't be to critical. I will be more careful on what I use next time. By the way, I had trouble with some lines on fleece. also. This same process helped a little but not much. Need to find another marking tool for fleece next time.