How to implement comma_to_crlf honoring delimiters

Description

The comma_to_crlf function does not take into account quoted delimiters.

For example if your input string is:

alpha,beta,"gamma has, a comma",delta

You can use comma_to_crlf2() rather than comma_to_crlf_honor_delimiters() if you running version 8602 or greater.

then it will blindly break the string on all of the commas, regardless of whether the string is inside a quoted delimiter.

Here is how to implement comma_to_crlf while honoring delimiters.

txt = <<%str%
,,abc,123,"this string,has a comma",,,98767,
%str%

txt2 = comma_to_crlf_honor_delimiters(txt)
showvar(txt2)

function comma_to_crlf_honor_delimiters as c (txt as c )

    dim s as stringscanner
    dim tokens[0] as c

    s = s.Create(txt)
    s.SkipOverWhitespace()

    WHILE .t.
        dim token as c
        dim num as n = s.SkipOverCharacter(",")
        IF num > 0 THEN
            token = " "
            tokens.push(token)
            FOR x = 2 TO num
                token = " "
                tokens.push(token)
            next
        ELSE
            token = s.ScanOverToken("*")
            tokens.push(token)
            s.SkipOverWhitespace()
            num = s.SkipOverCharacter(",")
            IF num < 1 THEN
                exit while
            ELSE
                FOR x = 2 TO num
                    token = " "
                    tokens.push(token)
                next
            END IF
        END IF
        s.SkipOverWhitespace()
    END WHILE

    dim txt2 as c 
    txt2 = tokens.dump()

    comma_to_crlf_honor_delimiters = txt2

end function