Handle trailing/leading method calls and syntax

[schneems]

From #64 the algorithm doesn't work well unless a chunk of code fits logically into the concept of a "line". This assumption fails when we have trailing slashes (mitigated in #28) as well as leading/trailing method calls.

User
  .where(name: "schneems")

The other known leading/trailing syntax is boolean operators:

irb(main):017:0* 1 == 1 &&
irb(main):018:0* 2 == 2 ||
irb(main):019:0> 3 == 3
=> true

[schneems]

We can use a lexer to detect leading/trailing method calls. However it becomes very hard to do without full context.

If you are using the full code it tells us that we have on_ignored_nl elements:

irb(main):005:1" Ripper.lex(<<~EOM
irb(main):006:1"   User
irb(main):007:1"   .where
irb(main):008:1"   .first
irb(main):009:1*   EOM
irb(main):010:0>   )
=>
[[[1, 0], :on_const, "User", CMDARG],
 [[1, 4], :on_ignored_nl, "\n", CMDARG],
 [[2, 0], :on_period, ".", DOT],
 [[2, 1], :on_ident, "where", ARG],
 [[2, 6], :on_ignored_nl, "\n", ARG],
 [[3, 0], :on_period, ".", DOT],
 [[3, 1], :on_ident, "first", ARG],
 [[3, 6], :on_nl, "\n", BEG]]
irb(main):011:1" Ripper.lex(<<~EOM
irb(main):012:1"   User.
irb(main):013:1"   where.
irb(main):014:1"   first
irb(main):015:1*   EOM
irb(main):016:0>   )
=>
[[[1, 0], :on_const, "User", CMDARG],
 [[1, 4], :on_period, ".", DOT],
 [[1, 5], :on_ignored_nl, "\n", DOT],
 [[2, 0], :on_ident, "where", ARG],
 [[2, 5], :on_period, ".", DOT],
 [[2, 6], :on_ignored_nl, "\n", DOT],
 [[3, 0], :on_ident, "first", ARG],
 [[3, 5], :on_nl, "\n", BEG]]

However, if we're lexing without full context, we need to piece that info together after the fact:

Ripper.lex(".first")
=> [[[1, 0], :on_period, ".", DOT], [[1, 1], :on_ident, "first", ARG]]

We also have to consider other elements such as spaces and comments.

irb(main):022:0> Ripper.lex(" where. # trailing method call with a comment")
=>
[[[1, 0], :on_sp, " ", BEG],
 [[1, 1], :on_ident, "where", CMDARG],
 [[1, 6], :on_period, ".", DOT],
 [[1, 7], :on_sp, " ", DOT],
 [[1, 8], :on_comment, "# trailing method call with a comment", DOT]]

It may be possible to piece-meal this together one line at a time, but I'm thinking we can maybe benefit from some technique that lexes the whole document and "stuffs" data back into codelines rather than needing codelines to lex themselves out of context. This could (possibly) also help with the "heredoc problem" which was partially mitigated in #19 but continues to be a thorn in my side.

Worth noting that trailing && also generates ignored newline lex:

irb(main):024:0> Ripper.lex("foo &&\n bar")
=>
[[[1, 0], :on_ident, "foo", CMDARG],
 [[1, 3], :on_sp, " ", CMDARG],
 [[1, 4], :on_op, "&&", BEG],
 [[1, 6], :on_ignored_nl, "\n", BEG],
 [[2, 0], :on_sp, " ", BEG],
 [[2, 1], :on_ident, "bar", ARG]]

The downside of changing code to "stuff" lexing data into code lines is that the CodeLine class was one of the first ones to be written and its fairly coupled to a large number of other test cases so it might be a large refactoring needed just to see if the concept can work. All the lex extraction logic would need to be re-written. Also there's the unknown of what problems might be intractable with this "injection/stuffing" approach. Ultimately I think it's the right thing to move towards rather than trying to manually replicate the lexer logic of detecting edge-cases like on_ignored_nl

[schneems]

Looks like the on_ignored_nl doesn't do exactly what I thought it did. It fires after do as well. I'm not sure why:

irb(main):280:0' Ripper.lex <<~'EOM'
irb(main):281:0'  foo "lol" \
irb(main):282:0'     "lol" do
irb(main):283:0'       blerg do
irb(main):284:0'      puts "lol"
irb(main):285:0'    end
irb(main):286:0'  end
irb(main):287:0>  EOM
=>
[[[1, 0], :on_ident, "foo", CMDARG],
 [[1, 3], :on_sp, " ", CMDARG],
 [[1, 4], :on_tstring_beg, "\"", CMDARG],
 [[1, 5], :on_tstring_content, "lol", CMDARG],
 [[1, 8], :on_tstring_end, "\"", END],
 [[1, 9], :on_sp, " ", END],
 [[1, 10], :on_sp, "\\\n", END],
 [[2, 0], :on_sp, "   ", END],
 [[2, 3], :on_tstring_beg, "\"", END],
 [[2, 4], :on_tstring_content, "lol", END],
 [[2, 7], :on_tstring_end, "\"", END],
 [[2, 8], :on_sp, " ", END],
 [[2, 9], :on_kw, "do", BEG],
 [[2, 11], :on_ignored_nl, "\n", BEG],
 [[3, 0], :on_sp, "     ", BEG],
 [[3, 5], :on_ident, "blerg", CMDARG],
 [[3, 10], :on_sp, " ", CMDARG],
 [[3, 11], :on_kw, "do", BEG],
 [[3, 13], :on_ignored_nl, "\n", BEG],
 [[4, 0], :on_sp, "    ", BEG],
 [[4, 4], :on_ident, "puts", CMDARG],
 [[4, 8], :on_sp, " ", CMDARG],
 [[4, 9], :on_tstring_beg, "\"", CMDARG],
 [[4, 10], :on_tstring_content, "lol", CMDARG],
 [[4, 13], :on_tstring_end, "\"", END],
 [[4, 14], :on_nl, "\n", BEG],
 [[5, 0], :on_sp, "  ", BEG],
 [[5, 2], :on_kw, "end", END],
 [[5, 5], :on_nl, "\n", BEG],
 [[6, 0], :on_kw, "end", END],
 [[6, 3], :on_nl, "\n", BEG]]

I wish some of these tokens had docs :/ Also note that we needed to do a non-escaping heredoc here for the trailing slash otherwise it joins the next line. Lex does the right thing.