这个问题比较细，估计只有做源码分析的盆友会有点兴趣。我在平时会用srcML，javalang，Understand做tokenization，有时也会一起使用，但是发现这些工具有不一致的地方，所以简单以一个例子记录一下：

例如下面这段Java代码：

class Test{public WndJournal(){int width = SPDSettings.landscape() ? WIDTH_L : WIDTH_P;int height = SPDSettings.landscape() ? HEIGHT_L : HEIGHT_P;resize(width, height);guideTab = new GuideTab();add(guideTab);guideTab.setRect(0, 0, width, height);guideTab.updateList();alchemyTab = new AlchemyTab();add(alchemyTab);alchemyTab.setRect(0, 0, width, height);notesTab = new NotesTab();add(notesTab);notesTab.setRect(0, 0, width, height);notesTab.updateList();catalogTab = new CatalogTab();add(catalogTab);catalogTab.setRect(0, 0, width, height);catalogTab.updateList();Tab[] tabs = {new IconTab( new ItemSprite(ItemSpriteSheet.GUIDE_PAGE, null) ) {protected void select( boolean value ) {super.select( value );guideTab.active = guideTab.visible = value;if (value) last_index = 0;}},new IconTab( new ItemSprite(ItemSpriteSheet.ALCH_PAGE, null) ) {protected void select( boolean value ) {super.select( value );alchemyTab.active = alchemyTab.visible = value;if (value) last_index = 1;}},new IconTab( Icons.get(Icons.DEPTH) ) {protected void select( boolean value ) {super.select( value );notesTab.active = notesTab.visible = value;if (value) last_index = 2;}},new IconTab( new ItemSprite(ItemSpriteSheet.WEAPON_HOLDER, null) ) {protected void select( boolean value ) {super.select( value );catalogTab.active = catalogTab.visible = value;if (value) last_index = 3;}}};for (Tab tab : tabs) {add( tab );}layoutTabs();select(last_index);}
}

如果用javalang对method内部的代码做tokenization是这个结果：

int
width
=
SPDSettings
(
)
?
WIDTH_L
:
WIDTH_P
;
int
height
=
SPDSettings
(
)
?
HEIGHT_L
:
HEIGHT_P
;
resize
(
width
,
height
)
;
guideTab
=
new
GuideTab
(
)
;
add
(
guideTab
)
;
guideTab
(
0
,
0
,
width
,
height
)
;
guideTab
(
)
;
alchemyTab
=
new
AlchemyTab
(
)
;
add
(
alchemyTab
)
;
alchemyTab
(
0
,
0
,
width
,
height
)
;
notesTab
=
new
NotesTab
(
)
;
add
(
notesTab
)
;
notesTab
(
0
,
0
,
width
,
height
)
;
notesTab
(
)
;
catalogTab
=
new
CatalogTab
(
)
;
add
(
catalogTab
)
;
catalogTab
(
0
,
0
,
width
,
height
)
;
catalogTab
(
)
;
Tab
[
]
tabs
=
{
new
IconTab
(
new
ItemSprite
(
ItemSpriteSheet
.
GUIDE_PAGE
,
null
)
)
{
protected
void
select
(
boolean
value
)
{
super
.
select
(
value
)
;
guideTab
.
active
=
guideTab
.
visible
=
value
;
if
(
value
)
last_index
=
0
;
}
}
,
new
IconTab
(
new
ItemSprite

这也是很好理解的，如果用Understand做tokenization，如果我们把每一个token当做一行输出，结果是类似于这个样子：

class|&|Keyword|&|1
Test|&|Identifier|&|1
{|&|Punctuation|&|1
public|&|Keyword|&|2
WndJournal|&|Identifier|&|2
(|&|Punctuation|&|2
)|&|Punctuation|&|2
{|&|Punctuation|&|2
int|&|Keyword|&|4
width|&|Identifier|&|4
=|&|Operator|&|4
SPDSettings|&|Identifier|&|4
.|&|Punctuation|&|4
landscape|&|Identifier|&|4
(|&|Punctuation|&|4
)|&|Punctuation|&|4
?|&|Operator|&|4
WIDTH_L|&|Identifier|&|4
:|&|Operator|&|4
WIDTH_P|&|Identifier|&|4
;|&|Punctuation|&|4
int|&|Keyword|&|5
height|&|Identifier|&|5
=|&|Operator|&|5
SPDSettings|&|Identifier|&|5
.|&|Punctuation|&|5
landscape|&|Identifier|&|5
(|&|Punctuation|&|5
)|&|Punctuation|&|5
?|&|Operator|&|5
HEIGHT_L|&|Identifier|&|5
:|&|Operator|&|5
HEIGHT_P|&|Identifier|&|5
;|&|Punctuation|&|5
resize|&|Identifier|&|7
(|&|Punctuation|&|7
width|&|Identifier|&|7
,|&|Punctuation|&|7
height|&|Identifier|&|7
)|&|Punctuation|&|7
;|&|Punctuation|&|7
guideTab|&|Identifier|&|9
=|&|Operator|&|9
new|&|Keyword|&|9
GuideTab|&|Identifier|&|9
(|&|Punctuation|&|9
)|&|Punctuation|&|9
;|&|Punctuation|&|9
add|&|Identifier|&|10
(|&|Punctuation|&|10
guideTab|&|Identifier|&|10
)|&|Punctuation|&|10
;|&|Punctuation|&|10
guideTab|&|Identifier|&|11
.|&|Punctuation|&|11
setRect|&|Identifier|&|11
(|&|Punctuation|&|11
0|&|Literal|&|11
,|&|Punctuation|&|11
0|&|Literal|&|11
,|&|Punctuation|&|11
width|&|Identifier|&|11
,|&|Punctuation|&|11
height|&|Identifier|&|11
)|&|Punctuation|&|11
;|&|Punctuation|&|11
guideTab|&|Identifier|&|12
.|&|Punctuation|&|12
updateList|&|Identifier|&|12
(|&|Punctuation|&|12
)|&|Punctuation|&|12
;|&|Punctuation|&|12
alchemyTab|&|Identifier|&|14
=|&|Operator|&|14
new|&|Keyword|&|14
AlchemyTab|&|Identifier|&|14
(|&|Punctuation|&|14
)|&|Punctuation|&|14
;|&|Punctuation|&|14
add|&|Identifier|&|15
(|&|Punctuation|&|15
alchemyTab|&|Identifier|&|15
)|&|Punctuation|&|15
;|&|Punctuation|&|15
alchemyTab|&|Identifier|&|16
.|&|Punctuation|&|16
setRect|&|Identifier|&|16
(|&|Punctuation|&|16
0|&|Literal|&|16
,|&|Punctuation|&|16
0|&|Literal|&|16
,|&|Punctuation|&|16
width|&|Identifier|&|16
,|&|Punctuation|&|16
height|&|Identifier|&|16
)|&|Punctuation|&|16
;|&|Punctuation|&|16
notesTab|&|Identifier|&|18
=|&|Operator|&|18
new|&|Keyword|&|18
NotesTab|&|Identifier|&|18
(|&|Punctuation|&|18
)|&|Punctuation|&|18
;|&|Punctuation|&|18
add|&|Identifier|&|19
(|&|Punctuation|&|19
notesTab|&|Identifier|&|19
)|&|Punctuation|&|19
;|&|Punctuation|&|19
notesTab|&|Identifier|&|20
.|&|Punctuation|&|20
setRect|&|Identifier|&|20
(|&|Punctuation|&|20
0|&|Literal|&|20
,|&|Punctuation|&|20
0|&|Literal|&|20
,|&|Punctuation|&|20
width|&|Identifier|&|20
,|&|Punctuation|&|20
height|&|Identifier|&|20
)|&|Punctuation|&|20
;|&|Punctuation|&|20
notesTab|&|Identifier|&|21
.|&|Punctuation|&|21
updateList|&|Identifier|&|21
(|&|Punctuation|&|21
)|&|Punctuation|&|21
;|&|Punctuation|&|21
catalogTab|&|Identifier|&|23
=|&|Operator|&|23
new|&|Keyword|&|23
CatalogTab|&|Identifier|&|23
(|&|Punctuation|&|23
)|&|Punctuation|&|23
;|&|Punctuation|&|23
add|&|Identifier|&|24
(|&|Punctuation|&|24
catalogTab|&|Identifier|&|24
)|&|Punctuation|&|24
;|&|Punctuation|&|24
catalogTab|&|Identifier|&|25
.|&|Punctuation|&|25
setRect|&|Identifier|&|25
(|&|Punctuation|&|25
0|&|Literal|&|25
,|&|Punctuation|&|25
0|&|Literal|&|25
,|&|Punctuation|&|25
width|&|Identifier|&|25
,|&|Punctuation|&|25
height|&|Identifier|&|25
)|&|Punctuation|&|25
;|&|Punctuation|&|25
catalogTab|&|Identifier|&|26
.|&|Punctuation|&|26
updateList|&|Identifier|&|26
(|&|Punctuation|&|26
)|&|Punctuation|&|26
;|&|Punctuation|&|26
Tab|&|Identifier|&|28
[|&|Punctuation|&|28
]|&|Punctuation|&|28
tabs|&|Identifier|&|28
=|&|Operator|&|28
{|&|Punctuation|&|28
new|&|Keyword|&|29
IconTab|&|Identifier|&|29
(|&|Punctuation|&|29
new|&|Keyword|&|29
ItemSprite|&|Identifier|&|29
(|&|Punctuation|&|29
ItemSpriteSheet|&|Identifier|&|29
.|&|Punctuation|&|29
GUIDE_PAGE|&|Identifier|&|29
,|&|Punctuation|&|29
null|&|Keyword|&|29
)|&|Punctuation|&|29
)|&|Punctuation|&|29
{|&|Punctuation|&|29
protected|&|Keyword|&|30
void|&|Keyword|&|30
select|&|Identifier|&|30
(|&|Punctuation|&|30
boolean|&|Keyword|&|30
value|&|Identifier|&|30
)|&|Punctuation|&|30
{|&|Punctuation|&|30
super|&|Keyword|&|31
.|&|Punctuation|&|31
select|&|Identifier|&|31
(|&|Punctuation|&|31
value|&|Identifier|&|31
)|&|Punctuation|&|31
;|&|Punctuation|&|31
guideTab|&|Identifier|&|32
.|&|Punctuation|&|32
active|&|Identifier|&|32
=|&|Operator|&|32
guideTab|&|Identifier|&|32
.|&|Punctuation|&|32
visible|&|Identifier|&|32
=|&|Operator|&|32
value|&|Identifier|&|32
;|&|Punctuation|&|32
if|&|Keyword|&|33
(|&|Punctuation|&|33
value|&|Identifier|&|33
)|&|Punctuation|&|33
last_index|&|Identifier|&|33
=|&|Operator|&|33
0|&|Literal|&|33
;|&|Punctuation|&|33
}|&|Punctuation|&|34
}|&|Punctuation|&|35
,|&|Punctuation|&|35
new|&|Keyword|&|36
IconTab|&|Identifier|&|36
(|&|Punctuation|&|36
new|&|Keyword|&|36
ItemSprite|&|Identifier|&|36
(|&|Punctuation|&|36
ItemSpriteSheet|&|Identifier|&|36
.|&|Punctuation|&|36
ALCH_PAGE|&|Identifier|&|36
,|&|Punctuation|&|36
null|&|Keyword|&|36
)|&|Punctuation|&|36
)|&|Punctuation|&|36
{|&|Punctuation|&|36
protected|&|Keyword|&|37
void|&|Keyword|&|37
select|&|Identifier|&|37
(|&|Punctuation|&|37
boolean|&|Keyword|&|37
value|&|Identifier|&|37
)|&|Punctuation|&|37
{|&|Punctuation|&|37
super|&|Keyword|&|38
.|&|Punctuation|&|38
select|&|Identifier|&|38
(|&|Punctuation|&|38
value|&|Identifier|&|38
)|&|Punctuation|&|38
;|&|Punctuation|&|38
alchemyTab|&|Identifier|&|39
.|&|Punctuation|&|39
active|&|Identifier|&|39
=|&|Operator|&|39
alchemyTab|&|Identifier|&|39
.|&|Punctuation|&|39
visible|&|Identifier|&|39
=|&|Operator|&|39
value|&|Identifier|&|39
;|&|Punctuation|&|39
if|&|Keyword|&|40
(|&|Punctuation|&|40
value|&|Identifier|&|40
)|&|Punctuation|&|40
last_index|&|Identifier|&|40
=|&|Operator|&|40
1|&|Literal|&|40
;|&|Punctuation|&|40
}|&|Punctuation|&|41
}|&|Punctuation|&|42
,|&|Punctuation|&|42
new|&|Keyword|&|43
IconTab|&|Identifier|&|43
(|&|Punctuation|&|43
Icons|&|Identifier|&|43
.|&|Punctuation|&|43
get|&|Identifier|&|43
(|&|Punctuation|&|43
Icons|&|Identifier|&|43
.|&|Punctuation|&|43
DEPTH|&|Identifier|&|43
)|&|Punctuation|&|43
)|&|Punctuation|&|43
{|&|Punctuation|&|43
protected|&|Keyword|&|44
void|&|Keyword|&|44
select|&|Identifier|&|44
(|&|Punctuation|&|44
boolean|&|Keyword|&|44
value|&|Identifier|&|44
)|&|Punctuation|&|44
{|&|Punctuation|&|44
super|&|Keyword|&|45
.|&|Punctuation|&|45
select|&|Identifier|&|45
(|&|Punctuation|&|45
value|&|Identifier|&|45
)|&|Punctuation|&|45
;|&|Punctuation|&|45
notesTab|&|Identifier|&|46
.|&|Punctuation|&|46
active|&|Identifier|&|46
=|&|Operator|&|46
notesTab|&|Identifier|&|46
.|&|Punctuation|&|46
visible|&|Identifier|&|46
=|&|Operator|&|46
value|&|Identifier|&|46
;|&|Punctuation|&|46
if|&|Keyword|&|47
(|&|Punctuation|&|47
value|&|Identifier|&|47
)|&|Punctuation|&|47
last_index|&|Identifier|&|47
=|&|Operator|&|47
2|&|Literal|&|47
;|&|Punctuation|&|47
}|&|Punctuation|&|48
}|&|Punctuation|&|49
,|&|Punctuation|&|49
new|&|Keyword|&|50
IconTab|&|Identifier|&|50
(|&|Punctuation|&|50
new|&|Keyword|&|50
ItemSprite|&|Identifier|&|50
(|&|Punctuation|&|50
ItemSpriteSheet|&|Identifier|&|50
.|&|Punctuation|&|50
WEAPON_HOLDER|&|Identifier|&|50
,|&|Punctuation|&|50
null|&|Keyword|&|50
)|&|Punctuation|&|50
)|&|Punctuation|&|50
{|&|Punctuation|&|50
protected|&|Keyword|&|51
void|&|Keyword|&|51
select|&|Identifier|&|51
(|&|Punctuation|&|51
boolean|&|Keyword|&|51
value|&|Identifier|&|51
)|&|Punctuation|&|51
{|&|Punctuation|&|51
super|&|Keyword|&|52
.|&|Punctuation|&|52
select|&|Identifier|&|52
(|&|Punctuation|&|52
value|&|Identifier|&|52
)|&|Punctuation|&|52
;|&|Punctuation|&|52
catalogTab|&|Identifier|&|53
.|&|Punctuation|&|53
active|&|Identifier|&|53
=|&|Operator|&|53
catalogTab|&|Identifier|&|53
.|&|Punctuation|&|53
visible|&|Identifier|&|53
=|&|Operator|&|53
value|&|Identifier|&|53
;|&|Punctuation|&|53
if|&|Keyword|&|54
(|&|Punctuation|&|54
value|&|Identifier|&|54
)|&|Punctuation|&|54
last_index|&|Identifier|&|54
=|&|Operator|&|54
3|&|Literal|&|54
;|&|Punctuation|&|54
}|&|Punctuation|&|55
}|&|Punctuation|&|56
}|&|Punctuation|&|57
;|&|Punctuation|&|57
for|&|Keyword|&|59
(|&|Punctuation|&|59
Tab|&|Identifier|&|59
tab|&|Identifier|&|59
:|&|Operator|&|59
tabs|&|Identifier|&|59
)|&|Punctuation|&|59
{|&|Punctuation|&|59
add|&|Identifier|&|60
(|&|Punctuation|&|60
tab|&|Identifier|&|60
)|&|Punctuation|&|60
;|&|Punctuation|&|60
}|&|Punctuation|&|61
layoutTabs|&|Identifier|&|63
(|&|Punctuation|&|63
)|&|Punctuation|&|63
;|&|Punctuation|&|63
select|&|Identifier|&|65
(|&|Punctuation|&|65
last_index|&|Identifier|&|65
)|&|Punctuation|&|65
;|&|Punctuation|&|65
}|&|Punctuation|&|66
}|&|Punctuation|&|67
|&|EOF|&|67

如果用srcML输出，则是这个样子：

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<unit xmlns="http://www.srcML.org/srcML/src" revision="0.9.5" language="Java" filename="Test/1/186.java-Filter.java"><class>class <name>Test</name><block>{<constructor><specifier>public</specifier> <name>WndJournal</name><parameter_list>()</parameter_list><block>{<decl_stmt><decl><type><name>int</name></type> <name>width</name> <init>= <expr><ternary><condition><expr><call><name><name>SPDSettings</name><operator>.</operator><name>landscape</name></name><argument_list>()</argument_list></call></expr> ?</condition><then> <expr><name>WIDTH_L</name></expr> </then><else>: <expr><name>WIDTH_P</name></expr></else></ternary></expr></init></decl>;</decl_stmt><decl_stmt><decl><type><name>int</name></type> <name>height</name> <init>= <expr><ternary><condition><expr><call><name><name>SPDSettings</name><operator>.</operator><name>landscape</name></name><argument_list>()</argument_list></call></expr> ?</condition><then> <expr><name>HEIGHT_L</name></expr> </then><else>: <expr><name>HEIGHT_P</name></expr></else></ternary></expr></init></decl>;</decl_stmt><expr_stmt><expr><call><name>resize</name><argument_list>(<argument><expr><name>width</name></expr></argument>, <argument><expr><name>height</name></expr></argument>)</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><name>guideTab</name> <operator>=</operator> <operator>new</operator> <call><name>GuideTab</name><argument_list>()</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name>add</name><argument_list>(<argument><expr><name>guideTab</name></expr></argument>)</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name><name>guideTab</name><operator>.</operator><name>setRect</name></name><argument_list>(<argument><expr><literal type="number">0</literal></expr></argument>, <argument><expr><literal type="number">0</literal></expr></argument>, <argument><expr><name>width</name></expr></argument>, <argument><expr><name>height</name></expr></argument>)</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name><name>guideTab</name><operator>.</operator><name>updateList</name></name><argument_list>()</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><name>alchemyTab</name> <operator>=</operator> <operator>new</operator> <call><name>AlchemyTab</name><argument_list>()</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name>add</name><argument_list>(<argument><expr><name>alchemyTab</name></expr></argument>)</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name><name>alchemyTab</name><operator>.</operator><name>setRect</name></name><argument_list>(<argument><expr><literal type="number">0</literal></expr></argument>, <argument><expr><literal type="number">0</literal></expr></argument>, <argument><expr><name>width</name></expr></argument>, <argument><expr><name>height</name></expr></argument>)</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><name>notesTab</name> <operator>=</operator> <operator>new</operator> <call><name>NotesTab</name><argument_list>()</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name>add</name><argument_list>(<argument><expr><name>notesTab</name></expr></argument>)</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name><name>notesTab</name><operator>.</operator><name>setRect</name></name><argument_list>(<argument><expr><literal type="number">0</literal></expr></argument>, <argument><expr><literal type="number">0</literal></expr></argument>, <argument><expr><name>width</name></expr></argument>, <argument><expr><name>height</name></expr></argument>)</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name><name>notesTab</name><operator>.</operator><name>updateList</name></name><argument_list>()</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><name>catalogTab</name> <operator>=</operator> <operator>new</operator> <call><name>CatalogTab</name><argument_list>()</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name>add</name><argument_list>(<argument><expr><name>catalogTab</name></expr></argument>)</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name><name>catalogTab</name><operator>.</operator><name>setRect</name></name><argument_list>(<argument><expr><literal type="number">0</literal></expr></argument>, <argument><expr><literal type="number">0</literal></expr></argument>, <argument><expr><name>width</name></expr></argument>, <argument><expr><name>height</name></expr></argument>)</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><call><name><name>catalogTab</name><operator>.</operator><name>updateList</name></name><argument_list>()</argument_list></call></expr>;</expr_stmt><decl_stmt><decl><type><name><name>Tab</name><index>[]</index></name></type> <name>tabs</name> <init>= <expr><block>{<expr><operator>new</operator> <class><super><name>IconTab</name></super><argument_list>( <argument><expr><operator>new</operator> <call><name>ItemSprite</name><argument_list>(<argument><expr><name><name>ItemSpriteSheet</name><operator>.</operator><name>GUIDE_PAGE</name></name></expr></argument>, <argument><expr><literal type="null">null</literal></expr></argument>)</argument_list></call></expr></argument> )</argument_list> <block>{<function><specifier>protected</specifier> <type><name>void</name></type> <name>select</name><parameter_list>( <parameter><decl><type><name>boolean</name></type> <name>value</name></decl></parameter> )</parameter_list> <block>{<expr_stmt><expr><call><name><name>super</name><operator>.</operator><name>select</name></name><argument_list>( <argument><expr><name>value</name></expr></argument> )</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><name><name>guideTab</name><operator>.</operator><name>active</name></name> <operator>=</operator> <name><name>guideTab</name><operator>.</operator><name>visible</name></name> <operator>=</operator> <name>value</name></expr>;</expr_stmt><if>if <condition>(<expr><name>value</name></expr>)</condition><then> <block type="pseudo"><expr_stmt><expr><name>last_index</name> <operator>=</operator> <literal type="number">0</literal></expr>;</expr_stmt></block></then></if><expr_stmt/>}</block>}<operator>,</operator>new IconTab<parameter_list>( <parameter><decl/></parameter></parameter_list>new ItemSprite<parameter_list>(<parameter><decl><type><name><name>ItemSpriteSheet</name><operator>.</operator><name>ALCH_PAGE</name></name></type></decl></parameter>, <parameter/></parameter_list>null</function>)</block></class></expr> ) <block>{<expr><specifier>protected</specifier> <name>void</name> <call><name>select</name><argument_list>( <argument><expr><name>boolean</name> <name>value</name></expr></argument> )</argument_list></call> <block>{<expr><call><name><name>super</name><operator>.</operator><name>select</name></name><argument_list>( <argument><expr><name>value</name></expr></argument> )</argument_list></call></expr>;<expr><name><name>alchemyTab</name><operator>.</operator><name>active</name></name> <operator>=</operator> <name><name>alchemyTab</name><operator>.</operator><name>visible</name></name> <operator>=</operator> <name>value</name></expr>;<if>if <condition>(<expr><name>value</name></expr>)</condition><then> <block type="pseudo"><expr_stmt><expr><name>last_index</name> <operator>=</operator> <literal type="number">1</literal></expr>;</expr_stmt></block></then></if>}</block>}</expr>,<expr><operator>new</operator> <class><super><name>IconTab</name></super><argument_list>( <argument><expr><call><name><name>Icons</name><operator>.</operator><name>get</name></name><argument_list>(<argument><expr><name><name>Icons</name><operator>.</operator><name>DEPTH</name></name></expr></argument>)</argument_list></call></expr></argument> )</argument_list> <block>{<function><specifier>protected</specifier> <type><name>void</name></type> <name>select</name><parameter_list>( <parameter><decl><type><name>boolean</name></type> <name>value</name></decl></parameter> )</parameter_list> <block>{<expr_stmt><expr><call><name><name>super</name><operator>.</operator><name>select</name></name><argument_list>( <argument><expr><name>value</name></expr></argument> )</argument_list></call></expr>;</expr_stmt><expr_stmt><expr><name><name>notesTab</name><operator>.</operator><name>active</name></name> <operator>=</operator> <name><name>notesTab</name><operator>.</operator><name>visible</name></name> <operator>=</operator> <name>value</name></expr>;</expr_stmt><if>if <condition>(<expr><name>value</name></expr>)</condition><then> <block type="pseudo"><expr_stmt><expr><name>last_index</name> <operator>=</operator> <literal type="number">2</literal></expr>;</expr_stmt></block></then></if><expr_stmt/>}</block>}<operator>,</operator>new IconTab<parameter_list>( <parameter><decl/></parameter></parameter_list>new ItemSprite<parameter_list>(<parameter><decl><type><name><name>ItemSpriteSheet</name><operator>.</operator><name>WEAPON_HOLDER</name></name></type></decl></parameter>, <parameter/></parameter_list>null</function>)</block></class></expr> ) <block>{<expr><specifier>protected</specifier> <name>void</name> <call><name>select</name><argument_list>( <argument><expr><name>boolean</name> <name>value</name></expr></argument> )</argument_list></call> <block>{<expr><call><name><name>super</name><operator>.</operator><name>select</name></name><argument_list>( <argument><expr><name>value</name></expr></argument> )</argument_list></call></expr>;<expr><name><name>catalogTab</name><operator>.</operator><name>active</name></name> <operator>=</operator> <name><name>catalogTab</name><operator>.</operator><name>visible</name></name> <operator>=</operator> <name>value</name></expr>;<if>if <condition>(<expr><name>value</name></expr>)</condition><then> <block type="pseudo"><expr_stmt><expr><name>last_index</name> <operator>=</operator> <literal type="number">3</literal></expr>;</expr_stmt></block></then></if>}</block>}}</expr>;<for>for <control>(<init><decl><type><name>Tab</name></type> <name>tab</name> <range>: <expr><name>tabs</name></expr></range></decl></init>)</control> <block>{<expr_stmt><expr><call><name>add</name><argument_list>( <argument><expr><name>tab</name></expr></argument> )</argument_list></call></expr>;</expr_stmt><expr_stmt/>}</block><expr_stmt><expr><call><name>layoutTabs</name><argument_list>()</argument_list></call></expr>;</expr_stmt></for><call><name>select</name><argument_list>(<argument><expr><name>last_index</name></expr></argument>)</argument_list></call></block></block></block></expr></init></decl>;</decl_stmt>}</block></constructor>
}</block></class>
</unit>

区别在于：像new ItemSprite这种，srcML有些地方会当成一个token！

记录一个srcML和javalang等工具做tokenization不一致的地方相关推荐

1. 记录一个SpringBoot集成邮件及工具类博客

   https://www.jianshu.com/p/5e5fd5b07f54

2. 用html5做一个简单网页_用Python做一个简单的翻译工具

   编程本身是跟年龄无关的一件事,不论你现在是十四五岁,还是四五十岁,如果你热爱它,并且愿意持续投入其中,必定会有所收获. 本文就来自编程教室一位"小"读者的投稿(互助学习1群里的同学 ...

3. 最近想学习一下编译原理，做一个编程规范的检测工具

   因为公司对编程规范要求的很严格,然而很多人却都没有规范的习惯,所以很多问题还是存在于代码中.有一次机会想做一个编程规范检测的工具,可惜自己能力有限,很多东西都不会.偶然想到了编译原理,可惜我没学过.拿 ...

4. 微服务应用大行其道，我提供一个dto和entity转换工具类，方便大家做转换，少写机械代码，多陪陪家人

   微服务应用大行其道,我提供一个dto和entity转换工具类,方便大家做转换,少写机械代码,多陪陪家人. 该工具类主要是对dozer进行了封装,使用过程代码量极少,废话少说,贴代码了 import j ...

5. 微服务应用大行其道，我提供一个dto和entity转换工具类，方便大家做转换，少写机械代码，多陪陪家人...

   微服务应用大行其道,我提供一个dto和entity转换工具类,方便大家做转换,少写机械代码,多陪陪家人. 该工具类主要是对dozer进行了封装,使用过程代码量极少,废话少说,贴代码了 import j ...

6. （六）一个交易者的资金管理系统：做交易记录并分析盈利和亏损

   作者:chen_h 微信号 & QQ:862251340 微信公众号:coderpai (一)一个交易者的资金管理系统(上) (二)一个交易者的资金管理系统(下) (三)一个交易者的资金管理系 ...

7. 记录一个菜鸡的大一立项：做一个跑步打卡APP

   记录一个菜鸡的大一立项:做一个跑步打卡APP(///▽///) 1. 思路: 2. 具体功能: [ 记录跑步时间.距离] [ 跑步计划制定] [ 跑步路程追踪] [ 跑步信息记录 ] [ 校园景点介绍 ...

8. 开发一个短信推送工具需要怎么做

   开发一个短信推送工具需要以下几步: 选择合适的编程语言,比如说 Python.Java 等. 选择短信 API 提供商,比如 Twilio.Alibaba Cloud 等. 注册一个账号,获取 API ...

9. 卸载后以前拍的视频会删除吗_可立拍！苹果自己的视频编辑App是一个被忽视的好工具...

   手机预装应用总是不如三方产品? 看到这个问题,你是不是会下意识反驳:iPhone自带 app 就很好用啊!的确如此,iPhone 的<Pages><备忘录>,这些 app 的优 ...

最新文章

1. VisualStudio 合并代码文件
2. Java之数组array和集合list、set、map
3. LoadRunner 测试Tuxedo的问题
4. 禁用software reporter tool.exe 解决CPU高占用率的问题
5. Unity 分离贴图 alpha 通道实践
6. MySQL查询时构建自增ID
7. 310. Minimum Height Trees
8. 银行it现状调研_中央银行系统行业现状调研分析及发展趋势预测报告(2019年版)...
9. 学点 C 语言(3): 转义字符
10. ecshop根目录调用_ecshop调用指定商品分类下的商品
11. 用java和tomcat安装jenkins过程
12. 由深圳的大树所想到的
13. mysql查询top5_MySQL常用函数TOP5
14. Python笔记（21）正则表达式
15. 违背基本假设的几种情况——自相关性（R语言）
16. vrep中接近开关与激光雷达
17. POI Cannot add merged region XXX to sheet because it overlaps with an existing merged region问题解决
18. Legion：基于Haskell开发的极简区块链服务器
19. HTML5 水平线标签 hr
20. 语音合成（TTS)论文优选：Learning to Speak Fluently in a Foreign Language: Multilingual Speech Synthesis and Cr

热门文章

1. Android Octa源码编译和下载过程详细记录
2. 【华为机试真题详解】高矮个子排队
3. python图片转文字easyocr_OCR识别软件将图片转文字的具体方法
4. 2018面试总结（阿里巴巴蚂蚁金服、饿了么、人人车等）
5. RGB565转换为RGB88，并作线性补偿
6. JVM内存与垃圾回收篇——JVM与Java体系结构
7. 中国古代神器 兵器详解
8. java log日志函数_Java 中的 AWS Lambda 函数日志记录 - AWS Lambda
9. 一年时间，拿到了人生中的第一个10万
10. 俞敏洪老师的回复真的太糟糕了！