[Lexer] Tweaks for string literal lexing #19356

rintaro · 2018-09-17T23:50:23Z

Add formStringLiteralToken() dedicated for forming string literal
Fix duplicated diagnostics for unterminated string literal in interpolation
Improve diagnostics for single quoted string literal
Improve fix-it to remove too-many-"#" in string literal
Bunch of code tweaks to improve readability.

rintaro · 2018-09-17T23:50:41Z

@swift-ci Please smoke test

rintaro · 2018-09-18T00:00:05Z

@swift-ci Please Test Source Compatibility

rintaro · 2018-09-18T01:33:04Z

@swift-ci Please test compiler performance

swift-ci · 2018-09-18T08:03:34Z

Build comment file:

Summary for master full

Unexpected test results, excluded stats for RxSwift, ReSwift, Kommander, Kingfisher, SwiftDate, SwiftyJSON, CoreStore, KeychainAccess, Result, ReactiveSwift, Alamofire, AsyncNinja, Wordy, SwifterSwift, ObjectMapper

No regressions above thresholds

Debug-batch
- debug-batch brief
- debug-batch detailed
Release
- release brief
- release detailed

Debug-batch

debug-batch brief

Regressed (0)

name	old	new	delta	delta_pct

Improved (0)

name	old	new	delta	delta_pct

Unchanged (delta < 1.0% or delta < 100.0ms) (3)

name	old	new	delta	delta_pct
Frontend.NumInstructionsExecuted	10,944,402,708,996	10,964,696,894,575	20,294,185,579	0.19%
LLVM.NumLLVMBytesOutput	361,644,626	361,620,148	-24,478	-0.01%
time.swift-driver.wall	1075.9s	1080.1s	4.2s	0.39%

debug-batch detailed

Regressed (0)

name	old	new	delta	delta_pct

Improved (2)

name	old	new	delta	delta_pct
Driver.NumDriverPipePolls	93,334	91,340	-1,994	-2.14% ✅
Driver.NumDriverPipeReads	98,595	96,644	-1,951	-1.98% ✅

Unchanged (delta < 1.0% or delta < 100.0ms) (92)

name	old	new	delta	delta_pct
AST.NumASTBytesAllocated	12,003,726,250	12,005,365,936	1,639,686	0.01%
AST.NumDecls	24,878	24,878	0	0.0%
AST.NumDependencies	81,893	81,893	0	0.0%
AST.NumImportedExternalDefinitions	672,222	672,222	0	0.0%
AST.NumInfixOperators	10,724	10,724	0	0.0%
AST.NumLinkLibraries	0	0	0	0.0%
AST.NumLoadedModules	92,875	92,875	0	0.0%
AST.NumLocalTypeDecls	51	51	0	0.0%
AST.NumObjCMethods	9,013	9,013	0	0.0%
AST.NumPostfixOperators	14	14	0	0.0%
AST.NumPrecedenceGroups	5,178	5,178	0	0.0%
AST.NumPrefixOperators	19	19	0	0.0%
AST.NumReferencedDynamicNames	68	68	0	0.0%
AST.NumReferencedMemberNames	1,584,463	1,584,463	0	0.0%
AST.NumReferencedTopLevelNames	88,717	88,717	0	0.0%
AST.NumSourceBuffers	108,953	108,954	1	0.0%
AST.NumSourceLines	625,439	625,439	0	0.0%
AST.NumSourceLinesPerSecond	338,907	336,687	-2,220	-0.66%
AST.NumTotalClangImportedEntities	2,205,277	2,205,277	0	0.0%
AST.NumUsedConformances	82,352	82,352	0	0.0%
Driver.ChildrenMaxRSS	31,266,680,832	31,358,547,968	91,867,136	0.29%
Driver.DriverDepCascadingDynamic	0	0	0	0.0%
Driver.DriverDepCascadingExternal	0	0	0	0.0%
Driver.DriverDepCascadingMember	0	0	0	0.0%
Driver.DriverDepCascadingNominal	0	0	0	0.0%
Driver.DriverDepCascadingTopLevel	0	0	0	0.0%
Driver.DriverDepDynamic	0	0	0	0.0%
Driver.DriverDepExternal	0	0	0	0.0%
Driver.DriverDepMember	0	0	0	0.0%
Driver.DriverDepNominal	0	0	0	0.0%
Driver.DriverDepTopLevel	0	0	0	0.0%
Driver.NumDriverJobsRun	5,305	5,305	0	0.0%
Driver.NumDriverJobsSkipped	0	0	0	0.0%
Driver.NumProcessFailures	0	0	0	0.0%
Frontend.MaxMallocUsage	185,126,485,432	185,236,310,248	109,824,816	0.06%
Frontend.NumInstructionsExecuted	10,944,402,708,996	10,964,696,894,575	20,294,185,579	0.19%
Frontend.NumProcessFailures	0	0	0	0.0%
IRModule.NumIRAliases	44,842	44,690	-152	-0.34%
IRModule.NumIRBasicBlocks	1,288,560	1,288,560	0	0.0%
IRModule.NumIRComdatSymbols	0	0	0	0.0%
IRModule.NumIRFunctions	726,849	726,849	0	0.0%
IRModule.NumIRGlobals	824,758	824,681	-77	-0.01%
IRModule.NumIRIFuncs	0	0	0	0.0%
IRModule.NumIRInsts	12,907,841	12,907,841	0	0.0%
IRModule.NumIRNamedMetaData	26,178	26,178	0	0.0%
IRModule.NumIRValueSymbols	1,410,011	1,409,782	-229	-0.02%
LLVM.NumLLVMBytesOutput	361,644,626	361,620,148	-24,478	-0.01%
Parse.NumFunctionsParsed	687,989	687,989	0	0.0%
Parse.NumIterableDeclContextParsed	274,779	274,779	0	0.0%
SILModule.NumSILGenDefaultWitnessTables	0	0	0	0.0%
SILModule.NumSILGenFunctions	661,376	661,376	0	0.0%
SILModule.NumSILGenGlobalVariables	9,309	9,309	0	0.0%
SILModule.NumSILGenVtables	2,469	2,469	0	0.0%
SILModule.NumSILGenWitnessTables	15,303	15,303	0	0.0%
SILModule.NumSILOptDefaultWitnessTables	0	0	0	0.0%
SILModule.NumSILOptFunctions	496,784	496,784	0	0.0%
SILModule.NumSILOptGlobalVariables	9,557	9,557	0	0.0%
SILModule.NumSILOptVtables	4,734	4,734	0	0.0%
SILModule.NumSILOptWitnessTables	29,228	29,228	0	0.0%
Sema.AccessLevelRequest	660,741	660,741	0	0.0%
Sema.DefaultAndMaxAccessLevelRequest	15,885	15,885	0	0.0%
Sema.EnumRawTypeRequest	5,666	5,666	0	0.0%
Sema.ExtendedNominalRequest	1,237,534	1,237,534	0	0.0%
Sema.InheritedDeclsReferencedRequest	42,176,257	42,176,257	0	0.0%
Sema.InheritedTypeRequest	266,579	266,579	0	0.0%
Sema.IsDynamicRequest	667,049	667,049	0	0.0%
Sema.IsObjCRequest	541,688	541,688	0	0.0%
Sema.NamedLazyMemberLoadFailureCount	11,048	11,048	0	0.0%
Sema.NamedLazyMemberLoadSuccessCount	2,124,655	2,124,655	0	0.0%
Sema.NominalTypeLookupDirectCount	12,320,734	12,320,733	-1	-0.0%
Sema.NumConformancesDeserialized	1,644,835	1,644,835	0	0.0%
Sema.NumConstraintScopes	5,198,213	5,198,213	0	0.0%
Sema.NumConstraintsConsideredForEdgeContraction	11,770,328	11,770,328	0	0.0%
Sema.NumDeclsDeserialized	13,439,902	13,439,902	0	0.0%
Sema.NumDeclsValidated	619,711	619,711	0	0.0%
Sema.NumFunctionsTypechecked	541,198	541,198	0	0.0%
Sema.NumGenericSignatureBuilders	618,028	618,028	0	0.0%
Sema.NumLazyGenericEnvironments	2,592,992	2,592,992	0	0.0%
Sema.NumLazyGenericEnvironmentsLoaded	244,472	244,472	0	0.0%
Sema.NumLazyIterableDeclContexts	2,490,779	2,490,779	0	0.0%
Sema.NumTypesDeserialized	5,937,520	5,937,520	0	0.0%
Sema.NumTypesValidated	312,974	312,974	0	0.0%
Sema.NumUnloadedLazyIterableDeclContexts	1,747,701	1,747,701	0	0.0%
Sema.OverriddenDeclsRequest	584,846	584,846	0	0.0%
Sema.RequirementRequest	13,418	13,418	0	0.0%
Sema.SelfBoundsFromWhereClauseRequest	28,789	28,789	0	0.0%
Sema.SetterAccessLevelRequest	48,280	48,280	0	0.0%
Sema.SuperclassDeclRequest	34,298,993	34,298,993	0	0.0%
Sema.SuperclassTypeRequest	8,404	8,404	0	0.0%
Sema.TypeDeclsFromWhereClauseRequest	7,658	7,658	0	0.0%
Sema.USRGenerationRequest	102,658	102,658	0	0.0%
Sema.UnderlyingTypeDeclsReferencedRequest	731,768	731,768	0	0.0%

Release

release brief

Regressed (0)

name	old	new	delta	delta_pct

Improved (0)

name	old	new	delta	delta_pct

Unchanged (delta < 1.0% or delta < 100.0ms) (3)

name	old	new	delta	delta_pct
Frontend.NumInstructionsExecuted	9,140,435,712,439	9,132,932,483,190	-7,503,229,249	-0.08%
LLVM.NumLLVMBytesOutput	299,176,754	299,156,344	-20,410	-0.01%
time.swift-driver.wall	1836.2s	1838.6s	2.4s	0.13%

release detailed

Regressed (0)

name	old	new	delta	delta_pct

Improved (0)

name	old	new	delta	delta_pct

Unchanged (delta < 1.0% or delta < 100.0ms) (23)

name	old	new	delta	delta_pct
AST.NumImportedExternalDefinitions	117,305	117,305	0	0.0%
AST.NumLoadedModules	6,071	6,071	0	0.0%
AST.NumTotalClangImportedEntities	376,246	376,246	0	0.0%
AST.NumUsedConformances	84,585	84,585	0	0.0%
IRModule.NumIRBasicBlocks	1,238,437	1,238,437	0	0.0%
IRModule.NumIRFunctions	551,848	551,848	0	0.0%
IRModule.NumIRGlobals	607,942	607,889	-53	-0.01%
IRModule.NumIRInsts	9,692,230	9,692,230	0	0.0%
IRModule.NumIRValueSymbols	1,093,383	1,093,174	-209	-0.02%
LLVM.NumLLVMBytesOutput	299,176,754	299,156,344	-20,410	-0.01%
SILModule.NumSILGenFunctions	236,277	236,277	0	0.0%
SILModule.NumSILOptFunctions	348,335	348,335	0	0.0%
Sema.NumConformancesDeserialized	853,961	853,961	0	0.0%
Sema.NumConstraintScopes	4,998,760	4,998,760	0	0.0%
Sema.NumDeclsDeserialized	2,561,455	2,561,455	0	0.0%
Sema.NumDeclsValidated	292,096	292,096	0	0.0%
Sema.NumFunctionsTypechecked	203,411	203,411	0	0.0%
Sema.NumGenericSignatureBuilders	90,957	90,957	0	0.0%
Sema.NumLazyGenericEnvironments	449,334	449,334	0	0.0%
Sema.NumLazyGenericEnvironmentsLoaded	51,568	51,568	0	0.0%
Sema.NumLazyIterableDeclContexts	302,846	302,846	0	0.0%
Sema.NumTypesDeserialized	1,470,197	1,470,197	0	0.0%
Sema.NumTypesValidated	109,808	109,808	0	0.0%

rintaro · 2018-09-18T09:30:39Z

include/swift/Parse/Token.h

+  /// \brief Set characteristics of string literal token.
+  void setStringLiteral(bool IsMultilineString, unsigned CustomDelimiterLen) {
+    assert(Kind == tok::string_literal &&
+           "only identifiers can be escaped identifiers");


assertion comment

rintaro · 2018-09-19T02:53:01Z

@swift-ci Please smoke test

rintaro · 2018-09-19T02:53:09Z

@swift-ci Please ASAN test

rintaro · 2018-09-19T05:02:16Z

ASAN: https://ci.swift.org/view/Pull%20Request/job/swift-PR-osx-ASAN-test/59/

rintaro · 2018-09-19T08:32:10Z

Small cosmetic changes
@swift-ci Please smoke test

rintaro · 2018-09-19T09:55:41Z

@swift-ci Please smoke test

We should not emit diagnostics in skipToEndOfInterpolatedExpression()

NFC

Removed Diags parameter from it. Skipped bytes are revisited by main lexer function anyway. So emitting diagnostics in it causes duplicated errors.

CurPtr[-1] must be '#' when called.

This simplifies main lexStringLiteral loop

EOF and newline (in non-multiline string literal) must be handled by call site. lexCharacter doesn't need to handle them. Added assertion instead.

NFC. Reorder code to improve readability.

Ignore the contents of interpolation.

For readability.

Fix-it to remove extra '#'s at once.

rintaro · 2018-09-19T09:59:44Z

@swift-ci Please smoke test

johnno1962 · 2018-09-21T09:29:45Z

👍👍, I like the new tidied up skipToEndOfInterpolatedExpression()

johnno1962 · 2018-09-21T10:12:22Z

I’m seeing an assertion firing on this code:

print("""
  \("--)
  "”")

Assertion failed: (*End == ')' && "invalid string literal interpolations should" " not be returned as string literals"), function getStringLiteralSegments, file /Volumes/Elements/swift-source/swift/lib/Parse/Lexer.cpp, line 2255.

This is a step forward as it used to compile with no error and an incorrect string. It's difficult to think of a solution.

rintaro · 2018-09-21T13:18:57Z

Confirmed. Thank you! I'll look into it.

rintaro changed the title ~~[Lexer] Tweaks for string literal flexing~~ [Lexer] Tweaks for string literal lexing Sep 17, 2018

rintaro commented Sep 18, 2018

View reviewed changes

rintaro force-pushed the lexer-stringliteral branch from 514fa02 to 713edce Compare September 19, 2018 02:52

rintaro force-pushed the lexer-stringliteral branch 2 times, most recently from 038f47b to 7b744ce Compare September 19, 2018 08:33

rintaro added 12 commits September 19, 2018 18:58

[Lexer] Add formStringLiteralToken dedicated for forming string literal

143f55a

[Lexer] Fix double diagnostics for unterminated string literal

0e9b232

We should not emit diagnostics in skipToEndOfInterpolatedExpression()

[Lexer] Simplify handling quotes in skipToEndOfInterpolatedExpression()

c9c6963

NFC

[Lexer] Don't emit diagnostics in skipToEndOfInterpolatedExpression()

4536e69

Removed Diags parameter from it. Skipped bytes are revisited by main lexer function anyway. So emitting diagnostics in it causes duplicated errors.

[Lexer] Add assertion in advanceIfCustomDelimiter()

893524e

CurPtr[-1] must be '#' when called.

[Lexer] Advance pointer to the end of end-quote in lexCharacter

ba51727

This simplifies main lexStringLiteral loop

[Lexer] Remove unnecessary logic from lexCharacter

1a4d375

EOF and newline (in non-multiline string literal) must be handled by call site. lexCharacter doesn't need to handle them. Added assertion instead.

[Lexer] Code tweaks in lexStringLiteral()

e6e55c2

NFC. Reorder code to improve readability.

[Lexer] Improve diagnostics for single-quote string literal

7b701d5

Ignore the contents of interpolation.

[Lexer] Factor out diagnostics for single-quoted string literal

aaa4110

For readability.

[Lexer] Improve fix-it to remove "too long" delimiter in string literal

a5759b7

Fix-it to remove extra '#'s at once.

[Lexer] Add a couple of test cases for extended escaping str literal

50497ff

rintaro force-pushed the lexer-stringliteral branch from 7b744ce to 50497ff Compare September 19, 2018 09:59

rintaro merged commit f472888 into swiftlang:master Sep 19, 2018

rintaro deleted the lexer-stringliteral branch September 19, 2018 11:39

rintaro mentioned this pull request Sep 21, 2018

[Lexer] Fix assertion failure for unterminated string literal #19457

Merged

johnno1962 mentioned this pull request Sep 24, 2018

Implementation for SE-200 (raw string escaping) #19191

Closed

[Lexer] Tweaks for string literal lexing #19356

[Lexer] Tweaks for string literal lexing #19356

Uh oh!

Conversation

rintaro commented Sep 17, 2018

Uh oh!

rintaro commented Sep 17, 2018

Uh oh!

rintaro commented Sep 18, 2018

Uh oh!

rintaro commented Sep 18, 2018

Uh oh!

swift-ci commented Sep 18, 2018

Build comment file:

Summary for master full

Debug-batch

debug-batch brief

debug-batch detailed

Release

release brief

release detailed

Uh oh!

rintaro Sep 18, 2018

Choose a reason for hiding this comment

Uh oh!

rintaro commented Sep 19, 2018

Uh oh!

rintaro commented Sep 19, 2018

Uh oh!

rintaro commented Sep 19, 2018

Uh oh!

rintaro commented Sep 19, 2018

Uh oh!

rintaro commented Sep 19, 2018

Uh oh!

rintaro commented Sep 19, 2018

Uh oh!

johnno1962 commented Sep 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

johnno1962 commented Sep 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rintaro commented Sep 21, 2018

Uh oh!

Uh oh!

johnno1962 commented Sep 21, 2018 •

edited

Loading

johnno1962 commented Sep 21, 2018 •

edited

Loading