Add Lucene1001 / Lucene1002 analyzers for TokenStream contract, #1146 (#1288)

* Add Lucene1001 / Lucene1002 analyzers for TokenStream contract (#1146)

Adds two new public Roslyn analyzers (C# + VB) that surface the
TokenStream subclassing rules at compile time as warnings instead of
runtime exceptions:

- Lucene1001: an override of End()/Reset()/Close() on a TokenStream
  subclass must call the corresponding base method.
- Lucene1002: a non-abstract Tokenizer subclass must override End()
  (passes if any class in the hierarchy between the subclass and
  Tokenizer already provides the override).

Both analyzers ship with code fix providers. The primary audience is
consumers of the library who write their own TokenStream-derived
types — Lucene.NET's in-tree TokenStream subclasses intentionally
match the upstream Java behavior, which itself does not always call
super.X() in End/Reset/Close. Each in-tree violation is therefore
suppressed with [SuppressMessage] and a "LUCENENET specific" comment
that explains why it matches the Java parent. Two End() overrides
(PrefixAwareTokenFilter, PrefixAndSuffixAwareTokenFilter) carry an
additional "LUCENENET TODO" because the upstream Java omission of
super.end() may be a latent bug worth revisiting.

The test verifier helper now references System.Runtime so analyzer
fixes that introduce base calls returning void don't trigger a
spurious CS0012 in the post-fix compilation check.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Address Copilot review feedback on Lucene1001/Lucene1002 analyzers

- Lucene1001 CS code fix: emit ThrowStatement (not ExpressionStatement) when
  converting an expression-bodied `=> throw ...` member, so the produced code
  is valid C#.
- Lucene1001 CS + VB analyzers: exclude lambdas, anonymous methods, and (CS)
  local functions from the base-call scan. A base call inside an uninvoked
  delegate body should not satisfy the contract.
- Lucene1002 CS + VB code fixes: use ElasticCarriageReturnLineFeed instead of
  hardcoded line endings so the formatter normalizes to the document's
  convention.
- DiagnosticVerifier: harden System.Runtime.dll lookup with a
  TRUSTED_PLATFORM_ASSEMBLIES fallback and a clear error if neither resolves.
- Tests: add VerifyCSharpFix/VerifyBasicFix coverage for Lucene1002, a
  VerifyCSharpFix for the Lucene1001 expression-bodied case plus a new
  throw-expression regression, lambda/local-function exclusion tests for
  Lucene1001 (CS + VB), and the missing VB indirect-Tokenizer-inheritance
  test for Lucene1002.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: paulirwin

src/Lucene.Net.Analysis.Common/Analysis/Miscellaneous/PrefixAndSuffixAwareTokenFilter.cs

@@ -1,4 +1,6 @@
 // Lucene version compatibility level 4.8.1
+using System.Diagnostics.CodeAnalysis;
+
 namespace Lucene.Net.Analysis.Miscellaneous
 {
     /*
@@ -85,16 +87,29 @@ public sealed override bool IncrementToken()
             return suffix.IncrementToken();
         }
 
+        // LUCENENET specific: matches the upstream Java implementation, which does not call super.reset().
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; TokenStream.Reset() is a no-op so the omission is equivalent.")]
         public override void Reset()
         {
             suffix.Reset();
         }
 
+        // LUCENENET specific: matches the upstream Java implementation, which does not call super.close().
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; TokenStream.Close() is a no-op so the omission is equivalent.")]
         public override void Close()
         {
             suffix.Close();
         }
 
+        // LUCENENET TODO: the upstream Java does not call super.end() here, so neither do we — but
+        // TokenStream.End() does ClearAttributes() and resets PositionIncrementAttribute, which the
+        // documented contract says End() overrides should do. This may be a latent bug in upstream
+        // Lucene that the .NET port should investigate; see Lucene.NET issue tracker if reproducing
+        // a problem with end-of-stream attribute state on this filter.
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; see LUCENENET TODO above for why this may need to be revisited.")]
         public override void End()
         {
             suffix.End();

src/Lucene.Net.Analysis.Common/Analysis/Miscellaneous/PrefixAwareTokenFilter.cs

@@ -1,6 +1,7 @@
 // Lucene version compatibility level 4.8.1
 using Lucene.Net.Analysis.TokenAttributes;
 using Lucene.Net.Util;
+using System.Diagnostics.CodeAnalysis;
 
 namespace Lucene.Net.Analysis.Miscellaneous
 {
@@ -169,12 +170,22 @@ public virtual Token UpdateSuffixToken(Token suffixToken, Token lastPrefixToken)
             return suffixToken;
         }
 
+        // LUCENENET TODO: the upstream Java does not call super.end() here, so neither do we — but
+        // TokenStream.End() does ClearAttributes() and resets PositionIncrementAttribute, which the
+        // documented contract says End() overrides should do. This may be a latent bug in upstream
+        // Lucene that the .NET port should investigate; see Lucene.NET issue tracker if reproducing
+        // a problem with end-of-stream attribute state on this filter.
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; see LUCENENET TODO above for why this may need to be revisited.")]
         public override void End()
         {
             prefix.End();
             suffix.End();
         }
 
+        // LUCENENET specific: matches the upstream Java implementation, which does not call super.close().
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; TokenStream.Close() is a no-op so the omission is equivalent.")]
         public override void Close()
         {
             prefix.Close();

src/Lucene.Net.Analysis.Common/Analysis/Miscellaneous/SingleTokenTokenStream.cs

@@ -2,6 +2,7 @@
 using Lucene.Net.Analysis.TokenAttributes;
 using Lucene.Net.Diagnostics;
 using System.Diagnostics;
+using System.Diagnostics.CodeAnalysis;
 
 namespace Lucene.Net.Analysis.Miscellaneous
 {
@@ -58,6 +59,10 @@ public override sealed bool IncrementToken()
             }
         }
 
+        // LUCENENET specific: matches the upstream Java implementation, which does not call super.reset().
+        // TokenStream.Reset() has an empty body so this is equivalent.
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; TokenStream.Reset() is a no-op so the omission is equivalent.")]
         public override void Reset()
         {
             exhausted = false;

src/Lucene.Net.Analysis.Common/Analysis/Sinks/TeeSinkTokenFilter.cs

@@ -2,6 +2,7 @@
 using Lucene.Net.Util;
 using System;
 using System.Collections.Generic;
+using System.Diagnostics.CodeAnalysis;
 using JCG = J2N.Collections.Generic;
 
 namespace Lucene.Net.Analysis.Sinks
@@ -248,6 +249,8 @@ public override sealed bool IncrementToken()
                 return true;
             }
 
+            [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+                Justification = "Sink streams replay cached state; calling base.End() would clear the attributes that RestoreState then sets. Matches Lucene Java behavior.")]
             public override sealed void End()
             {
                 if (finalState != null)
@@ -256,6 +259,8 @@ public override sealed void End()
                 }
             }
 
+            [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+                Justification = "Sink streams replay cached state; the iterator reset is sufficient and base.Reset() (no-op on TokenStream) provides nothing further. Matches Lucene Java behavior.")]
             public override sealed void Reset()
             {
                 it = cachedStates.GetEnumerator();

src/Lucene.Net.Analysis.Common/Analysis/Synonym/SlowSynonymFilter.cs

@@ -3,6 +3,7 @@
 using Lucene.Net.Util;
 using System;
 using System.Collections.Generic;
+using System.Diagnostics.CodeAnalysis;
 using JCG = J2N.Collections.Generic;
 
 namespace Lucene.Net.Analysis.Synonym
@@ -309,6 +310,11 @@ private void Copy(AttributeSource target, AttributeSource source)
             }
         }
 
+        // LUCENENET specific: matches the upstream Java implementation, which calls input.reset()
+        // directly rather than super.reset(). Behavior is identical (TokenFilter.Reset() just chains
+        // to m_input.Reset()), but we keep the call shape to preserve Java parity.
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; TokenFilter.Reset() chains to m_input.Reset() so this is equivalent.")]
         public override void Reset()
         {
             m_input.Reset();

src/Lucene.Net.Analysis.Phonetic/DoubleMetaphoneFilter.cs

@@ -5,6 +5,7 @@
 using Lucene.Net.Support;
 using System;
 using System.Collections.Generic;
+using System.Diagnostics.CodeAnalysis;
 
 namespace Lucene.Net.Analysis.Phonetic
 {
@@ -118,6 +119,10 @@ public override bool IncrementToken()
             }
         }
 
+        // LUCENENET specific: matches the upstream Java implementation, which calls input.reset()
+        // directly rather than super.reset(). Behavior is identical.
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; TokenFilter.Reset() chains to m_input.Reset() so this is equivalent.")]
         public override void Reset()
         {
             m_input.Reset();

src/Lucene.Net.Analysis.Phonetic/PhoneticFilter.cs

@@ -3,6 +3,7 @@
 using Lucene.Net.Analysis.TokenAttributes;
 using Lucene.Net.Analysis.TokenAttributes.Extensions;
 using System;
+using System.Diagnostics.CodeAnalysis;
 
 namespace Lucene.Net.Analysis.Phonetic
 {
@@ -104,6 +105,10 @@ public override bool IncrementToken()
             return true;
         }
 
+        // LUCENENET specific: matches the upstream Java implementation, which calls input.reset()
+        // directly rather than super.reset(). Behavior is identical.
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; TokenFilter.Reset() chains to m_input.Reset() so this is equivalent.")]
         public override void Reset()
         {
             m_input.Reset();

src/Lucene.Net.Highlighter/Highlight/TokenStreamFromTermPositionVector.cs

@@ -4,6 +4,7 @@
 using Lucene.Net.Index;
 using Lucene.Net.Util;
 using System.Collections.Generic;
+using System.Diagnostics.CodeAnalysis;
 using JCG = J2N.Collections.Generic;
 
 namespace Lucene.Net.Search.Highlight
@@ -126,6 +127,9 @@ public override bool IncrementToken()
             return false;
         }
 
+        // LUCENENET specific: matches the upstream Java implementation, which does not call super.reset().
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; TokenStream.Reset() is a no-op so the omission is equivalent.")]
         public override void Reset()
         {
             this.tokensAtCurrentPosition = this.positionedTokens.GetEnumerator();

src/Lucene.Net/Analysis/CachingTokenFilter.cs

@@ -1,4 +1,5 @@
 using System.Collections.Generic;
+using System.Diagnostics.CodeAnalysis;
 using JCG = J2N.Collections.Generic;
 
 namespace Lucene.Net.Analysis
@@ -67,6 +68,8 @@ public override bool IncrementToken()
             return true;
         }
 
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Caching filter intentionally restores the previously captured final state instead of chaining to base.End(); matches Lucene Java behavior.")]
         public override void End()
         {
             if (finalState != null)
@@ -82,6 +85,8 @@ public override void End()
         /// the first time. You should <see cref="Reset()"/> the inner tokenstream before wrapping
         /// it with <see cref="CachingTokenFilter"/>.
         /// </summary>
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Caching filter intentionally does not call base.Reset() (which would propagate to the wrapped input); matches Lucene Java behavior. See the doc comment.")]
         public override void Reset()
         {
             if (cache != null)

src/Lucene.Net/Analysis/NumericTokenStream.cs

@@ -2,6 +2,7 @@
 using Lucene.Net.Diagnostics;
 using Lucene.Net.Util;
 using System;
+using System.Diagnostics.CodeAnalysis;
 
 namespace Lucene.Net.Analysis
 {
@@ -330,6 +331,9 @@ public NumericTokenStream SetSingleValue(float value)
             return this;
         }
 
+        // LUCENENET specific: matches the upstream Java implementation, which does not call super.reset().
+        [SuppressMessage("Design", "Lucene1001:TokenStream override of End()/Reset()/Close() must call the corresponding base method.",
+            Justification = "Matches upstream Lucene Java behavior; TokenStream.Reset() is a no-op so the omission is equivalent.")]
         public override void Reset()
         {
             if (valSize == 0)