[TESTS] Ensure that `runContextRecreatingTests` runs ALL

[laz-001]

Ensure that runContextRecreatingTests executes ALL context-recreating tests.

Solution Paths:

• Create a task that identifies and executes
  • for example code, see: https://github.com/tolgee/tolgee-platform/issues/3071#issuecomment-2888831409
• identify and tag all Tests
  • use indicators to identify (Mock, Context, Spy, ...)
  • use log-files

---

Previous Issue Version, wrong assumtpion/impression

Consider renaming/dropping `runContextRecreatingTests`
=====================================================

The tests labeled as "ContextRecreating" are quite fast (70ms to 10s)

https://github.com/tolgee/tolgee-platform/runs/42060268802

It doesn't look like those are ContextRecreating.

This `runContextRecreatingTests` test runner has a dedicated job on the CI run

## Suggestion

- remove this from the CI
  - remove the tagging, thus those tests fall-back to the standard-tests
- if kept
  - it should be renamed, e.g. `runFastTests`
  - document what this test group is about

[JanCizmar]

Hey! That's weird. This tests usually contain @MockBean annotation or test specific SpringBoot properties. This makes Spring to create new context for each of such test class. Are you sure you don't see Spring Starting new context at the beginning of the test class?

[coderabbitai[bot]]

[!TIP] For best results, initiate chat on the files or code changes.

@laz-001 Yes, I do function in GitHub issues! I can help with:

• Answering questions about code and concepts
• Running scripts to analyze your repository
• Helping plan code changes
• Generating code examples
• Searching the web for relevant information

My capabilities in issues are similar to PRs, though I don't have direct access to code diffs for review. Is there something specific about your Spring test contexts that you'd like help with?

[coderabbitai[bot]]

[!TIP] For best results, initiate chat on the files or code changes.

🧩 Analysis chain

I'll search for the @ContextRecreating annotation to understand what it is and how it's being used in the codebase.

---

🏁 Script executed:

#!/bin/bash
# Find the ContextRecreating annotation definition
rg "ContextRecreating" -g "*.java" -A 5 -B 5

Length of output: 44

---

Let me try a more comprehensive search to find any references to ContextRecreating in the codebase.

---

🏁 Script executed:

#!/bin/bash
# Search for any occurrence of "ContextRecreating" in the codebase
rg "ContextRecreating" --no-ignore-vcs -A 2 -B 2

Length of output: 31537

---

🏁 Script executed:

#!/bin/bash
# Look at the definition of ContextRecreatingTest
cat backend/testing/src/main/kotlin/io/tolgee/testing/ContextRecreatingTest.kt

# Check the gradle task that runs context-recreating tests
grep -A 10 "runContextRecreatingTests" backend/app/build.gradle

Length of output: 566

---

🏁 Script executed:

#!/bin/bash
# Find all test classes with @MockBean annotations
echo "Test classes with @MockBean annotations:"
rg "@MockBean" --type kotlin --ignore-case -l

# Find all test classes with @SpringBootTest annotations that have custom properties
echo -e "\nTest classes with @SpringBootTest and custom properties:"
rg "@SpringBootTest\(\s*properties\s*=" --type kotlin -l

# Find test classes with @DirtiesContext annotation (another indicator of context recreation)
echo -e "\nTest classes with @DirtiesContext annotation:"
rg "@DirtiesContext" --type kotlin -l

Length of output: 4445

---

🏁 Script executed:

#!/bin/bash
# Find all test classes tagged with @ContextRecreatingTest
echo "Test classes tagged with @ContextRecreatingTest:"
rg "@ContextRecreatingTest" --type kotlin -l | sort > context_recreating_tests.txt
cat context_recreating_tests.txt

# Count how many we found
echo -e "\nTotal @ContextRecreatingTest classes: $(cat context_recreating_tests.txt | wc -l)"

# Find all classes with @MockBean or @DirtiesContext or custom @SpringBootTest properties
echo -e "\nCompiling list of classes with context recreation indicators..."
rg "@MockBean" --type kotlin -l | sort > mockbean_tests.txt
rg "@DirtiesContext" --type kotlin -l | sort > dirtiescontext_tests.txt
rg "@SpringBootTest\(\s*properties\s*=" --type kotlin -l | sort > custom_properties_tests.txt

# Create a combined list of all tests with context recreation indicators
cat mockbean_tests.txt dirtiescontext_tests.txt custom_properties_tests.txt | sort | uniq > all_context_recreating_indicators.txt

# A) Find classes tagged with @ContextRecreatingTest but don't have any indicators
echo -e "\nA) @ContextRecreatingTest classes WITHOUT context recreation indicators:"
comm -23 context_recreating_tests.txt all_context_recreating_indicators.txt

# B) Find classes with context recreation indicators but not tagged with @ContextRecreatingTest
echo -e "\nB) Classes WITH context recreation indicators but NOT tagged with @ContextRecreatingTest:"
comm -13 context_recreating_tests.txt all_context_recreating_indicators.txt

Length of output: 3599

---

🏁 Script executed:

#!/bin/bash
# Find all test classes tagged with @ContextRecreatingTest
echo "Test classes tagged with @ContextRecreatingTest:"
rg "@ContextRecreatingTest" --type kotlin -l | sort > context_recreating_tests.txt
cat context_recreating_tests.txt

# Count how many we found
echo -e "\nTotal @ContextRecreatingTest classes: $(cat context_recreating_tests.txt | wc -l)"

# Find all classes with @MockBean
echo -e "\nClasses with @MockBean:"
rg "@MockBean" --type kotlin -l | sort > mockbean_tests.txt
cat mockbean_tests.txt

# Find classes with @DirtiesContext
echo -e "\nClasses with @DirtiesContext:"
rg "@DirtiesContext" --type kotlin -l | sort > dirtiescontext_tests.txt
cat dirtiescontext_tests.txt

# Find classes with custom @SpringBootTest properties
echo -e "\nClasses with custom @SpringBootTest properties:"
rg "@SpringBootTest\(\s*properties\s*=" --type kotlin -l | sort > custom_properties_tests.txt
cat custom_properties_tests.txt

# Find classes with @ContextConfiguration
echo -e "\nClasses with @ContextConfiguration:"
rg "@ContextConfiguration" --type kotlin -l | sort > context_config_tests.txt
cat context_config_tests.txt

# A) For each @ContextRecreatingTest class, check if it has any of the indicators
echo -e "\nA) Checking if @ContextRecreatingTest classes have context recreation indicators:"
for test in $(cat context_recreating_tests.txt); do
  echo "Checking $test:"
  if grep -q "$test" mockbean_tests.txt || grep -q "$test" dirtiescontext_tests.txt || grep -q "$test" custom_properties_tests.txt || grep -q "$test" context_config_tests.txt; then
    echo "  Has indicators"
  else
    echo "  NO INDICATORS FOUND"
  fi
done

# B) For each class with indicators, check if it's tagged with @ContextRecreatingTest
echo -e "\nB) Classes with context recreation indicators that are NOT tagged with @ContextRecreatingTest:"
cat mockbean_tests.txt dirtiescontext_tests.txt custom_properties_tests.txt context_config_tests.txt | sort | uniq > all_indicators.txt
for test in $(cat all_indicators.txt); do
  if ! grep -q "$test" context_recreating_tests.txt; then
    echo "$test"
  fi
done

Length of output: 40394

---

@laz-001 I've analyzed the codebase to investigate your context recreation questions. Here's what I found:

A) Are all @ContextRecreatingTest classes actually context-recreating?

No. Out of 25 test classes tagged with @ContextRecreatingTest, 14 (56%) do not have common indicators of context recreation:

Checking backend/app/src/test/kotlin/io/tolgee/api/v2/controllers/V2UserControllerTest.kt:
  NO INDICATORS FOUND
Checking backend/app/src/test/kotlin/io/tolgee/api/v2/controllers/batch/BatchMoveToNamespaceTest.kt:
  NO INDICATORS FOUND
Checking backend/app/src/test/kotlin/io/tolgee/api/v2/controllers/contentDelivery/ContentDeliveryConfigControllerTest.kt:
  NO INDICATORS FOUND

etc. (and 11 other files)

B) Which test classes are context-recreating but NOT tagged as such?

I found 38 test classes that have context-recreating indicators but are NOT tagged with @ContextRecreatingTest. Here are some examples:

1. Classes with @MockBean:

backend/app/src/test/kotlin/io/tolgee/AuthProviderChangeTest.kt
backend/app/src/test/kotlin/io/tolgee/AuthTest.kt
backend/app/src/test/kotlin/io/tolgee/activity/ActivityLogTest.kt

2. Classes with @DirtiesContext:

backend/app/src/test/kotlin/io/tolgee/batch/BatchJobsCleanerTest.kt
backend/app/src/test/kotlin/io/tolgee/batch/BatchJobsGeneralWithRedisTest.kt

3. With @ContextConfiguration:

backend/app/src/test/kotlin/io/tolgee/websocket/WebsocketWithRedisTest.kt
ee/backend/tests/src/test/kotlin/io/tolgee/ee/selfHostedLimitsAndReporting/ScheduledUsageReportingRedisTest.kt

This suggests there's inconsistent tagging of tests that recreate Spring contexts, which could affect test execution efficiency if they're not properly grouped in the dedicated task found in backend/app/build.gradle.

[coderabbitai[bot]]

[!TIP] For best results, initiate chat on the files or code changes.

🧩 Analysis chain

@laz-001 Let me create a Gradle task that detects context-recreating indicators in test classes without relying on annotations. This approach will be more reliable and prevent tests from running in the wrong group.

// Add to your backend/app/build.gradle file

// Task that detects and runs tests with context recreation indicators
task runContextRecreatingTestsByDetection(type: Test) {
    // Use the same configuration as regular tests
    useJUnitPlatform()
    
    // Descriptive task name and group
    description = 'Runs tests that create a new Spring context based on detected indicators'
    group = 'verification'
    
    doFirst {
        // Find all test files with context recreation indicators
        def contextRecreatingTestFiles = findContextRecreatingTests()
        
        // Convert file paths to class names
        def testClassNames = convertToClassNames(contextRecreatingTestFiles)
        
        // Skip the task if no tests are found
        if (testClassNames.isEmpty()) {
            logger.warn("No context-recreating tests detected. Skipping task.")
            throw new StopExecutionException("No context-recreating tests detected")
        }
        
        // Set up filter to only run detected tests
        filter {
            testClassNames.each { className ->
                includeTestsMatching(className + "*")
            }
        }
        
        // Log what we're doing
        logger.lifecycle("Running ${testClassNames.size()} detected context-recreating tests")
    }
}

/**
 * Finds all test files containing indicators of context recreation
 * @return List of test file paths
 */
def findContextRecreatingTests() {
    def foundFiles = []
    def indicators = [
        '@MockBean',             // Mocked beans cause context recreation
        '@DirtiesContext',       // Explicitly marks context as dirty
        '@ContextConfiguration',  // Custom context config
        '@SpringBootTest\\(\\s*properties\\s*=' // Custom test properties
    ]
    
    // Root directory for test sources
    def testSourceDirs = sourceSets.test.kotlin.srcDirs
    
    testSourceDirs.each { srcDir ->
        if (srcDir.exists()) {
            // Get all Kotlin test files
            def testFiles = fileTree(srcDir).include('**/*Test.kt')
            
            testFiles.each { file ->
                def content = file.text
                // Check if file contains any indicator
                def hasIndicator = indicators.any { indicator ->
                    content =~ indicator
                }
                
                if (hasIndicator) {
                    foundFiles.add(file.absolutePath)
                }
            }
        }
    }
    
    return foundFiles
}

/**
 * Converts file paths to fully qualified class names
 * @param filePaths List of file paths to convert
 * @return List of fully qualified class names
 */
def convertToClassNames(List<String> filePaths) {
    def classNames = []
    def rootDir = project.projectDir.absolutePath
    
    filePaths.each { filePath ->
        // Extract relative path from project dir
        def relPath = filePath.replace(rootDir, '')
        
        // Handle src/test/kotlin directory structure
        def srcTestKotlin = '/src/test/kotlin/'
        if (relPath.contains(srcTestKotlin)) {
            // Remove file extension and convert path to package
            def className = relPath.substring(
                relPath.indexOf(srcTestKotlin) + srcTestKotlin.length(),
                relPath.lastIndexOf('.kt')
            ).replace('/', '.')
            
            classNames.add(className)
        }
    }
    
    return classNames
}

// Optional task to just print detected context-recreating tests without running them
task detectContextRecreatingTests {
    description = 'Detects tests that create a new Spring context based on indicators'
    group = 'verification'
    
    doLast {
        def files = findContextRecreatingTests()
        def classNames = convertToClassNames(files)
        
        logger.lifecycle("Detected ${classNames.size()} context-recreating tests:")
        classNames.each { className ->
            logger.lifecycle(" - $className")
        }
    }
}

This implementation:

1. Creates a task that scans test files for context recreation indicators
2. Filters the test execution to only run the detected tests
3. Provides clear subfunctions with proper documentation
4. Includes a utility task to just detect and list the files without running them

You could add this to your CI pipeline to replace the annotation-based approach, or run both for comparison during your testing optimization process.

---

🏁 Script executed:

#!/bin/bash
# Find and show current Gradle tasks related to tests
echo "Current test tasks:"
find . -name "build.gradle" -o -name "build.gradle.kts" | xargs grep -l "test" | xargs grep -l "task" | head -5 | xargs cat

# Look at the ContextRecreatingTest related task specifically
echo -e "\nContext recreating test task:"
find . -name "build.gradle" -o -name "build.gradle.kts" | xargs grep -A 15 -B 5 "runContextRecreatingTests" 2>/dev/null

Length of output: 21007

---

@laz-001 Based on your codebase structure, here's a Gradle task that detects context-recreating indicators in test files and runs only those tests:

// Add to backend/app/build.gradle

// Task that detects and runs tests with context recreation indicators
tasks.register('runDetectedContextRecreatingTests', Test) {
    // Follow same pattern as other test tasks
    outputs.upToDateWhen { false }
    useJUnitPlatform()
    maxHeapSize = "4096m"
    rootProject.setTestRetry(it)
    
    // Description for task listing
    description = 'Runs tests that create a new Spring context based on detected indicators'
    group = 'verification'
    
    doFirst {
        // Find and filter tests with context recreation indicators
        def detectedTests = findContextRecreatingTestClasses()
        
        if (detectedTests.isEmpty()) {
            logger.warn("No context-recreating tests detected. Task will be skipped.")
            throw new StopExecutionException("No context-recreating tests detected")
        }
        
        // Configure the test task to run only detected tests
        filter {
            detectedTests.each { className ->
                includeTestsMatching className
            }
        }
        
        logger.lifecycle("Running ${detectedTests.size()} detected context-recreating tests")
    }
}

/**
 * Finds test classes containing indicators of Spring context recreation
 * @return List of fully qualified test class names
 */
def findContextRecreatingTestClasses() {
    def detectedClasses = []
    def indicators = [
        '@MockBean',                              // Mocked beans force context recreation
        '@DirtiesContext',                        // Explicitly marks context as dirty
        '@ContextConfiguration',                  // Custom context configuration
        '@SpringBootTest\\(\\s*properties\\s*='   // Custom test properties
    ]
    
    // Search in Kotlin test sources
    sourceSets.test.kotlin.srcDirs.each { srcDir ->
        if (srcDir.exists()) {
            // Process each test file
            fileTree(srcDir).include('**/*Test.kt').each { file ->
                def content = file.text
                // Check if file contains any indicator
                if (indicators.any { indicator -> content.contains(indicator) }) {
                    // Convert file path to class name
                    def className = fileToClassName(file, srcDir)
                    if (className) {
                        detectedClasses.add(className)
                    }
                }
            }
        }
    }
    
    return detectedClasses
}

/**
 * Converts a file to its fully qualified class name
 * @param file The test file
 * @param srcDir The source directory
 * @return The fully qualified class name
 */
def fileToClassName(File file, File srcDir) {
    try {
        // Get relative path from source directory
        def relPath = file.absolutePath - srcDir.absolutePath
        
        // Remove leading slash and .kt extension
        relPath = relPath.replaceAll('^/', '').replaceAll('\\.kt$', '')
        
        // Convert path separators to package dots
        return relPath.replace(File.separator, '.')
    } catch (Exception e) {
        logger.error("Failed to convert file to class name: ${file}", e)
        return null
    }
}

// Task to just list detected tests without running them
tasks.register('listDetectedContextRecreatingTests') {
    description = 'Lists tests that would be considered context-recreating based on indicators'
    group = 'verification'
    
    doLast {
        def detectedTests = findContextRecreatingTestClasses()
        logger.lifecycle("Found ${detectedTests.size()} tests with context recreation indicators:")
        detectedTests.each { className ->
            logger.lifecycle(" - $className")
        }
    }
}

// Task to compare annotated vs detected context-recreating tests
tasks.register('compareContextRecreatingTestApproaches') {
    description = 'Compares annotation-based vs detection-based approaches for context-recreating tests'
    group = 'verification'
    
    doLast {
        // Get tests tagged with @ContextRecreatingTest
        def taggedTests = findTestsWithAnnotation('@ContextRecreatingTest')
        
        // Get tests with context recreation indicators
        def detectedTests = findContextRecreatingTestClasses()
        
        // Find tests that are tagged but don't have indicators
        def taggedButNotDetected = taggedTests.findAll { !detectedTests.contains(it) }
        
        // Find tests that have indicators but aren't tagged
        def detectedButNotTagged = detectedTests.findAll { !taggedTests.contains(it) }
        
        logger.lifecycle("=== Context Recreating Tests Comparison ===")
        logger.lifecycle("Tagged with @ContextRecreatingTest: ${taggedTests.size()}")
        logger.lifecycle("Detected with indicators: ${detectedTests.size()}")
        logger.lifecycle("Tagged but not detected: ${taggedButNotDetected.size()}")
        logger.lifecycle("Detected but not tagged: ${detectedButNotTagged.size()}")
        
        if (!taggedButNotDetected.isEmpty()) {
            logger.lifecycle("\nTests tagged but without indicators:")
            taggedButNotDetected.each { logger.lifecycle(" - $it") }
        }
        
        if (!detectedButNotTagged.isEmpty()) {
            logger.lifecycle("\nTests with indicators but not tagged:")
            detectedButNotTagged.each { logger.lifecycle(" - $it") }
        }
    }
}

/**
 * Finds test classes with a specific annotation
 * @param annotation The annotation to search for
 * @return List of fully qualified class names
 */
def findTestsWithAnnotation(String annotation) {
    def annotatedClasses = []
    
    sourceSets.test.kotlin.srcDirs.each { srcDir ->
        if (srcDir.exists()) {
            fileTree(srcDir).include('**/*Test.kt').each { file ->
                def content = file.text
                if (content.contains(annotation)) {
                    def className = fileToClassName(file, srcDir)
                    if (className) {
                        annotatedClasses.add(className)
                    }
                }
            }
        }
    }
    
    return annotatedClasses
}

This implementation:

1. Creates a runDetectedContextRecreatingTests task that automatically finds and runs tests with context recreation indicators
2. Provides a listDetectedContextRecreatingTests task to preview which tests would be included
3. Includes a compareContextRecreatingTestApproaches task to compare annotation-based vs detection-based approaches
4. Follows your project's existing patterns for test tasks
5. Uses descriptive subfunctions with appropriate comments
6. Handles file-to-class-name conversion correctly

You can integrate this into your CI pipeline as an alternative to (or alongside) the annotation-based approach. The comparison task will help you validate the detection logic against your current approach.

[coderabbitai[bot]]

[!TIP] For best results, initiate chat on the files or code changes.

[coderabbitai[bot]]

[!CAUTION] CodeRabbit did not find changes to make.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.