Argument retries for plugin `retryFailedStep` is not working

Hello!

What are you trying to achieve?

I'm trying to use the retryFailedStep plugin with Puppeteer helper; to be more precise I want to increase the number of retries globally for flaky steps.

What do you get instead?

The retryFailedStep plugin is still using the default number of retries ( 3 ) even though I've added in the codecept.conf.jsfile the argument retries: 9, and thus this is unfortunately not working currently.

Provide console output if related. Use --verbose mode for more details.

 I switch to {"name":"app-iframe"}
      › [Url] https://vars.hotjar.com/box-acca23410e696f2ca3087d947271c3d0.html
    Delaying for 100ms
    I click "All Collections"
    [2] Retrying... Attempt #2
    [2] Retrying... Attempt #3
    [2] Retrying... Attempt #4
    [2] Error | Error: Clickable element "All Collections" was not found by text|CSS|XPath
    [2] Error | Error: Clickable element "All Collections" was not found by text|CSS|XPath
    [2] Error | Error: Clickable element "All Collections" was not found by text|CSS|XPath
    [2] Error | Error: Clickable element "All Collections" was not found by text|CSS|XPath
    [2] Error | Error: Clickable element "All Collections" was not found by text|CSS|XPath
    [2] Error | Error: Clickable element "All Collections" was not found by text|CSS|XPath
    [2] Error | Error: Clickable element "All Collections" was not found by text|CSS|XPath
    [2] Error | Error: Clickable element "All Collections" was not found by text|CSS|XPath
    [2] <teardown> Stopping recording promises
 › <screenshotOnFail> Test failed, try to save a screenshot

Provide test source code if related

	plugins: {
		autoDelay: {
			enabled: true,
		},
		retryFailedStep: {
			enabled: true,
			retries: 9,
		},
		tryTo: {
			enabled: true,
		},
		screenshotOnFail: {
			enabled: true,
		},

Details

• CodeceptJS version: ^3.3.0
• NodeJS Version: v16.14.0
• Operating System: MacOS Monterey v12.12.1
• puppeteer Version: ^13.4.1
• Configuration file:

const { setHeadlessWhen } = require( "@codeceptjs/configure" );
require( "dotenv" ).config();

// Turn on headless mode when running with HEADLESS=true environment variable
// Export HEADLESS=true && npx codeceptjs run
setHeadlessWhen( process.env.HEADLESS );

exports.config = {
	tests: "!(node_modules)/*_test.js",
	output: "./output",
	helpers: {
		Puppeteer: {
			url: "https://myurl",
			show: true,
			waitForNavigation: [ "load", "domcontentloaded" ],
			windowSize: "1280x1024",
			keepCookies: true,
			restart: false,
			keepBrowserState: true,
			smartWait: 12000,
			timeouts: {
				script: 60000,
				pageload: 100000,
			},
			chrome: {
				args: [
					"--no-sandbox",
					"--disable-setuid-sandbox",
					"--disable-web-security",
					"--disable-features=IsolateOrigins,site-per-process",
				],
			},
		},
		ElementHelper: {
			require: "./helpers/element_helper.js",
		},
	},
	include: {
		I: "./steps_file.js",
	},
	bootstrap: null,
	mocha: {},
	name: "integration",
	plugins: {
		autoDelay: {
			enabled: true,
		},
		retryFailedStep: {
			enabled: true,
			retries: 9,
		},
		tryTo: {
			enabled: true,
		},
		screenshotOnFail: {
			enabled: true,
		},
		allure: {
			enabled: true,
		},
	},
};

[hanna-becker]

Some problem here. It's been broken for ages. I can't even upgrade past version 3.2.0 because of that, because 3.2.1 reduces the number of default retries to 3 along with the timeouts between them, and it just breaks all of my existing tests.

[hanna-becker]

I looked into the code base, wrote a unit test for more than default retries, but it passes. So the issue is probably with the puppeteer integration. Unfortunately I don't have enough context of the implementation details, otherwise would give fixing it a shot. Unless anyone from CodeceptJS core team would be able to provide me some hints how it's currently implemented? Not sure this is a good time to ask, though.

[mirao]

Related to #2829.
I have the same issue with Playwright in CodeceptJS 3.3.3.
In my config when I set more retries than the default value 5, still 5 is used.
It seems that a workaround is patching of

CodeceptJS/lib/helper/Playwright.js

Line 396 in 03bf08f

retries: 5,

to value you want, e.g. to 10

[lucky7104]

I have also the same problem. If i comment the recorder.retry part in puppeteer.js then it runs with my retryfailedsteps config values.

I don't know what is the right way for the problem.

My ideas are:
a new puppeteer parameter if puppeteer should to set the his default retry. Then i think a standard retry for some situations is right. OR
Puppeteer looks for plugin retryfailedsteps and if is it active. If true then not retry from puppeteer.

same for playwright...

retry at playwright is 5 and puppeteer 3