flutter
[integration_test] Plugin not registered unless running via XCTest
So as the title says, the integration_test native plugin is not being correctly registered unless running tests via XCTest, like when running via flutter drive.
This unfortunately breaks use cases like automatic taking screenshots via https://github.com/mmcc007/screenshots for instance.
I've tracked the issue down to a fix in https://github.com/flutter/plugins/pull/2465.
Changing IntegrationTestPlugin.m:registerWithRegistrar to the code below fixes the issue and seems to make things work, but I am assuming might bring back the XCTest init issue the PR tried to fix originally.
+ (void)registerWithRegistrar:(NSObject<FlutterPluginRegistrar> *)registrar {
[[IntegrationTestPlugin instance] setupChannels:registrar.messenger];
}
@gaaclarke, can you take a look / any idea on what should be done here?
Steps to Reproduce
-
flutter drive --target=integration_test/main_test.dart --driver=test_driver/integration_test.dart
Expected results:
Actual results:
Code sample
void main() {
final binding = IntegrationTestWidgetsFlutterBinding.ensureInitialized()
as IntegrationTestWidgetsFlutterBinding;
testWidgets("main test", (WidgetTester tester) async {
await binding.takeScreenshot(name);
}
}
Logs
flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
flutter: The following MissingPluginException was thrown running a test:
flutter: MissingPluginException(No implementation found for method captureScreenshot on channel
flutter: plugins.flutter.io/integration_test)
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #0 MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:165:7)
flutter: <asynchronous suspension>
flutter: #1 IOCallbackManager.takeScreenshot (package:integration_test/_callback_io.dart:108:9)
flutter: <asynchronous suspension>
flutter: #2 IntegrationTestWidgetsFlutterBinding.takeScreenshot (package:integration_test/integration_test.dart:180:9)
flutter: <asynchronous suspension>
flutter: #3 screenshot (file:///Users/dev/mobile/integration_test/main_test.dart:139:3)
flutter: <asynchronous suspension>
flutter: #4 main.<anonymous closure> (file:///Users/dev/mobile/integration_test/main_test.dart:26:5)
flutter: <asynchronous suspension>
flutter: #5 testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:172:15)
flutter: <asynchronous suspension>
flutter: #6 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:803:5)
flutter: <asynchronous suspension>
flutter:
flutter: The test description was:
flutter: main test
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
[✓] Flutter (Channel master, 2.6.0-12.0.pre.240, on macOS 11.4 20F71 darwin-x64, locale en-PT)
• Flutter version 2.6.0-12.0.pre.240 at /Users/joao/dev/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 46a52d03bd (5 days ago), 2021-10-07 08:52:09 -0700
• Engine revision e914da14f1
• Dart version 2.15.0 (build 2.15.0-178.0.dev)
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
• Android SDK at /Users/joao/Library/Android/sdk
• Platform android-30, build-tools 30.0.3
• Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 13.0)
• Xcode at /Applications/Xcode.app/Contents/Developer
• CocoaPods version 1.10.1
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 4.2)
• Android Studio at /Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
[✓] IntelliJ IDEA Community Edition (version 2020.2.4)
• IntelliJ at /Applications/IntelliJ IDEA CE.app
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
[✓] VS Code (version 1.61.0)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.27.0
[✓] Connected device (3 available)
• iPhone 11 Pro Max (mobile) • 1714BBB8-050B-4C2D-B118-456B827FFC2C • ios • com.apple.CoreSimulator.SimRuntime.iOS-15-0 (simulator)
• macOS (desktop) • macos • darwin-x64 • macOS 11.4 20F71 darwin-x64
• Chrome (web) • chrome • web-javascript • Google Chrome 94.0.4606.71
• No issues found!
Hi @tritao, Thanks for filing the issue. I can reproduce the issue on the latest stable and the master channel.
logs
Master
mahesh@Maheshs-MacBook-Air-M1 performance % flutter test integration_test
Running "flutter pub get" in performance... 2,213ms
00:10 +0: loading /Users/mahesh/Desktop/nevercode/ISSUES/performance/integration_test/main_test.dart Ru00:11 +0: loading /Users/mahesh/Desktop/nevercode/ISSUES/performance/integration_test/main_test.dart 974ms
00:29 +0: loading /Users/mahesh/Desktop/nevercode/ISSUES/performance/integration_test/main_test.dart
00:33 +0: loading /Users/mahesh/Desktop/nevercode/ISSUES/performance/integration_test/main_test.dart 4.7s
Xcode build done. 22.0s
00:37 +0: main test
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following MissingPluginException was thrown running a test:
MissingPluginException(No implementation found for method captureScreenshot on channel
plugins.flutter.io/integration_test)
When the exception was thrown, this was the stack:
#0 MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:165:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
The test description was:
main test
════════════════════════════════════════════════════════════════════════════════════════════════════
00:37 +0 -1: main test [E]
Test failed. See exception logs above.
The test description was: main test
00:37 +0 -1: Some tests failed.
Stable 2.5.2
mahesh@Maheshs-MacBook-Air-M1 performance % flutter test integration_test
00:07 +0: loading /Users/mahesh/Desktop/nevercode/ISSUES/performance/integration_test/main_test.dart R00:08 +0: loading /Users/mahesh/Desktop/nevercode/ISSUES/performance/integration_test/main_test.dart 1,018ms
00:27 +0: loading /Users/mahesh/Desktop/nevercode/ISSUES/performance/integration_test/main_test.dart
00:34 +0: loading /Users/mahesh/Desktop/nevercode/ISSUES/performance/integration_test/main_test.dart 6.9s
Xcode build done. 26.4s
00:38 +0: main test
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following StateError was thrown running a test:
Bad state: Call convertFlutterSurfaceToImage() before taking a screenshot
When the exception was thrown, this was the stack:
#0 IOCallbackManager.takeScreenshot (package:integration_test/_callback_io.dart:88:7)
#1 IntegrationTestWidgetsFlutterBinding.takeScreenshot (package:integration_test/integration_test.dart:175:61)
#2 main.<anonymous closure> (file:///Users/mahesh/Desktop/nevercode/ISSUES/performance/integration_test/main_test.dart:16:19)
#3 main.<anonymous closure> (file:///Users/mahesh/Desktop/nevercode/ISSUES/performance/integration_test/main_test.dart:15:28)
#4 testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:176:29)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
The test description was:
main test
════════════════════════════════════════════════════════════════════════════════════════════════════
00:38 +0 -1: main test [E]
Test failed. See exception logs above.
The test description was: main test
I am also having this same issue. Here is my code
Flutter Doctor: Doctor summary (to see all details, run flutter doctor -v): [✓] Flutter (Channel master, 2.6.0-12.0.pre.390, on macOS 11.6 20G165 darwin-x64, locale en-US) [✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2) [✓] Xcode - develop for iOS and macOS (Xcode 13.0) [✓] Chrome - develop for the web [✓] Android Studio (version 4.1) [✓] VS Code (version 1.61.1) [✓] Connected device (2 available)
Hi there,
I got the same issue when run flutter drive --dart-define=testing_mode=true --driver=test_driver/integration_test.dart --target=integration_test/app_test.dart to take a screenshot on latest master channel.
await binding.convertFlutterSurfaceToImage();
await tester.pumpAndSettle();
await binding.takeScreenshot('screenshot1');
Error logs
Failure in method: Test app flow
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
The following MissingPluginException was thrown running a test:
MissingPluginException(No implementation found for method
captureScreenshot on channel plugins.flutter.io/integration_test)
When the exception was thrown, this was the stack:
#0 MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:175:7)
<asynchronous suspension>
#1 IOCallbackManager.takeScreenshot (package:integration_test/_callback_io.dart:96:33)
<asynchronous suspension>
#2 IntegrationTestWidgetsFlutterBinding.takeScreenshot (package:integration_test/integration_test.dart:175:39)
<asynchronous suspension>
#3 main.<anonymous closure> (file:///Users/admin/flutter/myapp/integration_test/app_test.dart:28:5)
<asynchronous suspension>
#4 testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:170:15)
<asynchronous suspension>
#5 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:806:5)
<asynchronous suspension>
flutter doctor -v
[✓] Flutter (Channel master, 2.6.0-12.0.pre.701, on Mac OS X 10.15.7 19H1519 darwin-x64, locale en-US)
[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
[!] Xcode - develop for iOS and macOS (Xcode 12.4)
! Flutter recommends a minimum Xcode version of 13.
Download the latest version or update via the Mac App Store.
[✓] Chrome - develop for the web
[✓] Android Studio (version 2020.3)
[✓] VS Code (version 1.58.2)
[✓] VS Code (version 1.63.0-insider)
[✓] Connected device (3 available)
Is there anyway to workaround this one?
Is there anyway to workaround this one?
I posted a workaround in the initial post.
hi @maheshmnj I wonder when we can have the plan to fix this issue? Thank to @tritao, it can work with the workaround, but then I can't upgrade flutter unless stash this changed before doing upgrade, and the problem is that I can't do it on CI testing.
@ductranit, Please check the wiki to understand how the issues are prioritized. https://github.com/flutter/flutter/wiki/Issue-hygiene#when-will-my-bug-be-fixed
Can confirm the issue as well as the fix, thanks! Unfortunately fixing this locally doesn't help when working with remote CI tools like CodeMagic. So +1 for making this part of one of the next releases.
I can also confirm that editing flutter/packages/integration_test/ios/Classes/IntegrationTestPlugin.m and then running flutter clean fixes the problem for me as well.
I have the issue as well and it is a blocker, because I want to generate screenshots in CodeMagic as well.
/offtopic:
@tritao I thought https://github.com/mmcc007/screenshots uses a deprecated test API. Are you using it with integration_tests? I thought the package is also 2 years old? Would love if you could share some resources in regards of this.
@bwaide I'm testing CodeMagic as well, could you share some resources how to trigger tests in CodeMagic and how to upload those screenshots to the App Store backends?
@tritao I thought https://github.com/mmcc007/screenshots uses a deprecated test API. Are you using it with integration_tests? I thought the package is also 2 years old? Would love if you could share some resources in regards of this.
You can check out the version I am using at https://github.com/tritao/screenshots/commits/integration_test_latest
@tritao Thanks for linking and maintaining it. Would it be possible to publish the integration_test_latest branch on pub.dev for easier accessibility?
After doing some research I figured out that integration_tests are limited in a sense that I have to run them for each device type in sequence whereas traditional flutter tests run on the host and thus are capable of spawning several AVDs/Simulators to do all the screenshots for all device types in one go.
Since your fork is using integration_tests I wonder how you would implement generating screenshots for all devices within any CI/CD workflow, like on codemagic. May I ask if you fully automate screenshot generation with it, or so you manually run your tests?
Thank you @tritao , could you please provide a sample of how the pubspec.yaml would look in order to use the version you posted. Currenly mine look like this.
dev_dependencies:
integration_test:
sdk: flutter
So is the current approach to integration testing in flutter, that if you want to capture a screenshot, that you need to change/edit the source to Flutter SDK so that the Flutter supplied integration_test_driver_extended drivers work and actually do what the code was supposed to do - so is the "extended" driver currently outside Flutter's own testing program? How ironic that the testing tools aren't properly being tested... LOL... sorry for the light hearted comments here, but I just wanted to make sure that I've not missed another path to testing integration with Flutter.
I've just stopped using integration_test and continued using the older test driver that still works fine. I use Emulators package to control the simulators and perform screenshots for testing. It works and seems to be fairly issues free.
@maheshmnj @darshankawar Sorry I just don't understand, the fix (or at least workaround) is simple. Why did it take too much time for this bug?
I see it is marked as p4 for a long time but there is no working progress for it. This issue stops developers to take screenshot in testing, and I don't think it's minor.
Do I need to create a pull request with the workaround myselft? As I'm just not sure this is the correct way.
+ (void)registerWithRegistrar:(NSObject<FlutterPluginRegistrar> *)registrar {
[[IntegrationTestPlugin instance] setupChannels:registrar.messenger];
}
@workerbee22 it's in flutter/packages/integration_test/ios/Classes/IntegrationTestPlugin.m registerWithRegistrar method
Taking screenshots for testing is very common. Hopefully, this fix will be merged soon!
This is still present in flutter 3. Anyone trying to follow the instructions at https://github.com/flutter/flutter/blob/main/packages/integration_test/README.md on iOS hits this issue.
Despite testing being a Flutter team goal for Y22 there seems to be little effort being directed here... so I've found the traditional test driver to work fine. YMMV
Hey everyone, here's a quick script using the solution above that you can add to your pipeline. You should be able to run this anywhere:
# Set a variable with Flutter's path
FLUTTER_PATH="$(which flutter)"
FLUTTER_PATH=${FLUTTER_PATH/\/bin\/flutter/""}
# Update IntegrationTestPlugin.m
sed -i '' -e "51s/^//p; 51s/^.*/ \[\[IntegrationTestPlugin instance\] setupChannels\:registrar\.messenger\]\;/" $FLUTTER_PATH'/packages/integration_test/ios/Classes/IntegrationTestPlugin.m'
To rollback, simply go to your Flutter's folder and run git reset --hard and then flutter doctor to redownload dependencies.
@jmagman I was able to reproduce this in the following versions: 3.0.1, 3.0.5, and 3.3.2.
Hey everyone, here's a quick script using the solution above that you can add to your pipeline. You should be able to run this anywhere:
To rollback, simply go to your Flutter's folder and run
git reset --hardand thenflutter doctorto redownload dependencies.
A slight improvement to make it work with the symlink brew uses, and not to add it again if its already been run...
# Set a variable with Flutter's path
FLUTTER_PATH="$(readlink -f `which flutter`)"
FLUTTER_PATH=${FLUTTER_PATH/\/bin\/flutter/""}
FILE_TO_UPDATE=$FLUTTER_PATH'/packages/integration_test/ios/Classes/IntegrationTestPlugin.m'
if ! grep -q "setupChannels:registrar.messenger" "$FILE_TO_UPDATE"; then
# Update IntegrationTestPlugin.m
sed -i '' -e "51s/^//p; 51s/^.*/ \[\[IntegrationTestPlugin instance\] setupChannels\:registrar\.messenger\]\;/" $FILE_TO_UPDATE
fi
I agree with others that this is a problem, and also agree that it's unfortunate that I cannot run an integration test from VS Code and have it work as expected on iOS.
There is a workaround that doesn't involve patching the SDK and allows integration with CI/CD for iOS again, but it requires some changes to your app code and minor changes to your testing code.
- Create wrapper...
import 'dart:ui' as ui;
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';
class Screenshotable extends StatefulWidget {
Screenshotable({
required this.child,
super.key,
});
final Widget child;
@override
ScreenshotableState createState() => ScreenshotableState();
}
class ScreenshotableState extends State<Screenshotable> {
final GlobalKey _renderKey = GlobalKey();
/// Captures the image from the current widget. This will return `null` if
/// the image cannot be captured for any reason.
Future<Uint8List?> captureImage() async {
Uint8List? image;
try {
var boundary = _renderKey.currentContext!.findRenderObject()
as RenderRepaintBoundary?;
if (boundary != null &&
(!kDebugMode || boundary.debugNeedsPaint != true)) {
var mq = MediaQuery.of(context);
var img = await boundary.toImage(pixelRatio: mq.devicePixelRatio);
var byteData = await img.toByteData(
format: ui.ImageByteFormat.png,
);
image = byteData?.buffer.asUint8List();
}
} catch (e) {
// no-op; ignore it because screenshots may not always be supported
}
return image?.isEmpty == true ? null : image;
}
@override
Widget build(BuildContext context) => RepaintBoundary(
key: _renderKey,
child: widget.child,
);
}
- Update your launch config:
runApp(
(!kIsWeb && Platform.isIOS)
? MaterialApp( // This provides the MediaQuery
home: Screenshotable(
child: MyApp(),
),
)
: MyApp(),
);
- In your test, rather than calling the
takeScreenshot()directly against theWidgetsTester, use this instead...
@override
Future<Uint8List> takeScreenshot() async {
Uint8List? image;
try {
if (!kIsWeb && Platform.isIOS) {
try {
var ss = find.byType(Screenshotable).evaluate().first as StatefulElement;
await tester.pump();
image = await (ss.state as ScreenshotableState).captureImage();
} catch (e) {
_logger.info('Screenshot failed', e);
}
} else {
try {
if (!kIsWeb && Platform.isAndroid) {
await binding.convertFlutterSurfaceToImage();
}
} catch (e) {
// no-op
}
await tester.pump();
image = Uint8List.fromList(await binding.takeScreenshot('screenshot'));
}
} catch (e) {
_logger.info('Screenshot failed', e);
}
return image;
}
