federated icon indicating copy to clipboard operation
federated copied to clipboard
verified publisher icon tensorflow

Add macOS build configuration

Open evgri243 opened this issue 2 years ago • 4 comments

With the recent advancements in tensorflow-metal and large GPU shared memory variant, macOS is becoming a more attractive platform for federated experimentation. This PR adds the required configuration to build tensorflow-federated for macOS running on Apple Silicon.

  • Add bazel configuration to properly build macOS version
  • Adjust codebase to make it compatible with the latest Apple's clang and ld
  • Slightly improve usability of build_python_package.sh (side effect of testing iterations, may be removed)
  • Add a hack to "better" identify package's platform

The build succeeds on my current MacBook Pro M2, the PR is code-complete and seems to run "Hello, World" successfully. Going through the more thorough testing right now.

Verification:

  • [x] MacOS (M2 arm64) with XCode 15.0.3
% bazel test //tensorflow_federated/... --test_verbose_timeout_warnings
details 
INFO: Elapsed time: 227.125s, Critical Path: 96.72s
INFO: 593 processes: 359 internal, 234 local.
FAILED: Build did NOT complete successfully
//tensorflow_federated/cc/core/impl/executor_stacks:remote_stacks_test NO STATUS
//tensorflow_federated/cc/core/impl/executors:data_executor_test      NO STATUS
//tensorflow_federated/cc/core/impl/executors:dataset_from_tensor_structures_test NO STATUS
//tensorflow_federated/cc/core/impl/executors:dtensor_executor_test   NO STATUS
//tensorflow_federated/cc/core/impl/executors:eager_computation_test_cpu NO STATUS
//tensorflow_federated/cc/core/impl/executors:executor_service_test   NO STATUS
//tensorflow_federated/cc/core/impl/executors:federating_executor_test NO STATUS
//tensorflow_federated/cc/core/impl/executors:session_provider_test   NO STATUS
//tensorflow_federated/cc/core/impl/executors:struct_traversal_order_test NO STATUS
//tensorflow_federated/cc/core/impl/executors:tensorflow_executor_parameterized_test_cpu NO STATUS
//tensorflow_federated/cc/core/impl/executors:threading_test          NO STATUS
//tensorflow_federated/cc/core/impl/executors:xla_executor_gpu_test   NO STATUS
//tensorflow_federated/cc/core/impl/executors:xla_executor_test_gpu   NO STATUS
//tensorflow_federated/examples/learning/federated_program/vizier:vizier_service_test NO STATUS
//tensorflow_federated/examples/program:program_logic_test            NO STATUS
//tensorflow_federated/examples/simple_fedavg:simple_fedavg_test_gpu  NO STATUS
//tensorflow_federated/examples/stateful_clients:stateful_fedavg_test NO STATUS
//tensorflow_federated/proto/v0:computation_pb2_test                  NO STATUS
//tensorflow_federated/python/aggregators:aggregator_test_utils_test  NO STATUS
//tensorflow_federated/python/aggregators:concat_test                 NO STATUS
//tensorflow_federated/python/aggregators:deterministic_discretization_test NO STATUS
//tensorflow_federated/python/aggregators:discretization_test         NO STATUS
//tensorflow_federated/python/aggregators:distributed_dp_test         NO STATUS
//tensorflow_federated/python/aggregators:hadamard_test               NO STATUS
//tensorflow_federated/python/aggregators:modular_clipping_test       NO STATUS
//tensorflow_federated/python/aggregators:quantile_estimation_test    NO STATUS
//tensorflow_federated/python/aggregators:sampling_test               NO STATUS
//tensorflow_federated/python/aggregators:sparsifying_aggregator_factory_test NO STATUS
//tensorflow_federated/python/aggregators:stochastic_discretization_test NO STATUS
//tensorflow_federated/python/analytics:data_processing_test          NO STATUS
//tensorflow_federated/python/analytics:differential_privacy_test     NO STATUS
//tensorflow_federated/python/analytics:histogram_processing_test     NO STATUS
//tensorflow_federated/python/analytics/heavy_hitters/iblt:chunkers_test NO STATUS
//tensorflow_federated/python/analytics/heavy_hitters/iblt:hyperedge_hashers_test NO STATUS
//tensorflow_federated/python/analytics/heavy_hitters/iblt:iblt_clipping_test NO STATUS
//tensorflow_federated/python/analytics/heavy_hitters/iblt:iblt_factory_test NO STATUS
//tensorflow_federated/python/analytics/heavy_hitters/iblt:iblt_lib_test NO STATUS
//tensorflow_federated/python/analytics/heavy_hitters/iblt:iblt_tff_test NO STATUS
//tensorflow_federated/python/analytics/heavy_hitters/iblt:subsample_process_test NO STATUS
//tensorflow_federated/python/analytics/hierarchical_histogram:hierarchical_histogram_factory_test NO STATUS
//tensorflow_federated/python/common_libs:golden_test                 NO STATUS
//tensorflow_federated/python/common_libs:retrying_test               NO STATUS
//tensorflow_federated/python/common_libs:structure_test              NO STATUS
//tensorflow_federated/python/core/backends/mapreduce:form_utils_test NO STATUS
//tensorflow_federated/python/core/backends/native:compiler_test      NO STATUS
//tensorflow_federated/python/core/backends/native:cpp_execution_contexts_test NO STATUS
//tensorflow_federated/python/core/backends/native:execution_contexts_test NO STATUS
//tensorflow_federated/python/core/backends/test:execution_contexts_test NO STATUS
//tensorflow_federated/python/core/backends/xla:cpp_execution_contexts_test NO STATUS
//tensorflow_federated/python/core/backends/xla:runtime_test          NO STATUS
//tensorflow_federated/python/core/impl/compiler:compiled_computation_transformations_test NO STATUS
//tensorflow_federated/python/core/impl/compiler:intrinsic_defs_test  NO STATUS
//tensorflow_federated/python/core/impl/compiler:tensorflow_computation_factory_test_cpu NO STATUS
//tensorflow_federated/python/core/impl/compiler:transformations_test NO STATUS
//tensorflow_federated/python/core/impl/computation:computation_serialization_test NO STATUS
//tensorflow_federated/python/core/impl/computation:function_utils_test NO STATUS
//tensorflow_federated/python/core/impl/context_stack:context_stack_test_utils_test NO STATUS
//tensorflow_federated/python/core/impl/context_stack:set_default_context_test NO STATUS
//tensorflow_federated/python/core/impl/execution_contexts:mergeable_comp_execution_context_test NO STATUS
//tensorflow_federated/python/core/impl/executor_stacks:cpp_executor_factory_test NO STATUS
//tensorflow_federated/python/core/impl/executors:cardinalities_utils_test NO STATUS
//tensorflow_federated/python/core/impl/executors:cpp_to_python_executor_test NO STATUS
//tensorflow_federated/python/core/impl/executors:dtensor_executor_binding_test NO STATUS
//tensorflow_federated/python/core/impl/executors:remote_executor_test NO STATUS
//tensorflow_federated/python/core/impl/executors:value_serialization_test NO STATUS
//tensorflow_federated/python/core/impl/federated_context:data_test   NO STATUS
//tensorflow_federated/python/core/impl/federated_context:value_utils_test NO STATUS
//tensorflow_federated/python/core/impl/jax_context:jax_computation_test NO STATUS
//tensorflow_federated/python/core/impl/jax_context:jax_serialization_test NO STATUS
//tensorflow_federated/python/core/impl/tensorflow_context:tensorflow_computation_context_test NO STATUS
//tensorflow_federated/python/core/impl/tensorflow_context:tensorflow_computation_test NO STATUS
//tensorflow_federated/python/core/impl/types:array_shape_test        NO STATUS
//tensorflow_federated/python/core/impl/types:computation_types_test  NO STATUS
//tensorflow_federated/python/core/impl/types:placements_test         NO STATUS
//tensorflow_federated/python/core/impl/types:type_factory_test       NO STATUS
//tensorflow_federated/python/core/impl/types:type_serialization_test NO STATUS
//tensorflow_federated/python/core/impl/types:type_transformations_test NO STATUS
//tensorflow_federated/python/core/impl/utils:tensorflow_utils_test   NO STATUS
//tensorflow_federated/python/core/impl/xla_context:xla_serialization_test NO STATUS
//tensorflow_federated/python/core/templates:aggregation_process_test NO STATUS
//tensorflow_federated/python/core/templates:estimation_process_test  NO STATUS
//tensorflow_federated/python/learning:dataset_reduce_test_cpu        NO STATUS
//tensorflow_federated/python/learning:federated_evaluation_test_gpu  NO STATUS
//tensorflow_federated/python/learning/algorithms:fed_avg_with_optimizer_schedule_test_gpu NO STATUS
//tensorflow_federated/python/learning/algorithms:fed_eval_test_gpu   NO STATUS
//tensorflow_federated/python/learning/algorithms:personalization_eval_test_gpu NO STATUS
//tensorflow_federated/python/learning/framework:optimizer_utils_test_cpu NO STATUS
//tensorflow_federated/python/learning/framework:optimizer_utils_test_gpu NO STATUS
//tensorflow_federated/python/learning/metrics:counters_test          NO STATUS
//tensorflow_federated/python/learning/metrics:keras_finalizer_test   NO STATUS
//tensorflow_federated/python/learning/models:functional_test         NO STATUS
//tensorflow_federated/python/learning/models:model_weights_test      NO STATUS
//tensorflow_federated/python/learning/models:reconstruction_model_test NO STATUS
//tensorflow_federated/python/learning/optimizers:scheduling_test     NO STATUS
//tensorflow_federated/python/learning/optimizers:yogi_test           NO STATUS
//tensorflow_federated/python/learning/programs:training_program_logic_test NO STATUS
//tensorflow_federated/python/learning/programs:vizier_program_logic_test NO STATUS
//tensorflow_federated/python/learning/templates:apply_optimizer_finalizer_test NO STATUS
//tensorflow_federated/python/learning/templates:distributors_test    NO STATUS
//tensorflow_federated/python/learning/templates:finalizers_test      NO STATUS
//tensorflow_federated/python/learning/templates:learning_process_test NO STATUS
//tensorflow_federated/python/learning/templates:model_delta_client_work_test NO STATUS
//tensorflow_federated/python/learning/templates:proximal_client_work_test NO STATUS
//tensorflow_federated/python/program:logging_release_manager_test    NO STATUS
//tensorflow_federated/python/program:prefetching_data_source_test    NO STATUS
//tensorflow_federated/python/program:tensorboard_release_manager_test NO STATUS
//tensorflow_federated/python/program:value_reference_test            NO STATUS
//tensorflow_federated/python/simulation:iterative_process_compositions_test NO STATUS
//tensorflow_federated/python/simulation:training_loop_test           NO STATUS
//tensorflow_federated/python/simulation/baselines:baseline_task_test NO STATUS
//tensorflow_federated/python/simulation/baselines:task_data_test     NO STATUS
//tensorflow_federated/python/simulation/baselines/emnist:autoencoder_tasks_test NO STATUS
//tensorflow_federated/python/simulation/baselines/emnist:char_recognition_tasks_test NO STATUS
//tensorflow_federated/python/simulation/baselines/emnist:emnist_preprocessing_test NO STATUS
//tensorflow_federated/python/simulation/baselines/landmark:landmark_tasks_test NO STATUS
//tensorflow_federated/python/simulation/baselines/stackoverflow:tag_prediction_preprocessing_test NO STATUS
//tensorflow_federated/python/simulation/baselines/stackoverflow:word_prediction_preprocessing_test NO STATUS
//tensorflow_federated/python/simulation/datasets:cifar100_test       NO STATUS
//tensorflow_federated/python/simulation/datasets:file_per_user_client_data_test NO STATUS
//tensorflow_federated/python/simulation/datasets:from_tensor_slices_client_data_test NO STATUS
//tensorflow_federated/python/simulation/datasets:gldv2_test          NO STATUS
//tensorflow_federated/python/simulation/datasets:vision_datasets_utils_test NO STATUS
//tensorflow_federated/python/simulation/models:mobilenet_v2_test     NO STATUS
//tensorflow_federated/python/tensorflow_libs:function_test           NO STATUS
//tensorflow_federated/python/tensorflow_libs:graph_spec_test         NO STATUS
//tensorflow_federated/python/tensorflow_libs:tensor_utils_test       NO STATUS
//tensorflow_federated/python/tests:backend_accelerators_test_cpu     NO STATUS
//tensorflow_federated/python/tests:backend_accelerators_test_gpu     NO STATUS
//tensorflow_federated/python/tests:composite_tensor_e2e_test         NO STATUS
//tensorflow_federated/python/tests:simulation_test                   NO STATUS
//tensorflow_federated/python/tests:sync_local_cpp_execution_context_test NO STATUS
//tensorflow_federated/cc/core/impl/executors:sequence_executor_test FAILED TO BUILD
//tensorflow_federated/python/core/backends/test:cpp_execution_contexts_test FAILED in 6.9s
    ERROR   .tensorflow_federated/python/core/backends/test/cpp_execution_contexts_test (0.0s)
//tensorflow_federated/python/core/impl/executor_stacks:executor_stack_bindings_test FAILED in 7.6s
    ERROR   .tensorflow_federated/python/core/impl/executor_stacks/executor_stack_bindings_test (0.0s)
//tensorflow_federated/python/core/impl/executors:executor_bindings_test FAILED in 7.6s
    ERROR   .tensorflow_federated/python/core/impl/executors/executor_bindings_test (0.0s)
//tensorflow_federated/python/learning/optimizers:adam_test              FAILED in 15.1s
    FAILED  __main__.AdamTest.test_math (0.8s)
//tensorflow_federated/python/program:file_program_state_manager_test    FAILED in 10.4s
    ERROR   __main__.FileProgramStateManagerLoadTest.test_returns_saved_program_state_materializable_value_reference_sequence (0.1s)
//tensorflow_federated/python/program:file_release_manager_test          FAILED in 10.2s
    ERROR   __main__.SavedModelFileReleaseManagerGetValueTest.test_returns_saved_value_materializable_value_reference_sequence (0.1s)
//tensorflow_federated/examples/simple_fedavg:simple_fedavg_test_cpu     FAILED in 3 out of 3 in 22.3s
  Stats over 3 runs: max = 22.3s, min = 21.5s, avg = 21.9s, dev = 0.3s
    ERROR   __main__.RNNTest.test_client_adagrad_train_dtensor_server_side_use_dataset_iteration (0.0s)
    ERROR   __main__.SimpleFedAvgTest.test_process_construction_dtensor_on_only_client (0.0s)
    ERROR   __main__.SimpleFedAvgTest.test_training_custom_model_converges_dtensor_on_only_client (0.0s)
    ERROR   __main__.SimpleFedAvgTest.test_training_custom_model_converges_tensorflow_use_sequence_reduce (2.9s)
    ERROR   __main__.SimpleFedAvgTest.test_training_keras_model_converges_dtensor_server_side_use_dataset_iteration (0.0s)
    ERROR   __main__.SimpleFedAvgTest.test_process_construction_dtensor_on_both_server_client (0.0s)
    ERROR   __main__.SimpleFedAvgTest.test_training_custom_model_converges_dtensor_on_both_server_client (0.0s)
    ERROR   __main__.SimpleFedAvgTest.test_training_keras_model_converges_dtensor_on_only_client (0.0s)
    ERROR   __main__.SimpleFedAvgTest.test_training_keras_model_converges_tensorflow_use_sequence_reduce (2.6s)
    ERROR   __main__.SimpleFedAvgTest.test_process_construction_dtensor_server_side_use_dataset_iteration (0.0s)
    ERROR   __main__.SimpleFedAvgTest.test_training_custom_model_converges_dtensor_server_side_use_dataset_iteration (0.0s)
    ERROR   __main__.SimpleFedAvgTest.test_training_keras_model_converges_dtensor_on_both_server_client (0.0s)
//tensorflow_federated/python/learning/algorithms:mime_test_cpu       NO STATUS
//tensorflow_federated/python/learning/algorithms:fed_sgd_test_cpu       FAILED in 1 out of 10 in 17.3s
  Stats over 10 runs: max = 17.3s, min = 10.4s, avg = 13.1s, dev = 2.1s
//tensorflow_federated/python/learning/algorithms:fed_sgd_test_gpu       FAILED in 1 out of 10 in 14.6s
  Stats over 10 runs: max = 14.6s, min = 8.2s, avg = 11.0s, dev = 1.6s

Test cases: finished with 4630 passing and 18 failing out of 4650 test cases (some targets did not have test case information)

Executed 114 out of 292 tests: 150 tests pass, 1 fails to build, 9 fail locally and 132 were skipped.

Partially implements #3881. Publication of a prebuilt package is still blocked due to tensorflow-text and tensorflow-compression missing macOS builds (although staying buildable from sources).

evgri243 avatar Nov 08 '23 15:11 evgri243

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

google-cla[bot] avatar Nov 08 '23 15:11 google-cla[bot]

cc: @michaelreneer

evgri243 avatar Jan 23 '24 23:01 evgri243

We dropped support for MacOS because it was costly to maintain. I think before we accept a change like this we would need to discuss restoring official support for MacOS. Can we retain this PR (so we don't lose this work) and start some discussions for what that would look like?

Is there anything I can help with?

evgri243 avatar Feb 12 '24 18:02 evgri243

Was Mac OS Support ever added back?

hassan-kamran avatar Apr 03 '24 03:04 hassan-kamran