tensorflow
Core dumped when accessing HDFS after setting libhdfs.so in $LD_LIBRARY_PATH
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): centos 7.6
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
- TensorFlow installed from (source or binary): yes
- TensorFlow version (use command below): 2.6.0 2.8.0
- Python version: 3.7.10
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version:
- GPU model and memory:
You can collect some of this information using our environment capture script You can also obtain the TensorFlow version with:
- TF 1.0:
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" - TF 2.0:
python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
Core dumped when accessing HDFS if libhdfs.so is not installed in $HADOOP_HDFS_HOME/lib/native and $LD_LIBRARY_PATH is set correctly.
Environment Variables:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${JAVA_HOME}/jre/lib/amd64/server:/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib64/
export CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath --glob)
python code
import tensorflow as tf
import tensorflow_io as tfio
ratings = tf.data.experimental.load("hdfs://hdfs_ip:hdfs_port/path/to/datasets")
for elem in ratings:
print(elem)
break
Describe the expected behavior The code should run without error.
- Do you want to contribute a PR? (yes/no): no
- Briefly describe your candidate solution(if contributing):
Standalone code to reproduce the issue Provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/Jupyter/any notebook.
Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
@chunduriv I was able to replicate the issue on colab using TF v2.6.0,2.8.0 and tf-nightly, please find the gist here .Thanks!
HDFS filesystem support is now provided by SIG IO, in tensorflow/io repository. Moving there
