Add examples for all ApiWorkflow functions
Add examples for all ApiWorkflow functions
The ApiWorkflow class provides a lot of usability. You can e.g. create datasets, upload samples, upload embeddings, upload custom metadata, delete datasets, and more. But, currently, the documentation is rather thin. We should add a simple example on how to use these functions for all of them.
Is this issue available to work on?
Yes, it is. It's also a good first issue as it's mostly documentation.
Please assign it to me. I'll start working on it.
On Mon, 20 Sep 2021, 12:24 Philipp Wirth, @.***> wrote:
Yes, it is. It's also a good first issue as it's mostly documentation.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/lightly-ai/lightly/issues/458#issuecomment-922674929, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJUIFVBHPGV3Z6D67PAEVUTUC3LCVANCNFSM5BS4F5RA .
@philippmwirth I'm getting the below error log when trying to run make html for docs. Any ideas what might be the problem here?
make download
make[1]: Entering directory '/home/nikunj/Work/Lightly/Lightly_opensource/docs'
# inspired by https://github.com/pytorch/tutorials/blob/master/Makefile
echo "start downloading datasets..."
start downloading datasets...
mkdir -p _data
mkdir -p source/tutorials_source/platform/data
mkdir -p source/tutorials_source/package/data
# pizza dataset
wget -N https://storage.googleapis.com/datasets_boris/pizzas.zip -P _data
--2021-10-22 12:52:27-- https://storage.googleapis.com/datasets_boris/pizzas.zip
SSL_INIT
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving storage.googleapis.com (storage.googleapis.com)... 2404:6800:4009:82f::2010, 2404:6800:4009:830::2010, 2404:6800:4009:82a::2010, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|2404:6800:4009:82f::2010|:443... connected.
HTTP request sent, awaiting response... 304 Not Modified
File ‘_data/pizzas.zip’ not modified on server. Omitting download.
unzip -qo _data/pizzas.zip -d source/tutorials_source/platform
make[1]: unzip: No such file or directory
make[1]: *** [Makefile:37: download] Error 127
make[1]: Leaving directory '/home/nikunj/Work/Lightly/Lightly_opensource/docs'
make: *** [Makefile:26: html] Error 2
What system are you working on? Can you show me the output when you type
unzip
in your terminal?
What system are you working on? Can you show me the output when you type
unzipin your terminal?
Thanks for the help @philippmwirth . Unzip was not installed. I'm working on arch btw. Also can you assign more issues to me? I might be working on more stuff in this package.
@philippmwirth Here's the next error log I got.
Extension error:
Here is a summary of the problems encountered when running the examples
Unexpected failing examples:
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/platform/tutorial_active_learning.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/platform/tutorial_active_learning.py", line 133, in <module>
from sklearn.linear_model import LogisticRegression
ModuleNotFoundError: No module named 'sklearn'
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_simsiam_esa.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_simsiam_esa.py", line 109, in <module>
dataset_train_simsiam = lightly.data.LightlyDataset(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/data/dataset.py", line 173, in __init__
self.dataset = _load_dataset_from_folder(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/data/_helpers.py", line 107, in _load_dataset_from_folder
raise ValueError(f'The input directory {root} does not exist!')
ValueError: The input directory /datasets/sentinel-2-italy-v1/ does not exist!
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_moco_memory_bank.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_moco_memory_bank.py", line 148, in <module>
dataset_train_moco = lightly.data.LightlyDataset(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/data/dataset.py", line 173, in __init__
self.dataset = _load_dataset_from_folder(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/data/_helpers.py", line 107, in _load_dataset_from_folder
raise ValueError(f'The input directory {root} does not exist!')
ValueError: The input directory /datasets/cifar10/train/ does not exist!
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_custom_augmentations.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_custom_augmentations.py", line 43, in <module>
from sklearn.neighbors import NearestNeighbors
ModuleNotFoundError: No module named 'sklearn'
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/platform/tutorial_active_learning_detectron2.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/platform/tutorial_active_learning_detectron2.py", line 77, in <module>
import detectron2
ModuleNotFoundError: No module named 'detectron2'
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_simclr_clothing.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_simclr_clothing.py", line 45, in <module>
from sklearn.neighbors import NearestNeighbors
ModuleNotFoundError: No module named 'sklearn'
-------------------------------------------------------------------------------
make: *** [Makefile:27: html] Error 2
You need to install scikit-learn and detectron2. Thanks for noticing, I added an issue about this: https://github.com/lightly-ai/lightly/issues/534.
You're also missing the datasets, add cifar10 to /datasets/
You need to install scikit-learn and detectron2. Thanks for noticing, I added an issue about this: #534.
You're also missing the datasets, add
cifar10to/datasets/
I'm facing errors while installing detectron2
Also where can I find the datasets directory?
This is the error output for detectron2
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-req-build-ng0e6bto/setup.py", line 147, in <module>
setup(
File "/usr/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.9/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.9/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/home/nikunj/.local/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 299, in run
self.run_command('build')
File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3.9/distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/usr/lib/python3.9/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/usr/lib/python3.9/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/usr/lib/python3.9/site-packages/setuptools/command/build_ext.py", line 79, in run
_build_ext.run(self)
File "/home/nikunj/.local/lib/python3.9/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "/usr/lib/python3.9/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/home/nikunj/.local/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 390, in build_extensions
self._check_abi()
File "/home/nikunj/.local/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 761, in _check_abi
check_compiler_abi_compatibility(compiler)
File "/home/nikunj/.local/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 294, in check_compiler_abi_compatibility
if not check_compiler_ok_for_platform(compiler):
File "/home/nikunj/.local/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 254, in check_compiler_ok_for_platform
which = subprocess.check_output(['which', compiler], stderr=subprocess.STDOUT)
File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['which', 'g++']' returned non-zero exit status 1.
----------------------------------------
ERROR: Failed building wheel for detectron2
Running setup.py clean for detectron2
Building wheel for pycocotools (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ljy_9_7c/pycocotools_896cf722bc5648b6be66f79a835cf739/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ljy_9_7c/pycocotools_896cf722bc5648b6be66f79a835cf739/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-gi208qze
cwd: /tmp/pip-install-ljy_9_7c/pycocotools_896cf722bc5648b6be66f79a835cf739/
Complete output (20 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/pycocotools
copying pycocotools/mask.py -> build/lib.linux-x86_64-3.9/pycocotools
copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-3.9/pycocotools
copying pycocotools/coco.py -> build/lib.linux-x86_64-3.9/pycocotools
copying pycocotools/__init__.py -> build/lib.linux-x86_64-3.9/pycocotools
running build_ext
cythoning pycocotools/_mask.pyx to pycocotools/_mask.c
/home/nikunj/.local/lib/python3.9/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-ljy_9_7c/pycocotools_896cf722bc5648b6be66f79a835cf739/pycocotools/_mask.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
building 'pycocotools._mask' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/common
creating build/temp.linux-x86_64-3.9/pycocotools
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC -I/home/nikunj/.local/lib/python3.9/site-packages/numpy/core/include -I./common -I/usr/include/python3.9 -c ./common/maskApi.c -o build/temp.linux-x86_64-3.9/./common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Failed building wheel for pycocotools
Running setup.py clean for pycocotools
Failed to build detectron2 pycocotools
Installing collected packages: pycocotools, fvcore, cloudpickle, black, detectron2
Running setup.py install for pycocotools ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ljy_9_7c/pycocotools_896cf722bc5648b6be66f79a835cf739/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ljy_9_7c/pycocotools_896cf722bc5648b6be66f79a835cf739/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-tq2zfb66/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/nikunj/.local/include/python3.9/pycocotools
cwd: /tmp/pip-install-ljy_9_7c/pycocotools_896cf722bc5648b6be66f79a835cf739/
Complete output (18 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/pycocotools
copying pycocotools/mask.py -> build/lib.linux-x86_64-3.9/pycocotools
copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-3.9/pycocotools
copying pycocotools/coco.py -> build/lib.linux-x86_64-3.9/pycocotools
copying pycocotools/__init__.py -> build/lib.linux-x86_64-3.9/pycocotools
running build_ext
skipping 'pycocotools/_mask.c' Cython extension (up-to-date)
building 'pycocotools._mask' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/common
creating build/temp.linux-x86_64-3.9/pycocotools
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fno-semantic-interposition -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC -I/home/nikunj/.local/lib/python3.9/site-packages/numpy/core/include -I./common -I/usr/include/python3.9 -c ./common/maskApi.c -o build/temp.linux-x86_64-3.9/./common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ljy_9_7c/pycocotools_896cf722bc5648b6be66f79a835cf739/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ljy_9_7c/pycocotools_896cf722bc5648b6be66f79a835cf739/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-tq2zfb66/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/nikunj/.local/include/python3.9/pycocotools Check the logs for full command output.
I tried to install some stuff but currently this is the error log I'm getting. How can I add datasets to the project?
Extension error:
Here is a summary of the problems encountered when running the examples
Unexpected failing examples:
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_simclr_clothing.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_simclr_clothing.py", line 107, in <module>
dataset_train_simclr = lightly.data.LightlyDataset(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/data/dataset.py", line 173, in __init__
self.dataset = _load_dataset_from_folder(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/data/_helpers.py", line 107, in _load_dataset_from_folder
raise ValueError(f'The input directory {root} does not exist!')
ValueError: The input directory /datasets/clothing-dataset/images does not exist!
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_moco_memory_bank.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_moco_memory_bank.py", line 148, in <module>
dataset_train_moco = lightly.data.LightlyDataset(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/data/dataset.py", line 173, in __init__
self.dataset = _load_dataset_from_folder(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/data/_helpers.py", line 107, in _load_dataset_from_folder
raise ValueError(f'The input directory {root} does not exist!')
ValueError: The input directory /datasets/cifar10/train/ does not exist!
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/platform/tutorial_active_learning.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/platform/tutorial_active_learning.py", line 190, in <module>
api_workflow_client.create_dataset(dataset_name="active_learning_clothing_dataset")
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/api/api_workflow_datasets.py", line 41, in create_dataset
self.set_dataset_id_by_name(dataset_name)
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/api/api_workflow_datasets.py", line 22, in set_dataset_id_by_name
= self.datasets_api.get_datasets()
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/openapi_generated/swagger_client/api/datasets_api.py", line 343, in get_datasets
(data) = self.get_datasets_with_http_info(**kwargs) # noqa: E501
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/openapi_generated/swagger_client/api/datasets_api.py", line 396, in get_datasets_with_http_info
return self.api_client.call_api(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/openapi_generated/swagger_client/api_client.py", line 326, in call_api
return self.__call_api(resource_path, method,
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/openapi_generated/swagger_client/api_client.py", line 158, in __call_api
response_data = self.request(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/openapi_generated/swagger_client/api_client.py", line 348, in request
return self.rest_client.GET(url,
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/openapi_generated/swagger_client/rest.py", line 234, in GET
return self.request("GET", url,
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/openapi_generated/swagger_client/rest.py", line 228, in request
raise ApiException(http_resp=r)
lightly.openapi_generated.swagger_client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json; charset=utf-8', 'Vary': 'Accept-Encoding, Origin', 'x-cloud-trace-context': 'aefcf7a4754ebcbd95382afbc1fa943a/8129697949176437613;o=1', 'Content-Security-Policy': "script-src https://*.lightly.ai https://*.youtube.com https://*.gstatic.com https://*.google.com https://*.doubleclick.net https://*.google-analytics.com https://*.googletagmanager.com https://*.fullstory.com;img-src *;default-src 'self';base-uri 'self';block-all-mixed-content;font-src 'self' https: data:;frame-ancestors 'self';object-src 'none';script-src-attr 'none';style-src 'self' https: 'unsafe-inline';upgrade-insecure-requests", 'X-DNS-Prefetch-Control': 'off', 'Expect-CT': 'max-age=0', 'X-Frame-Options': 'SAMEORIGIN', 'Strict-Transport-Security': 'max-age=15552000; includeSubDomains', 'X-Download-Options': 'noopen', 'X-Content-Type-Options': 'nosniff', 'X-Permitted-Cross-Domain-Policies': 'none', 'Origin-Agent-Cluster': '?1', 'X-XSS-Protection': '0', 'Cross-Origin-Opener-Policy': 'same-origin-allow-popups', 'Cross-Origin-Resource-Policy': 'same-site', 'ETag': 'W/"3c-2MASddOo7f/Y1UTHEz2BMijz4sE"', 'Date': 'Sun, 24 Oct 2021 10:57:27 GMT', 'Server': 'Google Frontend', 'Content-Length': '60'})
HTTP response body: {
"error": "Not able to authenticate user using token"
}
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_simsiam_esa.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_simsiam_esa.py", line 109, in <module>
dataset_train_simsiam = lightly.data.LightlyDataset(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/data/dataset.py", line 173, in __init__
self.dataset = _load_dataset_from_folder(
File "/home/nikunj/Work/Lightly/Lightly_opensource/lightly/data/_helpers.py", line 107, in _load_dataset_from_folder
raise ValueError(f'The input directory {root} does not exist!')
ValueError: The input directory /datasets/sentinel-2-italy-v1/ does not exist!
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/platform/tutorial_active_learning_detectron2.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/platform/tutorial_active_learning_detectron2.py", line 77, in <module>
import detectron2
ModuleNotFoundError: No module named 'detectron2'
/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_custom_augmentations.py failed leaving traceback:
Traceback (most recent call last):
File "/home/nikunj/Work/Lightly/Lightly_opensource/docs/source/tutorials_source/package/tutorial_custom_augmentations.py", line 164, in <module>
example_image = np.array(Image.open(example_image_path))
File "/usr/lib/python3.9/site-packages/PIL/Image.py", line 2968, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/datasets/vinbigdata/train_small/55e8e3db7309febee415515d06418171.tiff'
-------------------------------------------------------------------------------
make: *** [Makefile:27: html] Error 2
From what I see I believe the make file requires the download of additional datasets other than pizzas and sunflowers.
Hi @Nike682631, can you check if the html code was still generated? You can simply do this by running e.g.
python -m http.server 1234
and then browse the html files in your browser on port http://localhost:1234. Because it should not be necessary to build all the tutorials for this issue.
Hey @philippmwirth, so I guess a good part of it is building fine. You can see that in the pic below:
Further, I wanted to ask on that I have to add examples for all in functions in this class right? : https://docs.lightly.ai/lightly.api.html#module-lightly.api.api_workflow_client. Also what all stuff do I need to study to be able to properly understand the implementation of the functions of this class? Currently I'll try to make some examples taking reference from the examples given for other functions but I'm relatively new to all this so do let me know if I have to watch some other tutorials to properly grab everything given in the docs.
Yes, you can find all the functions under lightly.api.api_workflow*. Most of them should be self-explaining with the help of the docstring. If you have questions about certain functions you can just ask them here.
If you want to read up I would suggest you take a look at the active learning tutorials here.
Hi @philippmwirth , Is this issue still available to work on? Apologies for getting back after a long time.
Completed.